Nov 04 04:46:35.664645 ip-10-0-133-180 systemd[1]: Starting Kubernetes Kubelet... Nov 04 04:46:36.064908 ip-10-0-133-180 kubenswrapper[2579]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 04 04:46:36.064908 ip-10-0-133-180 kubenswrapper[2579]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 04 04:46:36.064908 ip-10-0-133-180 kubenswrapper[2579]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 04 04:46:36.064908 ip-10-0-133-180 kubenswrapper[2579]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 04 04:46:36.072211 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.072132 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 04:46:36.072211 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.072210 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 04:46:36.072211 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.072215 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 04:46:36.072211 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.072218 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 04:46:36.072338 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.072222 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 04:46:36.072338 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.072224 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 04:46:36.072338 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.072227 2579 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 04:46:36.072338 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.072230 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 04:46:36.072338 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.072233 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 04:46:36.072338 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.072235 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 04:46:36.072338 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.072238 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 04:46:36.072338 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.072240 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 04:46:36.072338 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.072243 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 04:46:36.072338 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.072245 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 04:46:36.072338 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.072248 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 04:46:36.072338 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.072256 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 04:46:36.072338 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.072259 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 04:46:36.072338 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.072261 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 04:46:36.072338 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.072265 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 04:46:36.072338 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.072271 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 04:46:36.072338 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.072274 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 04:46:36.072338 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.072277 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 04:46:36.072338 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.072280 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 04:46:36.072760 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.072283 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 04:46:36.072760 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.072285 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 04:46:36.072760 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.072288 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 04:46:36.072760 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.072290 2579 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 04:46:36.072760 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.072293 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 04:46:36.072760 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.072295 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 04:46:36.072760 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.072298 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 04:46:36.072760 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.072300 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 04:46:36.072760 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.072303 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 04:46:36.072760 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.072305 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 04:46:36.072760 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.072308 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 04:46:36.072760 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.072310 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 04:46:36.072760 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.072313 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 04:46:36.072760 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.072315 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 04:46:36.072760 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.072330 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 04:46:36.072760 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.072334 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 04:46:36.072760 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.072336 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 04:46:36.072760 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.072339 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 04:46:36.072760 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.072341 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 04:46:36.073197 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.072344 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 04:46:36.073197 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.072347 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 04:46:36.073197 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.072350 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 04:46:36.073197 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.072352 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 04:46:36.073197 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.072357 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 04:46:36.073197 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.072361 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 04:46:36.073197 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.072365 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 04:46:36.073197 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.072368 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 04:46:36.073197 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.072371 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 04:46:36.073197 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.072373 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 04:46:36.073197 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.072375 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 04:46:36.073197 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.072378 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 04:46:36.073197 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.072380 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 04:46:36.073197 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.072383 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 04:46:36.073197 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.072385 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 04:46:36.073197 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.072388 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 04:46:36.073197 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.072390 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 04:46:36.073197 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.072392 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 04:46:36.073197 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.072395 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 04:46:36.073643 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.072397 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 04:46:36.073643 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.072400 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 04:46:36.073643 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.072403 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 04:46:36.073643 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.072405 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 04:46:36.073643 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.072408 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 04:46:36.073643 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.072410 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 04:46:36.073643 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.072412 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 04:46:36.073643 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.072415 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 04:46:36.073643 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.072419 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 04:46:36.073643 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.072422 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 04:46:36.073643 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.072424 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 04:46:36.073643 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.072427 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 04:46:36.073643 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.072430 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 04:46:36.073643 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.072432 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 04:46:36.073643 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.072435 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 04:46:36.073643 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.072437 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 04:46:36.073643 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.072440 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 04:46:36.073643 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.072442 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 04:46:36.073643 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.072445 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 04:46:36.073643 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.072447 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 04:46:36.074122 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.072450 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 04:46:36.074122 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.072453 2579 feature_gate.go:386] unrecognized feature gate: Example Nov 04 04:46:36.074122 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.072455 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 04:46:36.074122 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.072457 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 04:46:36.074122 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.072460 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 04:46:36.074122 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.072462 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 04:46:36.074122 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.072465 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 04:46:36.074122 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.072467 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 04:46:36.074122 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.072470 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 04:46:36.074122 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.072472 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 04:46:36.074122 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.072475 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 04:46:36.074122 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.072477 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 04:46:36.074122 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.072479 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 04:46:36.074122 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.072482 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 04:46:36.074122 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.072485 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 04:46:36.074122 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.072487 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 04:46:36.074122 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.072490 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 04:46:36.074122 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.072492 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 04:46:36.074122 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.072497 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 04:46:36.074122 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.072499 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 04:46:36.074609 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.072502 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 04:46:36.074609 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.072505 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 04:46:36.074609 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.072507 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 04:46:36.074609 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.072509 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 04:46:36.074609 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.072512 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 04:46:36.074609 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.073261 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 04:46:36.074609 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.073267 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 04:46:36.074609 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.073270 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 04:46:36.074609 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.073273 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 04:46:36.074609 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.073275 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 04:46:36.074609 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.073278 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 04:46:36.074609 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.073280 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 04:46:36.074609 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.073283 2579 feature_gate.go:386] unrecognized feature gate: Example Nov 04 04:46:36.074609 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.073286 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 04:46:36.074609 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.073290 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 04:46:36.074609 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.073293 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 04:46:36.074609 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.073295 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 04:46:36.074609 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.073298 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 04:46:36.074609 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.073301 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 04:46:36.074609 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.073303 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 04:46:36.075088 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.073305 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 04:46:36.075088 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.073309 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 04:46:36.075088 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.073311 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 04:46:36.075088 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.073314 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 04:46:36.075088 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.073317 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 04:46:36.075088 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.073334 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 04:46:36.075088 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.073337 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 04:46:36.075088 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.073339 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 04:46:36.075088 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.073342 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 04:46:36.075088 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.073344 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 04:46:36.075088 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.073347 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 04:46:36.075088 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.073349 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 04:46:36.075088 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.073352 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 04:46:36.075088 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.073355 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 04:46:36.075088 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.073357 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 04:46:36.075088 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.073360 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 04:46:36.075088 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.073362 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 04:46:36.075088 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.073365 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 04:46:36.075088 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.073367 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 04:46:36.075525 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.073370 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 04:46:36.075525 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.073372 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 04:46:36.075525 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.073375 2579 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 04:46:36.075525 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.073378 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 04:46:36.075525 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.073380 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 04:46:36.075525 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.073383 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 04:46:36.075525 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.073385 2579 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 04:46:36.075525 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.073388 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 04:46:36.075525 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.073390 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 04:46:36.075525 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.073394 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 04:46:36.075525 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.073396 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 04:46:36.075525 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.073399 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 04:46:36.075525 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.073401 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 04:46:36.075525 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.073404 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 04:46:36.075525 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.073406 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 04:46:36.075525 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.073409 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 04:46:36.075525 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.073411 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 04:46:36.075525 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.073413 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 04:46:36.075525 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.073416 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 04:46:36.075525 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.073418 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 04:46:36.075980 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.073420 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 04:46:36.075980 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.073423 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 04:46:36.075980 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.073425 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 04:46:36.075980 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.073428 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 04:46:36.075980 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.073430 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 04:46:36.075980 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.073433 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 04:46:36.075980 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.073436 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 04:46:36.075980 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.073439 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 04:46:36.075980 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.073441 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 04:46:36.075980 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.073444 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 04:46:36.075980 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.073448 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 04:46:36.075980 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.073452 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 04:46:36.075980 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.073455 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 04:46:36.075980 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.073458 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 04:46:36.075980 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.073462 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 04:46:36.075980 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.073465 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 04:46:36.075980 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.073467 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 04:46:36.075980 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.073470 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 04:46:36.075980 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.073473 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 04:46:36.076431 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.073476 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 04:46:36.076431 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.073480 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 04:46:36.076431 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.073483 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 04:46:36.076431 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.073486 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 04:46:36.076431 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.073489 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 04:46:36.076431 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.073491 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 04:46:36.076431 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.073494 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 04:46:36.076431 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.073497 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 04:46:36.076431 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.073499 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 04:46:36.076431 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.073502 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 04:46:36.076431 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.073504 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 04:46:36.076431 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.073507 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 04:46:36.076431 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.073509 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 04:46:36.076431 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.073512 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 04:46:36.076431 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.073514 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 04:46:36.076431 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.073522 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 04:46:36.076431 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.073525 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 04:46:36.076431 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.073528 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 04:46:36.076431 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.073531 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 04:46:36.076431 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.073533 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 04:46:36.076926 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.073536 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 04:46:36.076926 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.073538 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 04:46:36.076926 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.073541 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 04:46:36.076926 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.073543 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 04:46:36.076926 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.073546 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 04:46:36.076926 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.073548 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 04:46:36.076926 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.073551 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 04:46:36.076926 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.073553 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 04:46:36.076926 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.073555 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 04:46:36.076926 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.073558 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 04:46:36.076926 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.073560 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 04:46:36.076926 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.073563 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 04:46:36.076926 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.073565 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 04:46:36.076926 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.074112 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 04:46:36.076926 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.074118 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 04:46:36.076926 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.074121 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 04:46:36.076926 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.074125 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 04:46:36.076926 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.074128 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 04:46:36.076926 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.074131 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 04:46:36.076926 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.074134 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 04:46:36.077500 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.074137 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 04:46:36.077500 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.074139 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 04:46:36.077500 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.074142 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 04:46:36.077500 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.074146 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 04:46:36.077500 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.074148 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 04:46:36.077500 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.074151 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 04:46:36.077500 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.074153 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 04:46:36.077500 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.074157 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 04:46:36.077500 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.074159 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 04:46:36.077500 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.074162 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 04:46:36.077500 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.074164 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 04:46:36.077500 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.074167 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 04:46:36.077500 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.074169 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 04:46:36.077500 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.074172 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 04:46:36.077500 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.074175 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 04:46:36.077500 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.074177 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 04:46:36.077500 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.074180 2579 feature_gate.go:386] unrecognized feature gate: Example Nov 04 04:46:36.077500 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.074182 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 04:46:36.077500 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.074185 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 04:46:36.077500 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.074188 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 04:46:36.078001 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.074191 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 04:46:36.078001 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.074195 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 04:46:36.078001 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.074198 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 04:46:36.078001 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.074201 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 04:46:36.078001 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.074203 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 04:46:36.078001 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.074206 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 04:46:36.078001 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.074210 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 04:46:36.078001 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.074213 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 04:46:36.078001 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.074216 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 04:46:36.078001 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.074218 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 04:46:36.078001 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.074221 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 04:46:36.078001 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.074223 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 04:46:36.078001 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.074226 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 04:46:36.078001 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.074229 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 04:46:36.078001 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.074231 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 04:46:36.078001 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.074234 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 04:46:36.078001 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.074236 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 04:46:36.078001 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.074238 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 04:46:36.078001 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.074241 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 04:46:36.078503 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.074244 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 04:46:36.078503 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.074247 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 04:46:36.078503 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.074249 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 04:46:36.078503 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.074252 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 04:46:36.078503 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.074254 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 04:46:36.078503 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.074257 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 04:46:36.078503 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.074259 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 04:46:36.078503 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.074262 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 04:46:36.078503 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.074264 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 04:46:36.078503 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.074267 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 04:46:36.078503 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.074270 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 04:46:36.078503 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.074273 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 04:46:36.078503 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.074275 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 04:46:36.078503 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.074278 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 04:46:36.078503 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.074280 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 04:46:36.078503 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.074283 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 04:46:36.078503 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.074285 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 04:46:36.078503 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.074288 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 04:46:36.078503 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.074290 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 04:46:36.078952 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.074293 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 04:46:36.078952 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.074298 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 04:46:36.078952 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.074301 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 04:46:36.078952 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.074304 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 04:46:36.078952 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.074307 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 04:46:36.078952 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.074310 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 04:46:36.078952 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.074312 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 04:46:36.078952 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.074315 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 04:46:36.078952 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.074318 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 04:46:36.078952 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.074334 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 04:46:36.078952 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.074337 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 04:46:36.078952 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.074339 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 04:46:36.078952 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.074341 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 04:46:36.078952 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.074344 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 04:46:36.078952 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.074347 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 04:46:36.078952 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.074350 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 04:46:36.078952 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.074352 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 04:46:36.078952 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.074355 2579 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 04:46:36.078952 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.074358 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 04:46:36.079400 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.074360 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 04:46:36.079400 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.074363 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 04:46:36.079400 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.074365 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 04:46:36.079400 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.074368 2579 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 04:46:36.079400 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.074373 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 04:46:36.079400 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.074376 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 04:46:36.079400 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.074379 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 04:46:36.079400 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.074381 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 04:46:36.079400 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.074384 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 04:46:36.079400 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.074386 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 04:46:36.079400 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.074389 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 04:46:36.079400 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.074392 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 04:46:36.079400 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.074394 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 04:46:36.079400 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.074398 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 04:46:36.079400 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.074401 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 04:46:36.079400 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.074404 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 04:46:36.079400 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.074406 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 04:46:36.079400 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.074409 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 04:46:36.079400 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.074411 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 04:46:36.079860 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.074414 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 04:46:36.079860 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.074417 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 04:46:36.079860 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.074419 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 04:46:36.079860 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.075103 2579 flags.go:64] FLAG: --address="0.0.0.0" Nov 04 04:46:36.079860 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.075111 2579 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 04 04:46:36.079860 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.075119 2579 flags.go:64] FLAG: --anonymous-auth="true" Nov 04 04:46:36.079860 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.075124 2579 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 04 04:46:36.079860 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.075128 2579 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 04 04:46:36.079860 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.075132 2579 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 04 04:46:36.079860 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.075136 2579 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 04 04:46:36.079860 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.075140 2579 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 04 04:46:36.079860 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.075143 2579 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 04 04:46:36.079860 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.075146 2579 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 04 04:46:36.079860 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.075150 2579 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 04 04:46:36.079860 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.075153 2579 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 04 04:46:36.079860 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.075156 2579 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 04 04:46:36.079860 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.075159 2579 flags.go:64] FLAG: --cgroup-root="" Nov 04 04:46:36.079860 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.075164 2579 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 04 04:46:36.079860 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.075167 2579 flags.go:64] FLAG: --client-ca-file="" Nov 04 04:46:36.079860 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.075169 2579 flags.go:64] FLAG: --cloud-provider="external" Nov 04 04:46:36.079860 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.075172 2579 flags.go:64] FLAG: --cluster-dns="[]" Nov 04 04:46:36.079860 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.075176 2579 flags.go:64] FLAG: --cluster-domain="" Nov 04 04:46:36.079860 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.075179 2579 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 04 04:46:36.079860 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.075182 2579 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 04 04:46:36.080419 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.075186 2579 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 04 04:46:36.080419 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.075189 2579 flags.go:64] FLAG: --container-log-max-files="5" Nov 04 04:46:36.080419 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.075193 2579 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 04 04:46:36.080419 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.075196 2579 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 04 04:46:36.080419 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.075199 2579 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 04 04:46:36.080419 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.075202 2579 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 04 04:46:36.080419 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.075205 2579 flags.go:64] FLAG: --contention-profiling="false" Nov 04 04:46:36.080419 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.075208 2579 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 04 04:46:36.080419 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.075211 2579 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 04 04:46:36.080419 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.075214 2579 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 04 04:46:36.080419 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.075217 2579 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 04 04:46:36.080419 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.075221 2579 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 04 04:46:36.080419 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.075224 2579 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 04 04:46:36.080419 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.075226 2579 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 04 04:46:36.080419 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.075229 2579 flags.go:64] FLAG: --enable-load-reader="false" Nov 04 04:46:36.080419 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.075234 2579 flags.go:64] FLAG: --enable-server="true" Nov 04 04:46:36.080419 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.075237 2579 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 04 04:46:36.080419 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.075242 2579 flags.go:64] FLAG: --event-burst="100" Nov 04 04:46:36.080419 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.075245 2579 flags.go:64] FLAG: --event-qps="50" Nov 04 04:46:36.080419 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.075248 2579 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 04 04:46:36.080419 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.075251 2579 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 04 04:46:36.080419 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.075254 2579 flags.go:64] FLAG: --eviction-hard="" Nov 04 04:46:36.080419 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.075261 2579 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 04 04:46:36.080419 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.075264 2579 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 04 04:46:36.080971 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.075267 2579 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 04 04:46:36.080971 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.075271 2579 flags.go:64] FLAG: --eviction-soft="" Nov 04 04:46:36.080971 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.075275 2579 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 04 04:46:36.080971 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.075277 2579 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 04 04:46:36.080971 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.075281 2579 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 04 04:46:36.080971 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.075283 2579 flags.go:64] FLAG: --experimental-mounter-path="" Nov 04 04:46:36.080971 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.075286 2579 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 04 04:46:36.080971 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.075289 2579 flags.go:64] FLAG: --fail-swap-on="true" Nov 04 04:46:36.080971 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.075292 2579 flags.go:64] FLAG: --feature-gates="" Nov 04 04:46:36.080971 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.075296 2579 flags.go:64] FLAG: --file-check-frequency="20s" Nov 04 04:46:36.080971 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.075299 2579 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 04 04:46:36.080971 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.075302 2579 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 04 04:46:36.080971 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.075306 2579 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 04 04:46:36.080971 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.075309 2579 flags.go:64] FLAG: --healthz-port="10248" Nov 04 04:46:36.080971 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.075312 2579 flags.go:64] FLAG: --help="false" Nov 04 04:46:36.080971 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.075315 2579 flags.go:64] FLAG: --hostname-override="ip-10-0-133-180.us-west-1.compute.internal" Nov 04 04:46:36.080971 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.075332 2579 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 04 04:46:36.080971 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.075336 2579 flags.go:64] FLAG: --http-check-frequency="20s" Nov 04 04:46:36.080971 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.075339 2579 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 04 04:46:36.080971 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.075342 2579 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 04 04:46:36.080971 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.075346 2579 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 04 04:46:36.080971 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.075348 2579 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 04 04:46:36.080971 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.075351 2579 flags.go:64] FLAG: --image-service-endpoint="" Nov 04 04:46:36.081553 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.075355 2579 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 04 04:46:36.081553 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.075358 2579 flags.go:64] FLAG: --kube-api-burst="100" Nov 04 04:46:36.081553 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.075361 2579 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 04 04:46:36.081553 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.075364 2579 flags.go:64] FLAG: --kube-api-qps="50" Nov 04 04:46:36.081553 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.075367 2579 flags.go:64] FLAG: --kube-reserved="" Nov 04 04:46:36.081553 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.075370 2579 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 04 04:46:36.081553 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.075372 2579 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 04 04:46:36.081553 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.075375 2579 flags.go:64] FLAG: --kubelet-cgroups="" Nov 04 04:46:36.081553 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.075378 2579 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 04 04:46:36.081553 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.075381 2579 flags.go:64] FLAG: --lock-file="" Nov 04 04:46:36.081553 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.075385 2579 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 04 04:46:36.081553 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.075388 2579 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 04 04:46:36.081553 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.075391 2579 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 04 04:46:36.081553 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.075397 2579 flags.go:64] FLAG: --log-json-split-stream="false" Nov 04 04:46:36.081553 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.075399 2579 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 04 04:46:36.081553 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.075402 2579 flags.go:64] FLAG: --log-text-split-stream="false" Nov 04 04:46:36.081553 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.075405 2579 flags.go:64] FLAG: --logging-format="text" Nov 04 04:46:36.081553 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.075408 2579 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 04 04:46:36.081553 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.075411 2579 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 04 04:46:36.081553 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.075414 2579 flags.go:64] FLAG: --manifest-url="" Nov 04 04:46:36.081553 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.075417 2579 flags.go:64] FLAG: --manifest-url-header="" Nov 04 04:46:36.081553 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.075421 2579 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 04 04:46:36.081553 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.075424 2579 flags.go:64] FLAG: --max-open-files="1000000" Nov 04 04:46:36.081553 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.075428 2579 flags.go:64] FLAG: --max-pods="110" Nov 04 04:46:36.081553 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.075432 2579 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 04 04:46:36.082147 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.075435 2579 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 04 04:46:36.082147 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.075437 2579 flags.go:64] FLAG: --memory-manager-policy="None" Nov 04 04:46:36.082147 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.075440 2579 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 04 04:46:36.082147 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.075443 2579 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 04 04:46:36.082147 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.075446 2579 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 04 04:46:36.082147 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.075448 2579 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 04 04:46:36.082147 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.075455 2579 flags.go:64] FLAG: --node-status-max-images="50" Nov 04 04:46:36.082147 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.075460 2579 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 04 04:46:36.082147 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.075463 2579 flags.go:64] FLAG: --oom-score-adj="-999" Nov 04 04:46:36.082147 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.075465 2579 flags.go:64] FLAG: --pod-cidr="" Nov 04 04:46:36.082147 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.075468 2579 flags.go:64] FLAG: --pod-manifest-path="" Nov 04 04:46:36.082147 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.075471 2579 flags.go:64] FLAG: --pod-max-pids="-1" Nov 04 04:46:36.082147 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.075474 2579 flags.go:64] FLAG: --pods-per-core="0" Nov 04 04:46:36.082147 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.075477 2579 flags.go:64] FLAG: --port="10250" Nov 04 04:46:36.082147 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.075481 2579 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 04 04:46:36.082147 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.075483 2579 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-0e23cc09baf76542f" Nov 04 04:46:36.082147 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.075486 2579 flags.go:64] FLAG: --qos-reserved="" Nov 04 04:46:36.082147 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.075490 2579 flags.go:64] FLAG: --read-only-port="10255" Nov 04 04:46:36.082147 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.075494 2579 flags.go:64] FLAG: --register-node="true" Nov 04 04:46:36.082147 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.075496 2579 flags.go:64] FLAG: --register-with-taints="" Nov 04 04:46:36.082147 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.075500 2579 flags.go:64] FLAG: --registry-burst="10" Nov 04 04:46:36.082147 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.075503 2579 flags.go:64] FLAG: --registry-qps="5" Nov 04 04:46:36.082147 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.075506 2579 flags.go:64] FLAG: --reserved-cpus="" Nov 04 04:46:36.082147 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.075509 2579 flags.go:64] FLAG: --reserved-memory="" Nov 04 04:46:36.082147 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.075512 2579 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 04 04:46:36.082147 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.075515 2579 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 04 04:46:36.082777 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.075518 2579 flags.go:64] FLAG: --rotate-certificates="false" Nov 04 04:46:36.082777 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.075521 2579 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 04 04:46:36.082777 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.075524 2579 flags.go:64] FLAG: --runonce="false" Nov 04 04:46:36.082777 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.075527 2579 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 04 04:46:36.082777 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.075531 2579 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 04 04:46:36.082777 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.075534 2579 flags.go:64] FLAG: --seccomp-default="false" Nov 04 04:46:36.082777 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.075537 2579 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 04 04:46:36.082777 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.075540 2579 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 04 04:46:36.082777 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.075543 2579 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 04 04:46:36.082777 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.075546 2579 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 04 04:46:36.082777 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.075549 2579 flags.go:64] FLAG: --storage-driver-password="root" Nov 04 04:46:36.082777 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.075551 2579 flags.go:64] FLAG: --storage-driver-secure="false" Nov 04 04:46:36.082777 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.075554 2579 flags.go:64] FLAG: --storage-driver-table="stats" Nov 04 04:46:36.082777 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.075558 2579 flags.go:64] FLAG: --storage-driver-user="root" Nov 04 04:46:36.082777 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.075561 2579 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 04 04:46:36.082777 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.075564 2579 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 04 04:46:36.082777 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.075567 2579 flags.go:64] FLAG: --system-cgroups="" Nov 04 04:46:36.082777 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.075569 2579 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 04 04:46:36.082777 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.075574 2579 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 04 04:46:36.082777 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.075577 2579 flags.go:64] FLAG: --tls-cert-file="" Nov 04 04:46:36.082777 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.075580 2579 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 04 04:46:36.082777 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.075584 2579 flags.go:64] FLAG: --tls-min-version="" Nov 04 04:46:36.082777 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.075586 2579 flags.go:64] FLAG: --tls-private-key-file="" Nov 04 04:46:36.082777 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.075590 2579 flags.go:64] FLAG: --topology-manager-policy="none" Nov 04 04:46:36.082777 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.075593 2579 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 04 04:46:36.083415 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.075596 2579 flags.go:64] FLAG: --topology-manager-scope="container" Nov 04 04:46:36.083415 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.075598 2579 flags.go:64] FLAG: --v="2" Nov 04 04:46:36.083415 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.075603 2579 flags.go:64] FLAG: --version="false" Nov 04 04:46:36.083415 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.075606 2579 flags.go:64] FLAG: --vmodule="" Nov 04 04:46:36.083415 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.075610 2579 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 04 04:46:36.083415 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.075613 2579 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 04 04:46:36.083415 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.075749 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 04:46:36.083415 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.075753 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 04:46:36.083415 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.075755 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 04:46:36.083415 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.075758 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 04:46:36.083415 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.075760 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 04:46:36.083415 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.075763 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 04:46:36.083415 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.075765 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 04:46:36.083415 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.075768 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 04:46:36.083415 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.075771 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 04:46:36.083415 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.075773 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 04:46:36.083415 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.075776 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 04:46:36.083415 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.075778 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 04:46:36.083415 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.075780 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 04:46:36.083415 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.075784 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 04:46:36.083859 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.075789 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 04:46:36.083859 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.075792 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 04:46:36.083859 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.075795 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 04:46:36.083859 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.075798 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 04:46:36.083859 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.075800 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 04:46:36.083859 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.075803 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 04:46:36.083859 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.075805 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 04:46:36.083859 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.075808 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 04:46:36.083859 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.075810 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 04:46:36.083859 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.075813 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 04:46:36.083859 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.075815 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 04:46:36.083859 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.075818 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 04:46:36.083859 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.075820 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 04:46:36.083859 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.075823 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 04:46:36.083859 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.075825 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 04:46:36.083859 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.075828 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 04:46:36.083859 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.075830 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 04:46:36.083859 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.075833 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 04:46:36.083859 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.075835 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 04:46:36.083859 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.075838 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 04:46:36.084310 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.075840 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 04:46:36.084310 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.075843 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 04:46:36.084310 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.075845 2579 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 04:46:36.084310 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.075848 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 04:46:36.084310 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.075851 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 04:46:36.084310 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.075855 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 04:46:36.084310 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.075858 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 04:46:36.084310 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.075861 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 04:46:36.084310 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.075863 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 04:46:36.084310 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.075865 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 04:46:36.084310 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.075868 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 04:46:36.084310 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.075870 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 04:46:36.084310 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.075874 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 04:46:36.084310 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.075876 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 04:46:36.084310 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.075879 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 04:46:36.084310 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.075882 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 04:46:36.084310 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.075884 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 04:46:36.084310 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.075886 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 04:46:36.084310 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.075889 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 04:46:36.084769 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.075891 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 04:46:36.084769 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.075894 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 04:46:36.084769 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.075896 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 04:46:36.084769 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.075899 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 04:46:36.084769 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.075901 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 04:46:36.084769 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.075903 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 04:46:36.084769 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.075906 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 04:46:36.084769 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.075908 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 04:46:36.084769 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.075911 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 04:46:36.084769 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.075913 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 04:46:36.084769 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.075916 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 04:46:36.084769 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.075918 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 04:46:36.084769 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.075920 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 04:46:36.084769 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.075923 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 04:46:36.084769 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.075925 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 04:46:36.084769 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.075927 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 04:46:36.084769 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.075930 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 04:46:36.084769 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.075932 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 04:46:36.084769 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.075935 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 04:46:36.084769 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.075937 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 04:46:36.085216 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.075940 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 04:46:36.085216 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.075943 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 04:46:36.085216 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.075945 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 04:46:36.085216 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.075948 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 04:46:36.085216 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.075950 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 04:46:36.085216 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.075954 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 04:46:36.085216 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.075956 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 04:46:36.085216 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.075959 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 04:46:36.085216 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.075961 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 04:46:36.085216 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.075963 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 04:46:36.085216 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.075966 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 04:46:36.085216 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.075969 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 04:46:36.085216 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.075971 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 04:46:36.085216 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.075973 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 04:46:36.085216 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.075976 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 04:46:36.085216 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.075979 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 04:46:36.085216 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.075981 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 04:46:36.085216 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.075984 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 04:46:36.085216 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.075986 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 04:46:36.085216 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.075989 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 04:46:36.085703 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.075992 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 04:46:36.085703 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.075994 2579 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 04:46:36.085703 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.075996 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 04:46:36.085703 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.075999 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 04:46:36.085703 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.076001 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 04:46:36.085703 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.076003 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 04:46:36.085703 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.076006 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 04:46:36.085703 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.076008 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 04:46:36.085703 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.076011 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 04:46:36.085703 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.076013 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 04:46:36.085703 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.076015 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 04:46:36.085703 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.076019 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 04:46:36.085703 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.076021 2579 feature_gate.go:386] unrecognized feature gate: Example Nov 04 04:46:36.085703 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.076046 2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 04 04:46:36.085703 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.076086 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 04:46:36.085703 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.076091 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 04:46:36.086115 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.076094 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 04:46:36.086115 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.076097 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 04:46:36.086115 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.076101 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 04:46:36.086115 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.076104 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 04:46:36.086115 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.076106 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 04:46:36.086115 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.076109 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 04:46:36.086115 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.076111 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 04:46:36.086115 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.076114 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 04:46:36.086115 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.076116 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 04:46:36.086115 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.076119 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 04:46:36.086115 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.076121 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 04:46:36.086115 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.076124 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 04:46:36.086115 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.076126 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 04:46:36.086115 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.076129 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 04:46:36.086115 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.076131 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 04:46:36.086115 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.076134 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 04:46:36.086115 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.076136 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 04:46:36.086115 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.076139 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 04:46:36.086115 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.076142 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 04:46:36.086115 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.076144 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 04:46:36.086614 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.076147 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 04:46:36.086614 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.076149 2579 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 04:46:36.086614 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.076152 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 04:46:36.086614 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.076155 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 04:46:36.086614 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.076159 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 04:46:36.086614 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.076162 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 04:46:36.086614 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.076165 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 04:46:36.086614 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.076169 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 04:46:36.086614 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.076171 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 04:46:36.086614 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.076174 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 04:46:36.086614 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.076176 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 04:46:36.086614 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.076179 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 04:46:36.086614 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.076182 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 04:46:36.086614 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.076185 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 04:46:36.086614 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.076189 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 04:46:36.086614 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.076192 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 04:46:36.086614 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.076194 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 04:46:36.086614 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.076197 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 04:46:36.086614 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.076199 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 04:46:36.087197 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.076202 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 04:46:36.087197 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.076204 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 04:46:36.087197 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.076207 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 04:46:36.087197 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.076209 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 04:46:36.087197 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.076212 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 04:46:36.087197 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.076214 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 04:46:36.087197 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.076217 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 04:46:36.087197 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.076219 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 04:46:36.087197 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.076222 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 04:46:36.087197 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.076224 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 04:46:36.087197 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.076227 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 04:46:36.087197 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.076230 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 04:46:36.087197 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.076232 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 04:46:36.087197 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.076235 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 04:46:36.087197 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.076237 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 04:46:36.087197 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.076239 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 04:46:36.087197 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.076242 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 04:46:36.087197 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.076244 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 04:46:36.087197 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.076246 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 04:46:36.087197 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.076249 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 04:46:36.087944 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.076252 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 04:46:36.087944 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.076255 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 04:46:36.087944 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.076257 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 04:46:36.087944 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.076260 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 04:46:36.087944 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.076262 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 04:46:36.087944 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.076266 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 04:46:36.087944 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.076268 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 04:46:36.087944 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.076272 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 04:46:36.087944 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.076274 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 04:46:36.087944 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.076277 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 04:46:36.087944 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.076280 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 04:46:36.087944 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.076282 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 04:46:36.087944 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.076284 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 04:46:36.087944 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.076287 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 04:46:36.087944 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.076289 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 04:46:36.087944 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.076292 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 04:46:36.087944 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.076294 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 04:46:36.087944 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.076296 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 04:46:36.087944 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.076299 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 04:46:36.087944 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.076301 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 04:46:36.088730 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.076304 2579 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 04:46:36.088730 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.076306 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 04:46:36.088730 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.076309 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 04:46:36.088730 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.076311 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 04:46:36.088730 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.076314 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 04:46:36.088730 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.076316 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 04:46:36.088730 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.076330 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 04:46:36.088730 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.076333 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 04:46:36.088730 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.076335 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 04:46:36.088730 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.076338 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 04:46:36.088730 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.076341 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 04:46:36.088730 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.076343 2579 feature_gate.go:386] unrecognized feature gate: Example Nov 04 04:46:36.088730 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.076347 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 04:46:36.088730 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.076349 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 04:46:36.088730 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.076352 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 04:46:36.088730 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.076355 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 04:46:36.088730 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.076357 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 04:46:36.088730 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.076359 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 04:46:36.088730 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.076362 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 04:46:36.088730 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.076366 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 04:46:36.089281 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.076369 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 04:46:36.089281 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.076371 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 04:46:36.089281 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.076375 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 04:46:36.089281 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.076377 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 04:46:36.089281 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.076380 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 04:46:36.089281 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.082799 2579 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 04 04:46:36.089281 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.082816 2579 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 04 04:46:36.089281 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.082834 2579 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 04 04:46:36.089281 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.082838 2579 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 04 04:46:36.089281 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.082898 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 04:46:36.089281 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.082903 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 04:46:36.089281 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.082907 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 04:46:36.089281 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.082910 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 04:46:36.089281 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.082926 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 04:46:36.089281 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.082929 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 04:46:36.089281 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.082931 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 04:46:36.089281 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.082934 2579 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 04:46:36.089281 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.082937 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 04:46:36.089281 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.082939 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 04:46:36.089281 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.082941 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 04:46:36.089759 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.082944 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 04:46:36.089759 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.082946 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 04:46:36.089759 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.082949 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 04:46:36.089759 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.082952 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 04:46:36.089759 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.082954 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 04:46:36.089759 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.082957 2579 feature_gate.go:386] unrecognized feature gate: Example Nov 04 04:46:36.089759 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.082959 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 04:46:36.089759 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.082962 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 04:46:36.089759 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.082965 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 04:46:36.089759 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.082968 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 04:46:36.089759 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.082971 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 04:46:36.089759 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.082974 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 04:46:36.089759 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.082976 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 04:46:36.089759 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.082979 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 04:46:36.089759 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.082981 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 04:46:36.089759 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.082984 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 04:46:36.089759 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.082987 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 04:46:36.089759 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.082989 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 04:46:36.089759 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.082993 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 04:46:36.090263 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.082996 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 04:46:36.090263 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.082998 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 04:46:36.090263 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083001 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 04:46:36.090263 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083003 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 04:46:36.090263 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083006 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 04:46:36.090263 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083008 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 04:46:36.090263 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083011 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 04:46:36.090263 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083013 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 04:46:36.090263 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083016 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 04:46:36.090263 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083018 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 04:46:36.090263 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083020 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 04:46:36.090263 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083023 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 04:46:36.090263 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083025 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 04:46:36.090263 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083028 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 04:46:36.090263 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083030 2579 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 04:46:36.090263 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083033 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 04:46:36.090263 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083035 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 04:46:36.090263 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083038 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 04:46:36.090263 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083041 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 04:46:36.090848 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083045 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 04:46:36.090848 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083048 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 04:46:36.090848 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083051 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 04:46:36.090848 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083054 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 04:46:36.090848 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083058 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 04:46:36.090848 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083060 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 04:46:36.090848 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083063 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 04:46:36.090848 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083065 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 04:46:36.090848 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083068 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 04:46:36.090848 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083070 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 04:46:36.090848 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083073 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 04:46:36.090848 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083075 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 04:46:36.090848 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083078 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 04:46:36.090848 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083080 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 04:46:36.090848 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083083 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 04:46:36.090848 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083085 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 04:46:36.090848 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083088 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 04:46:36.090848 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083090 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 04:46:36.090848 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083093 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 04:46:36.090848 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083095 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 04:46:36.091650 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083098 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 04:46:36.091650 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083100 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 04:46:36.091650 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083102 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 04:46:36.091650 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083105 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 04:46:36.091650 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083107 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 04:46:36.091650 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083109 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 04:46:36.091650 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083112 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 04:46:36.091650 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083114 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 04:46:36.091650 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083116 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 04:46:36.091650 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083119 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 04:46:36.091650 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083121 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 04:46:36.091650 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083124 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 04:46:36.091650 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083126 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 04:46:36.091650 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083129 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 04:46:36.091650 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083131 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 04:46:36.091650 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083134 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 04:46:36.091650 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083136 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 04:46:36.091650 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083139 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 04:46:36.091650 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083141 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 04:46:36.092108 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083144 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 04:46:36.092108 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083147 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 04:46:36.092108 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083149 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 04:46:36.092108 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083151 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 04:46:36.092108 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083154 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 04:46:36.092108 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083158 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 04:46:36.092108 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083163 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 04:46:36.092108 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083166 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 04:46:36.092108 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083169 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 04:46:36.092108 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083171 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 04:46:36.092108 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083174 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 04:46:36.092108 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083176 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 04:46:36.092108 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083179 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 04:46:36.092108 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083181 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 04:46:36.092108 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083184 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 04:46:36.092108 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083186 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 04:46:36.092108 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083188 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 04:46:36.092108 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083190 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 04:46:36.092607 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.083217 2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 04 04:46:36.092607 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083410 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 04:46:36.092607 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083416 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 04:46:36.092607 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083419 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 04:46:36.092607 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083423 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 04:46:36.092607 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083426 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 04:46:36.092607 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083428 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 04:46:36.092607 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083431 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 04:46:36.092607 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083433 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 04:46:36.092607 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083436 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 04:46:36.092607 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083438 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 04:46:36.092607 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083441 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 04:46:36.092607 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083443 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 04:46:36.092607 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083446 2579 feature_gate.go:386] unrecognized feature gate: Example Nov 04 04:46:36.092607 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083448 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 04:46:36.092607 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083451 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 04:46:36.092966 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083453 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 04:46:36.092966 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083456 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 04:46:36.092966 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083459 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 04:46:36.092966 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083461 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 04:46:36.092966 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083464 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 04:46:36.092966 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083466 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 04:46:36.092966 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083468 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 04:46:36.092966 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083471 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 04:46:36.092966 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083473 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 04:46:36.092966 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083476 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 04:46:36.092966 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083478 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 04:46:36.092966 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083480 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 04:46:36.092966 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083483 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 04:46:36.092966 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083485 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 04:46:36.092966 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083488 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 04:46:36.092966 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083490 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 04:46:36.092966 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083493 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 04:46:36.092966 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083495 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 04:46:36.092966 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083497 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 04:46:36.092966 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083500 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 04:46:36.093802 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083503 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 04:46:36.093802 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083505 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 04:46:36.093802 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083507 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 04:46:36.093802 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083510 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 04:46:36.093802 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083512 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 04:46:36.093802 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083514 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 04:46:36.093802 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083517 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 04:46:36.093802 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083519 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 04:46:36.093802 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083522 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 04:46:36.093802 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083524 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 04:46:36.093802 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083527 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 04:46:36.093802 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083529 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 04:46:36.093802 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083532 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 04:46:36.093802 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083534 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 04:46:36.093802 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083537 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 04:46:36.093802 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083539 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 04:46:36.093802 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083541 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 04:46:36.093802 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083544 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 04:46:36.093802 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083546 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 04:46:36.094307 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083550 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 04:46:36.094307 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083554 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 04:46:36.094307 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083557 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 04:46:36.094307 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083560 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 04:46:36.094307 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083562 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 04:46:36.094307 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083565 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 04:46:36.094307 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083567 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 04:46:36.094307 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083570 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 04:46:36.094307 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083572 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 04:46:36.094307 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083574 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 04:46:36.094307 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083577 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 04:46:36.094307 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083579 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 04:46:36.094307 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083581 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 04:46:36.094307 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083584 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 04:46:36.094307 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083586 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 04:46:36.094307 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083588 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 04:46:36.094307 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083591 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 04:46:36.094307 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083593 2579 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 04:46:36.094307 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083595 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 04:46:36.094767 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083598 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 04:46:36.094767 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083600 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 04:46:36.094767 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083602 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 04:46:36.094767 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083605 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 04:46:36.094767 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083607 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 04:46:36.094767 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083613 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 04:46:36.094767 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083615 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 04:46:36.094767 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083618 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 04:46:36.094767 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083620 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 04:46:36.094767 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083623 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 04:46:36.094767 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083625 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 04:46:36.094767 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083628 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 04:46:36.094767 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083631 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 04:46:36.094767 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083633 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 04:46:36.094767 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083636 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 04:46:36.094767 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083638 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 04:46:36.094767 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083640 2579 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 04:46:36.094767 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083643 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 04:46:36.094767 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083645 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 04:46:36.094767 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083648 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 04:46:36.095266 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083650 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 04:46:36.095266 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083652 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 04:46:36.095266 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083655 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 04:46:36.095266 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083657 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 04:46:36.095266 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083659 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 04:46:36.095266 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083662 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 04:46:36.095266 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083665 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 04:46:36.095266 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083669 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 04:46:36.095266 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083671 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 04:46:36.095266 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083674 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 04:46:36.095266 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083677 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 04:46:36.095266 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083679 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 04:46:36.095266 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083682 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 04:46:36.095266 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.083706 2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 04 04:46:36.095266 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083739 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 04:46:36.095646 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083743 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 04:46:36.095646 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083748 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 04:46:36.095646 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083751 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 04:46:36.095646 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083755 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 04:46:36.095646 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083759 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 04:46:36.095646 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083761 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 04:46:36.095646 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083764 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 04:46:36.095646 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083766 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 04:46:36.095646 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083769 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 04:46:36.095646 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083771 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 04:46:36.095646 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083774 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 04:46:36.095646 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083776 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 04:46:36.095646 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083779 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 04:46:36.095646 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083781 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 04:46:36.095646 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083783 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 04:46:36.095646 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083786 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 04:46:36.095646 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083788 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 04:46:36.095646 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083791 2579 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 04:46:36.095646 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083793 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 04:46:36.096066 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083795 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 04:46:36.096066 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083798 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 04:46:36.096066 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083801 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 04:46:36.096066 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083803 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 04:46:36.096066 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083805 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 04:46:36.096066 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083808 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 04:46:36.096066 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083810 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 04:46:36.096066 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083813 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 04:46:36.096066 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083815 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 04:46:36.096066 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083817 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 04:46:36.096066 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083821 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 04:46:36.096066 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083823 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 04:46:36.096066 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083826 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 04:46:36.096066 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083828 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 04:46:36.096066 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083831 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 04:46:36.096066 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083833 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 04:46:36.096066 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083837 2579 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 04:46:36.096066 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083839 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 04:46:36.096066 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083842 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 04:46:36.096066 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083844 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 04:46:36.096562 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083847 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 04:46:36.096562 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083849 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 04:46:36.096562 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083852 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 04:46:36.096562 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083854 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 04:46:36.096562 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083857 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 04:46:36.096562 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083859 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 04:46:36.096562 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083864 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 04:46:36.096562 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083867 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 04:46:36.096562 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083871 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 04:46:36.096562 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083873 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 04:46:36.096562 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083876 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 04:46:36.096562 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083878 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 04:46:36.096562 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083881 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 04:46:36.096562 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083883 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 04:46:36.096562 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083885 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 04:46:36.096562 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083888 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 04:46:36.096562 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083890 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 04:46:36.096562 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083893 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 04:46:36.096971 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083895 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 04:46:36.096971 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083897 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 04:46:36.096971 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083900 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 04:46:36.096971 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083902 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 04:46:36.096971 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083905 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 04:46:36.096971 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083908 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 04:46:36.096971 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083910 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 04:46:36.096971 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083913 2579 feature_gate.go:386] unrecognized feature gate: Example Nov 04 04:46:36.096971 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083915 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 04:46:36.096971 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083918 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 04:46:36.096971 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083921 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 04:46:36.096971 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083923 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 04:46:36.096971 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083926 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 04:46:36.096971 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083929 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 04:46:36.096971 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083932 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 04:46:36.096971 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083934 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 04:46:36.096971 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083937 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 04:46:36.096971 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083939 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 04:46:36.096971 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083941 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 04:46:36.096971 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083944 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 04:46:36.097440 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083947 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 04:46:36.097440 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083949 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 04:46:36.097440 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083951 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 04:46:36.097440 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083954 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 04:46:36.097440 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083956 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 04:46:36.097440 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083959 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 04:46:36.097440 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083961 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 04:46:36.097440 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083964 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 04:46:36.097440 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083966 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 04:46:36.097440 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083968 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 04:46:36.097440 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083970 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 04:46:36.097440 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083973 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 04:46:36.097440 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083975 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 04:46:36.097440 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083978 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 04:46:36.097440 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083980 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 04:46:36.097440 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083983 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 04:46:36.097440 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083987 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 04:46:36.097440 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083990 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 04:46:36.097440 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083993 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 04:46:36.097440 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083995 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 04:46:36.097973 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.083997 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 04:46:36.097973 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.084000 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 04:46:36.097973 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.084004 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 04:46:36.097973 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.084006 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 04:46:36.097973 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.084009 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 04:46:36.097973 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.084012 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 04:46:36.097973 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.084014 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 04:46:36.097973 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:36.084018 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 04:46:36.097973 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.084068 2579 server.go:957] "Client rotation is on, will bootstrap in background" Nov 04 04:46:36.097973 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.085838 2579 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Nov 04 04:46:36.097973 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.088008 2579 server.go:1014] "Starting client certificate rotation" Nov 04 04:46:36.097973 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.088653 2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 04 04:46:36.097973 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.088711 2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 04 04:46:36.097973 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.090761 2579 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 04 04:46:36.097973 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.093550 2579 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 04 04:46:36.103958 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.103942 2579 log.go:25] "Validated CRI v1 runtime API" Nov 04 04:46:36.106961 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.106947 2579 log.go:25] "Validated CRI v1 image API" Nov 04 04:46:36.107802 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.107782 2579 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 04 04:46:36.112616 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.112589 2579 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 8284cc23-11ba-4c56-b47a-6f747cf2d934:/dev/nvme0n1p3 c3d7e991-8daa-45be-bbbc-3a25fd29be14:/dev/nvme0n1p4] Nov 04 04:46:36.112678 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.112615 2579 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Nov 04 04:46:36.117722 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.117616 2579 manager.go:217] Machine: {Timestamp:2026-11-04 04:46:36.116287883 +0000 UTC m=+0.331535812 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3147268 MemoryCapacity:32812019712 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec28fd50b4bd208995dd578b7a38fd2c SystemUUID:ec28fd50-b4bd-2089-95dd-578b7a38fd2c BootID:667e3c15-2317-4bec-8bd3-7357e4534b20 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406011904 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:5a:64:b6:d5:bb Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:5a:64:b6:d5:bb Speed:0 Mtu:9001} {Name:ovs-system MacAddress:16:8b:a8:fa:d3:2d Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812019712 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 04 04:46:36.117722 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.117722 2579 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 04 04:46:36.117863 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.117805 2579 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 04 04:46:36.120549 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.120529 2579 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 04 04:46:36.121238 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.120551 2579 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-180.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 04 04:46:36.121284 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.121242 2579 topology_manager.go:143] "Creating topology manager with none policy" Nov 04 04:46:36.121284 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.121251 2579 container_manager_linux.go:311] "Creating device plugin manager" Nov 04 04:46:36.121284 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.121280 2579 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 04 04:46:36.121377 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.121299 2579 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 04 04:46:36.122050 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.122040 2579 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 04 04:46:36.125403 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.125392 2579 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 04 04:46:36.125524 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.125514 2579 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 04 04:46:36.126227 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.126216 2579 kubelet.go:493] "Attempting to sync node with API server" Nov 04 04:46:36.126263 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.126234 2579 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 04 04:46:36.126263 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.126253 2579 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 04 04:46:36.126312 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.126267 2579 kubelet.go:405] "Adding apiserver pod source" Nov 04 04:46:36.126312 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.126278 2579 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 04 04:46:36.126724 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.126712 2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 04 04:46:36.129720 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.129694 2579 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 04 04:46:36.134459 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.134444 2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 04 04:46:36.135215 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.135203 2579 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 04 04:46:36.135252 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.135233 2579 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 04 04:46:36.135400 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.135390 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 04 04:46:36.135437 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.135405 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 04 04:46:36.135437 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.135411 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 04 04:46:36.135437 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.135417 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 04 04:46:36.135437 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.135423 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 04 04:46:36.135437 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.135428 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 04 04:46:36.136291 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.136283 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 04 04:46:36.136335 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.136292 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 04 04:46:36.136335 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.136299 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 04 04:46:36.136335 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.136304 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 04 04:46:36.136335 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.136314 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 04 04:46:36.136335 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.136334 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 04 04:46:36.136979 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.136970 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 04 04:46:36.136979 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.136979 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 04 04:46:36.140806 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.140793 2579 server.go:1263] "Started kubelet" Nov 04 04:46:36.140912 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.140867 2579 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 04 04:46:36.140971 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.140876 2579 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 04 04:46:36.140971 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.140950 2579 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 04 04:46:36.141515 ip-10-0-133-180 systemd[1]: Started Kubernetes Kubelet. Nov 04 04:46:36.142190 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.142174 2579 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 04 04:46:36.143057 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.143038 2579 server.go:317] "Adding debug handlers to kubelet server" Nov 04 04:46:36.147301 ip-10-0-133-180 kubenswrapper[2579]: E1104 04:46:36.147277 2579 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Nov 04 04:46:36.147387 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.147307 2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 04 04:46:36.148131 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.148118 2579 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 04 04:46:36.148244 ip-10-0-133-180 kubenswrapper[2579]: E1104 04:46:36.148222 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-180.us-west-1.compute.internal\" not found" Nov 04 04:46:36.148619 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.148599 2579 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 04 04:46:36.148619 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.148602 2579 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 04 04:46:36.148757 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.148628 2579 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 04 04:46:36.148757 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.148724 2579 reconstruct.go:97] "Volume reconstruction finished" Nov 04 04:46:36.148757 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.148733 2579 reconciler.go:29] "Reconciler: start to sync state" Nov 04 04:46:36.148947 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.148935 2579 factory.go:55] Registering systemd factory Nov 04 04:46:36.148979 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.148964 2579 factory.go:223] Registration of the systemd container factory successfully Nov 04 04:46:36.149134 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.149126 2579 factory.go:153] Registering CRI-O factory Nov 04 04:46:36.149167 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.149136 2579 factory.go:223] Registration of the crio container factory successfully Nov 04 04:46:36.149192 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.149176 2579 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 04 04:46:36.149192 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.149189 2579 factory.go:103] Registering Raw factory Nov 04 04:46:36.149246 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.149199 2579 manager.go:1199] Started watching for new ooms in manager Nov 04 04:46:36.149938 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.149928 2579 manager.go:319] Starting recovery of all containers Nov 04 04:46:36.159962 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.159893 2579 manager.go:324] Recovery completed Nov 04 04:46:36.161859 ip-10-0-133-180 kubenswrapper[2579]: E1104 04:46:36.161790 2579 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory Nov 04 04:46:36.165310 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.165298 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 04:46:36.166961 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.166945 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-180.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 04 04:46:36.167027 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.166974 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-180.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 04 04:46:36.167027 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.166984 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-180.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 04 04:46:36.167473 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.167460 2579 cpu_manager.go:226] "Starting" policy="none" Nov 04 04:46:36.167473 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.167467 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 04:46:36.167562 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.167469 2579 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 04 04:46:36.167562 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.167524 2579 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 04 04:46:36.168113 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.168100 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-180.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 04 04:46:36.168144 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.168127 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-180.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 04 04:46:36.168144 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.168139 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-180.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 04 04:46:36.169672 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.169659 2579 policy_none.go:50] "Start" Nov 04 04:46:36.169672 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.169674 2579 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 04 04:46:36.169770 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.169684 2579 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 04 04:46:36.171738 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.171726 2579 policy_none.go:44] "Start" Nov 04 04:46:36.207942 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.207910 2579 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 04 04:46:36.210110 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.210096 2579 manager.go:341] "Starting Device Plugin manager" Nov 04 04:46:36.210189 ip-10-0-133-180 kubenswrapper[2579]: E1104 04:46:36.210129 2579 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 04 04:46:36.210189 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.210140 2579 server.go:87] "Starting device plugin registration server" Nov 04 04:46:36.210413 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.210399 2579 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 04 04:46:36.210485 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.210416 2579 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 04 04:46:36.210542 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.210524 2579 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 04 04:46:36.210628 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.210616 2579 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 04 04:46:36.210679 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.210629 2579 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 04 04:46:36.211148 ip-10-0-133-180 kubenswrapper[2579]: E1104 04:46:36.211124 2579 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 04 04:46:36.211217 ip-10-0-133-180 kubenswrapper[2579]: E1104 04:46:36.211174 2579 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-180.us-west-1.compute.internal\" not found" Nov 04 04:46:36.287550 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.287519 2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 04 04:46:36.289490 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.288792 2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 04 04:46:36.289490 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.288814 2579 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 04 04:46:36.289490 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.288834 2579 kubelet.go:2521] "Starting kubelet main sync loop" Nov 04 04:46:36.289490 ip-10-0-133-180 kubenswrapper[2579]: E1104 04:46:36.288867 2579 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Nov 04 04:46:36.310863 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.310844 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 04:46:36.311667 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.311648 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-180.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 04 04:46:36.311751 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.311686 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-180.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 04 04:46:36.311751 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.311705 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-180.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 04 04:46:36.311751 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.311729 2579 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-133-180.us-west-1.compute.internal" Nov 04 04:46:36.372192 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.372111 2579 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 04 04:46:36.372192 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.372180 2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-kp9gd" Nov 04 04:46:36.389333 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.389284 2579 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-180.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-180.us-west-1.compute.internal"] Nov 04 04:46:36.389489 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.389365 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 04:46:36.390343 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.390305 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-180.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 04 04:46:36.390428 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.390357 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-180.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 04 04:46:36.390428 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.390370 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-180.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 04 04:46:36.392421 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.392408 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 04:46:36.392581 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.392566 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-180.us-west-1.compute.internal" Nov 04 04:46:36.392614 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.392598 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 04:46:36.393151 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.393130 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-180.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 04 04:46:36.393151 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.393140 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-180.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 04 04:46:36.393284 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.393165 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-180.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 04 04:46:36.393284 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.393178 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-180.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 04 04:46:36.393284 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.393166 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-180.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 04 04:46:36.393284 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.393250 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-180.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 04 04:46:36.395218 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.395201 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-180.us-west-1.compute.internal" Nov 04 04:46:36.395308 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.395232 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 04:46:36.395986 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.395970 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-180.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 04 04:46:36.396042 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.396005 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-180.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 04 04:46:36.396042 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.396023 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-180.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 04 04:46:36.423289 ip-10-0-133-180 kubenswrapper[2579]: E1104 04:46:36.423266 2579 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-180.us-west-1.compute.internal\" not found" node="ip-10-0-133-180.us-west-1.compute.internal" Nov 04 04:46:36.425399 ip-10-0-133-180 kubenswrapper[2579]: E1104 04:46:36.425381 2579 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-180.us-west-1.compute.internal\" not found" node="ip-10-0-133-180.us-west-1.compute.internal" Nov 04 04:46:36.441098 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.441077 2579 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 04 04:46:36.450454 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.450398 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/46addbc2e24e99ce85548c61cc2707a9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-180.us-west-1.compute.internal\" (UID: \"46addbc2e24e99ce85548c61cc2707a9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-180.us-west-1.compute.internal" Nov 04 04:46:36.450574 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.450464 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/46addbc2e24e99ce85548c61cc2707a9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-180.us-west-1.compute.internal\" (UID: \"46addbc2e24e99ce85548c61cc2707a9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-180.us-west-1.compute.internal" Nov 04 04:46:36.450574 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.450508 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6aa9201e961e4ed5a49af4c50453ef92-config\") pod \"kube-apiserver-proxy-ip-10-0-133-180.us-west-1.compute.internal\" (UID: \"6aa9201e961e4ed5a49af4c50453ef92\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-180.us-west-1.compute.internal" Nov 04 04:46:36.490058 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.489885 2579 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 04 04:46:36.491006 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.490986 2579 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 04 04:46:36.540153 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.540123 2579 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 04 04:46:36.540539 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.540520 2579 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-133-180.us-west-1.compute.internal" Nov 04 04:46:36.548688 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.548668 2579 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-180.us-west-1.compute.internal" Nov 04 04:46:36.550889 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.550870 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/46addbc2e24e99ce85548c61cc2707a9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-180.us-west-1.compute.internal\" (UID: \"46addbc2e24e99ce85548c61cc2707a9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-180.us-west-1.compute.internal" Nov 04 04:46:36.550955 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.550897 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/46addbc2e24e99ce85548c61cc2707a9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-180.us-west-1.compute.internal\" (UID: \"46addbc2e24e99ce85548c61cc2707a9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-180.us-west-1.compute.internal" Nov 04 04:46:36.550955 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.550928 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6aa9201e961e4ed5a49af4c50453ef92-config\") pod \"kube-apiserver-proxy-ip-10-0-133-180.us-west-1.compute.internal\" (UID: \"6aa9201e961e4ed5a49af4c50453ef92\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-180.us-west-1.compute.internal" Nov 04 04:46:36.551022 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.550990 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/46addbc2e24e99ce85548c61cc2707a9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-180.us-west-1.compute.internal\" (UID: \"46addbc2e24e99ce85548c61cc2707a9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-180.us-west-1.compute.internal" Nov 04 04:46:36.551022 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.550995 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6aa9201e961e4ed5a49af4c50453ef92-config\") pod \"kube-apiserver-proxy-ip-10-0-133-180.us-west-1.compute.internal\" (UID: \"6aa9201e961e4ed5a49af4c50453ef92\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-180.us-west-1.compute.internal" Nov 04 04:46:36.551022 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.550995 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/46addbc2e24e99ce85548c61cc2707a9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-180.us-west-1.compute.internal\" (UID: \"46addbc2e24e99ce85548c61cc2707a9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-180.us-west-1.compute.internal" Nov 04 04:46:36.611243 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.611212 2579 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 04 04:46:36.611377 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.611338 2579 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-180.us-west-1.compute.internal" Nov 04 04:46:36.668576 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.668504 2579 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 04 04:46:36.726067 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.726036 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-180.us-west-1.compute.internal" Nov 04 04:46:36.728173 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:36.728156 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-180.us-west-1.compute.internal" Nov 04 04:46:37.087918 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.087830 2579 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Nov 04 04:46:37.088482 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.088020 2579 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 04 04:46:37.088482 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.088040 2579 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 04 04:46:37.088482 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.088111 2579 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 04 04:46:37.088482 ip-10-0-133-180 kubenswrapper[2579]: E1104 04:46:37.088031 2579 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://ad7f0a69d1aeb4fbea22a6fe43d5cdd1-3a8265abc582d153.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-133-180.us-west-1.compute.internal.18e4be1ffb239b3c\": read tcp 10.0.133.180:55188->3.132.101.194:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-133-180.us-west-1.compute.internal.18e4be1ffb239b3c default 5167 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-180.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-133-180.us-west-1.compute.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-133-180.us-west-1.compute.internal,},FirstTimestamp:2026-11-04 04:46:36 +0000 UTC,LastTimestamp:2026-11-04 04:46:36.311693805 +0000 UTC m=+0.526941735,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-180.us-west-1.compute.internal,}" Nov 04 04:46:37.126993 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.126962 2579 apiserver.go:52] "Watching apiserver" Nov 04 04:46:37.147862 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.147837 2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Nov 04 04:46:37.376828 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.376737 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-03 04:41:37 +0000 UTC" deadline="2028-07-01 11:06:49.013334749 +0000 UTC" Nov 04 04:46:37.376828 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.376772 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14526h20m11.636565778s" Nov 04 04:46:37.392854 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.392830 2579 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 04 04:46:37.393157 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.393138 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-180.us-west-1.compute.internal","openshift-multus/multus-additional-cni-plugins-cplsq","kube-system/konnectivity-agent-7tw97","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f2zrp","openshift-network-operator/iptables-alerter-wdk7z","openshift-ovn-kubernetes/ovnkube-node-dvc6z","openshift-cluster-node-tuning-operator/tuned-c65v8","openshift-multus/multus-7m8fc","openshift-multus/network-metrics-daemon-25flt","kube-system/kube-apiserver-proxy-ip-10-0-133-180.us-west-1.compute.internal","openshift-network-diagnostics/network-check-target-8hsp8","kube-system/global-pull-secret-syncer-gfjqz","openshift-dns/node-resolver-gzctm","openshift-image-registry/node-ca-4z89d"] Nov 04 04:46:37.395597 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.395580 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-cplsq" Nov 04 04:46:37.397560 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.397538 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-7tw97" Nov 04 04:46:37.399574 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.399559 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f2zrp" Nov 04 04:46:37.401603 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.401588 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-wdk7z" Nov 04 04:46:37.403798 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.403784 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-dvc6z" Nov 04 04:46:37.405782 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.405756 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-c65v8" Nov 04 04:46:37.407759 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.407745 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-7m8fc" Nov 04 04:46:37.409994 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.409980 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-25flt" Nov 04 04:46:37.410068 ip-10-0-133-180 kubenswrapper[2579]: E1104 04:46:37.410051 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-25flt" podUID="4dff9faf-d91f-4ed5-be82-555b9a6a924e" Nov 04 04:46:37.412180 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.412166 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8hsp8" Nov 04 04:46:37.412309 ip-10-0-133-180 kubenswrapper[2579]: E1104 04:46:37.412226 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8hsp8" podUID="5f6922fa-9ba7-4d02-b550-606731852fde" Nov 04 04:46:37.414520 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.414502 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gfjqz" Nov 04 04:46:37.414607 ip-10-0-133-180 kubenswrapper[2579]: E1104 04:46:37.414564 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gfjqz" podUID="c79eff94-6543-4313-b59b-502ca4089e84" Nov 04 04:46:37.416652 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.416638 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-gzctm" Nov 04 04:46:37.418672 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.418657 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-4z89d" Nov 04 04:46:37.448257 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.448238 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 04 04:46:37.448472 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.448236 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 04 04:46:37.448554 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.448238 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 04 04:46:37.448554 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.448245 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 04 04:46:37.448554 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.448257 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 04 04:46:37.448554 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.448263 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-ssxrh\"" Nov 04 04:46:37.449444 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.449426 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-jqdfn\"" Nov 04 04:46:37.449559 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.449455 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 04 04:46:37.449920 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.449908 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 04 04:46:37.450268 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.450249 2579 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 04 04:46:37.454705 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.454684 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/844cc12f-a7f6-4327-968e-5e9b2da00ad0-var-lib-openvswitch\") pod \"ovnkube-node-dvc6z\" (UID: \"844cc12f-a7f6-4327-968e-5e9b2da00ad0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvc6z" Nov 04 04:46:37.454792 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.454718 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/844cc12f-a7f6-4327-968e-5e9b2da00ad0-log-socket\") pod \"ovnkube-node-dvc6z\" (UID: \"844cc12f-a7f6-4327-968e-5e9b2da00ad0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvc6z" Nov 04 04:46:37.454792 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.454739 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/58be7092-9156-4787-8f49-a173fa0bd5a3-host-var-lib-cni-bin\") pod \"multus-7m8fc\" (UID: \"58be7092-9156-4787-8f49-a173fa0bd5a3\") " pod="openshift-multus/multus-7m8fc" Nov 04 04:46:37.454882 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.454803 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8446aa71-5638-4b23-8f5b-12283ede21d4-tmp-dir\") pod \"node-resolver-gzctm\" (UID: \"8446aa71-5638-4b23-8f5b-12283ede21d4\") " pod="openshift-dns/node-resolver-gzctm" Nov 04 04:46:37.454882 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.454835 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nch9\" (UniqueName: \"kubernetes.io/projected/844cc12f-a7f6-4327-968e-5e9b2da00ad0-kube-api-access-6nch9\") pod \"ovnkube-node-dvc6z\" (UID: \"844cc12f-a7f6-4327-968e-5e9b2da00ad0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvc6z" Nov 04 04:46:37.454882 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.454852 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/22d31d0c-6edf-4351-8f41-a6dd4e02368b-etc-kubernetes\") pod \"tuned-c65v8\" (UID: \"22d31d0c-6edf-4351-8f41-a6dd4e02368b\") " pod="openshift-cluster-node-tuning-operator/tuned-c65v8" Nov 04 04:46:37.455003 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.454927 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/22d31d0c-6edf-4351-8f41-a6dd4e02368b-etc-systemd\") pod \"tuned-c65v8\" (UID: \"22d31d0c-6edf-4351-8f41-a6dd4e02368b\") " pod="openshift-cluster-node-tuning-operator/tuned-c65v8" Nov 04 04:46:37.455003 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.454979 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/198e9ab0-f615-4686-9acf-a6242f7e3584-cni-binary-copy\") pod \"multus-additional-cni-plugins-cplsq\" (UID: \"198e9ab0-f615-4686-9acf-a6242f7e3584\") " pod="openshift-multus/multus-additional-cni-plugins-cplsq" Nov 04 04:46:37.455098 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.455019 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/844cc12f-a7f6-4327-968e-5e9b2da00ad0-host-cni-netd\") pod \"ovnkube-node-dvc6z\" (UID: \"844cc12f-a7f6-4327-968e-5e9b2da00ad0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvc6z" Nov 04 04:46:37.455098 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.455042 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/844cc12f-a7f6-4327-968e-5e9b2da00ad0-ovnkube-config\") pod \"ovnkube-node-dvc6z\" (UID: \"844cc12f-a7f6-4327-968e-5e9b2da00ad0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvc6z" Nov 04 04:46:37.455098 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.455088 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/22d31d0c-6edf-4351-8f41-a6dd4e02368b-etc-sysctl-conf\") pod \"tuned-c65v8\" (UID: \"22d31d0c-6edf-4351-8f41-a6dd4e02368b\") " pod="openshift-cluster-node-tuning-operator/tuned-c65v8" Nov 04 04:46:37.455236 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.455107 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/22d31d0c-6edf-4351-8f41-a6dd4e02368b-run\") pod \"tuned-c65v8\" (UID: \"22d31d0c-6edf-4351-8f41-a6dd4e02368b\") " pod="openshift-cluster-node-tuning-operator/tuned-c65v8" Nov 04 04:46:37.455236 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.455130 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/22d31d0c-6edf-4351-8f41-a6dd4e02368b-etc-tuned\") pod \"tuned-c65v8\" (UID: \"22d31d0c-6edf-4351-8f41-a6dd4e02368b\") " pod="openshift-cluster-node-tuning-operator/tuned-c65v8" Nov 04 04:46:37.455236 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.455154 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/58be7092-9156-4787-8f49-a173fa0bd5a3-cni-binary-copy\") pod \"multus-7m8fc\" (UID: \"58be7092-9156-4787-8f49-a173fa0bd5a3\") " pod="openshift-multus/multus-7m8fc" Nov 04 04:46:37.455236 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.455159 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 04 04:46:37.455236 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.455202 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/58be7092-9156-4787-8f49-a173fa0bd5a3-hostroot\") pod \"multus-7m8fc\" (UID: \"58be7092-9156-4787-8f49-a173fa0bd5a3\") " pod="openshift-multus/multus-7m8fc" Nov 04 04:46:37.455236 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.455225 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9a06ec07-c72b-43af-aa27-7414aa9c9aba-host\") pod \"node-ca-4z89d\" (UID: \"9a06ec07-c72b-43af-aa27-7414aa9c9aba\") " pod="openshift-image-registry/node-ca-4z89d" Nov 04 04:46:37.455549 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.455250 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/00e0471e-5fe8-4494-8f17-632aee4f6c30-socket-dir\") pod \"aws-ebs-csi-driver-node-f2zrp\" (UID: \"00e0471e-5fe8-4494-8f17-632aee4f6c30\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f2zrp" Nov 04 04:46:37.455549 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.455272 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/00e0471e-5fe8-4494-8f17-632aee4f6c30-sys-fs\") pod \"aws-ebs-csi-driver-node-f2zrp\" (UID: \"00e0471e-5fe8-4494-8f17-632aee4f6c30\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f2zrp" Nov 04 04:46:37.455549 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.455296 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/844cc12f-a7f6-4327-968e-5e9b2da00ad0-run-openvswitch\") pod \"ovnkube-node-dvc6z\" (UID: \"844cc12f-a7f6-4327-968e-5e9b2da00ad0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvc6z" Nov 04 04:46:37.455549 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.455371 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9a06ec07-c72b-43af-aa27-7414aa9c9aba-serviceca\") pod \"node-ca-4z89d\" (UID: \"9a06ec07-c72b-43af-aa27-7414aa9c9aba\") " pod="openshift-image-registry/node-ca-4z89d" Nov 04 04:46:37.455549 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.455397 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/58be7092-9156-4787-8f49-a173fa0bd5a3-host-run-netns\") pod \"multus-7m8fc\" (UID: \"58be7092-9156-4787-8f49-a173fa0bd5a3\") " pod="openshift-multus/multus-7m8fc" Nov 04 04:46:37.455549 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.455423 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hx8dr\" (UniqueName: \"kubernetes.io/projected/4dff9faf-d91f-4ed5-be82-555b9a6a924e-kube-api-access-hx8dr\") pod \"network-metrics-daemon-25flt\" (UID: \"4dff9faf-d91f-4ed5-be82-555b9a6a924e\") " pod="openshift-multus/network-metrics-daemon-25flt" Nov 04 04:46:37.455549 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.455448 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c79eff94-6543-4313-b59b-502ca4089e84-dbus\") pod \"global-pull-secret-syncer-gfjqz\" (UID: \"c79eff94-6543-4313-b59b-502ca4089e84\") " pod="kube-system/global-pull-secret-syncer-gfjqz" Nov 04 04:46:37.455549 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.455478 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/198e9ab0-f615-4686-9acf-a6242f7e3584-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cplsq\" (UID: \"198e9ab0-f615-4686-9acf-a6242f7e3584\") " pod="openshift-multus/multus-additional-cni-plugins-cplsq" Nov 04 04:46:37.455549 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.455503 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/00e0471e-5fe8-4494-8f17-632aee4f6c30-device-dir\") pod \"aws-ebs-csi-driver-node-f2zrp\" (UID: \"00e0471e-5fe8-4494-8f17-632aee4f6c30\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f2zrp" Nov 04 04:46:37.455549 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.455525 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/00e0471e-5fe8-4494-8f17-632aee4f6c30-etc-selinux\") pod \"aws-ebs-csi-driver-node-f2zrp\" (UID: \"00e0471e-5fe8-4494-8f17-632aee4f6c30\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f2zrp" Nov 04 04:46:37.455549 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.455546 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/58be7092-9156-4787-8f49-a173fa0bd5a3-host-var-lib-kubelet\") pod \"multus-7m8fc\" (UID: \"58be7092-9156-4787-8f49-a173fa0bd5a3\") " pod="openshift-multus/multus-7m8fc" Nov 04 04:46:37.456052 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.455561 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 04 04:46:37.456052 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.455573 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c79eff94-6543-4313-b59b-502ca4089e84-kubelet-config\") pod \"global-pull-secret-syncer-gfjqz\" (UID: \"c79eff94-6543-4313-b59b-502ca4089e84\") " pod="kube-system/global-pull-secret-syncer-gfjqz" Nov 04 04:46:37.456052 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.455613 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 04 04:46:37.456052 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.455625 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c79eff94-6543-4313-b59b-502ca4089e84-original-pull-secret\") pod \"global-pull-secret-syncer-gfjqz\" (UID: \"c79eff94-6543-4313-b59b-502ca4089e84\") " pod="kube-system/global-pull-secret-syncer-gfjqz" Nov 04 04:46:37.456052 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.455647 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 04 04:46:37.456052 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.455649 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kz2hd\" (UniqueName: \"kubernetes.io/projected/d737ccc2-5686-4130-9d00-d2f21d5b417f-kube-api-access-kz2hd\") pod \"iptables-alerter-wdk7z\" (UID: \"d737ccc2-5686-4130-9d00-d2f21d5b417f\") " pod="openshift-network-operator/iptables-alerter-wdk7z" Nov 04 04:46:37.456052 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.455673 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/844cc12f-a7f6-4327-968e-5e9b2da00ad0-systemd-units\") pod \"ovnkube-node-dvc6z\" (UID: \"844cc12f-a7f6-4327-968e-5e9b2da00ad0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvc6z" Nov 04 04:46:37.456052 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.455697 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/844cc12f-a7f6-4327-968e-5e9b2da00ad0-host-run-netns\") pod \"ovnkube-node-dvc6z\" (UID: \"844cc12f-a7f6-4327-968e-5e9b2da00ad0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvc6z" Nov 04 04:46:37.456052 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.455705 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 04 04:46:37.456052 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.455736 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 04 04:46:37.456052 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.455796 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/22d31d0c-6edf-4351-8f41-a6dd4e02368b-etc-sysctl-d\") pod \"tuned-c65v8\" (UID: \"22d31d0c-6edf-4351-8f41-a6dd4e02368b\") " pod="openshift-cluster-node-tuning-operator/tuned-c65v8" Nov 04 04:46:37.456052 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.455820 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/58be7092-9156-4787-8f49-a173fa0bd5a3-system-cni-dir\") pod \"multus-7m8fc\" (UID: \"58be7092-9156-4787-8f49-a173fa0bd5a3\") " pod="openshift-multus/multus-7m8fc" Nov 04 04:46:37.456052 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.455828 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-g79lj\"" Nov 04 04:46:37.456052 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.455841 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/58be7092-9156-4787-8f49-a173fa0bd5a3-cnibin\") pod \"multus-7m8fc\" (UID: \"58be7092-9156-4787-8f49-a173fa0bd5a3\") " pod="openshift-multus/multus-7m8fc" Nov 04 04:46:37.456052 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.455865 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/58be7092-9156-4787-8f49-a173fa0bd5a3-multus-conf-dir\") pod \"multus-7m8fc\" (UID: \"58be7092-9156-4787-8f49-a173fa0bd5a3\") " pod="openshift-multus/multus-7m8fc" Nov 04 04:46:37.456052 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.455897 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/22d31d0c-6edf-4351-8f41-a6dd4e02368b-lib-modules\") pod \"tuned-c65v8\" (UID: \"22d31d0c-6edf-4351-8f41-a6dd4e02368b\") " pod="openshift-cluster-node-tuning-operator/tuned-c65v8" Nov 04 04:46:37.456052 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.455961 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/58be7092-9156-4787-8f49-a173fa0bd5a3-host-run-k8s-cni-cncf-io\") pod \"multus-7m8fc\" (UID: \"58be7092-9156-4787-8f49-a173fa0bd5a3\") " pod="openshift-multus/multus-7m8fc" Nov 04 04:46:37.456052 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.455989 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/844cc12f-a7f6-4327-968e-5e9b2da00ad0-host-slash\") pod \"ovnkube-node-dvc6z\" (UID: \"844cc12f-a7f6-4327-968e-5e9b2da00ad0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvc6z" Nov 04 04:46:37.456052 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.456025 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/844cc12f-a7f6-4327-968e-5e9b2da00ad0-node-log\") pod \"ovnkube-node-dvc6z\" (UID: \"844cc12f-a7f6-4327-968e-5e9b2da00ad0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvc6z" Nov 04 04:46:37.456052 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.456048 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/844cc12f-a7f6-4327-968e-5e9b2da00ad0-env-overrides\") pod \"ovnkube-node-dvc6z\" (UID: \"844cc12f-a7f6-4327-968e-5e9b2da00ad0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvc6z" Nov 04 04:46:37.456973 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.456071 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/58be7092-9156-4787-8f49-a173fa0bd5a3-multus-daemon-config\") pod \"multus-7m8fc\" (UID: \"58be7092-9156-4787-8f49-a173fa0bd5a3\") " pod="openshift-multus/multus-7m8fc" Nov 04 04:46:37.456973 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.456100 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d737ccc2-5686-4130-9d00-d2f21d5b417f-iptables-alerter-script\") pod \"iptables-alerter-wdk7z\" (UID: \"d737ccc2-5686-4130-9d00-d2f21d5b417f\") " pod="openshift-network-operator/iptables-alerter-wdk7z" Nov 04 04:46:37.456973 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.456154 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/844cc12f-a7f6-4327-968e-5e9b2da00ad0-host-run-ovn-kubernetes\") pod \"ovnkube-node-dvc6z\" (UID: \"844cc12f-a7f6-4327-968e-5e9b2da00ad0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvc6z" Nov 04 04:46:37.456973 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.456176 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/844cc12f-a7f6-4327-968e-5e9b2da00ad0-host-cni-bin\") pod \"ovnkube-node-dvc6z\" (UID: \"844cc12f-a7f6-4327-968e-5e9b2da00ad0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvc6z" Nov 04 04:46:37.456973 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.456206 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/58be7092-9156-4787-8f49-a173fa0bd5a3-multus-socket-dir-parent\") pod \"multus-7m8fc\" (UID: \"58be7092-9156-4787-8f49-a173fa0bd5a3\") " pod="openshift-multus/multus-7m8fc" Nov 04 04:46:37.456973 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.456231 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/23ed4a39-18b5-4bc6-9c7b-8b60a7043ad2-konnectivity-ca\") pod \"konnectivity-agent-7tw97\" (UID: \"23ed4a39-18b5-4bc6-9c7b-8b60a7043ad2\") " pod="kube-system/konnectivity-agent-7tw97" Nov 04 04:46:37.456973 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.456255 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/844cc12f-a7f6-4327-968e-5e9b2da00ad0-ovn-node-metrics-cert\") pod \"ovnkube-node-dvc6z\" (UID: \"844cc12f-a7f6-4327-968e-5e9b2da00ad0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvc6z" Nov 04 04:46:37.456973 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.456279 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbtxf\" (UniqueName: \"kubernetes.io/projected/22d31d0c-6edf-4351-8f41-a6dd4e02368b-kube-api-access-zbtxf\") pod \"tuned-c65v8\" (UID: \"22d31d0c-6edf-4351-8f41-a6dd4e02368b\") " pod="openshift-cluster-node-tuning-operator/tuned-c65v8" Nov 04 04:46:37.456973 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.456303 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/58be7092-9156-4787-8f49-a173fa0bd5a3-etc-kubernetes\") pod \"multus-7m8fc\" (UID: \"58be7092-9156-4787-8f49-a173fa0bd5a3\") " pod="openshift-multus/multus-7m8fc" Nov 04 04:46:37.456973 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.456348 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/22d31d0c-6edf-4351-8f41-a6dd4e02368b-etc-modprobe-d\") pod \"tuned-c65v8\" (UID: \"22d31d0c-6edf-4351-8f41-a6dd4e02368b\") " pod="openshift-cluster-node-tuning-operator/tuned-c65v8" Nov 04 04:46:37.456973 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.456420 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlz69\" (UniqueName: \"kubernetes.io/projected/9a06ec07-c72b-43af-aa27-7414aa9c9aba-kube-api-access-mlz69\") pod \"node-ca-4z89d\" (UID: \"9a06ec07-c72b-43af-aa27-7414aa9c9aba\") " pod="openshift-image-registry/node-ca-4z89d" Nov 04 04:46:37.456973 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.456444 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/58be7092-9156-4787-8f49-a173fa0bd5a3-multus-cni-dir\") pod \"multus-7m8fc\" (UID: \"58be7092-9156-4787-8f49-a173fa0bd5a3\") " pod="openshift-multus/multus-7m8fc" Nov 04 04:46:37.456973 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.456468 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/58be7092-9156-4787-8f49-a173fa0bd5a3-host-run-multus-certs\") pod \"multus-7m8fc\" (UID: \"58be7092-9156-4787-8f49-a173fa0bd5a3\") " pod="openshift-multus/multus-7m8fc" Nov 04 04:46:37.456973 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.456492 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25g6k\" (UniqueName: \"kubernetes.io/projected/8446aa71-5638-4b23-8f5b-12283ede21d4-kube-api-access-25g6k\") pod \"node-resolver-gzctm\" (UID: \"8446aa71-5638-4b23-8f5b-12283ede21d4\") " pod="openshift-dns/node-resolver-gzctm" Nov 04 04:46:37.456973 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.456541 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/198e9ab0-f615-4686-9acf-a6242f7e3584-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cplsq\" (UID: \"198e9ab0-f615-4686-9acf-a6242f7e3584\") " pod="openshift-multus/multus-additional-cni-plugins-cplsq" Nov 04 04:46:37.456973 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.456612 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/198e9ab0-f615-4686-9acf-a6242f7e3584-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cplsq\" (UID: \"198e9ab0-f615-4686-9acf-a6242f7e3584\") " pod="openshift-multus/multus-additional-cni-plugins-cplsq" Nov 04 04:46:37.457697 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.456643 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/844cc12f-a7f6-4327-968e-5e9b2da00ad0-host-kubelet\") pod \"ovnkube-node-dvc6z\" (UID: \"844cc12f-a7f6-4327-968e-5e9b2da00ad0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvc6z" Nov 04 04:46:37.457697 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.456669 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/844cc12f-a7f6-4327-968e-5e9b2da00ad0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dvc6z\" (UID: \"844cc12f-a7f6-4327-968e-5e9b2da00ad0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvc6z" Nov 04 04:46:37.457697 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.456714 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/22d31d0c-6edf-4351-8f41-a6dd4e02368b-sys\") pod \"tuned-c65v8\" (UID: \"22d31d0c-6edf-4351-8f41-a6dd4e02368b\") " pod="openshift-cluster-node-tuning-operator/tuned-c65v8" Nov 04 04:46:37.457697 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.456740 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/58be7092-9156-4787-8f49-a173fa0bd5a3-host-var-lib-cni-multus\") pod \"multus-7m8fc\" (UID: \"58be7092-9156-4787-8f49-a173fa0bd5a3\") " pod="openshift-multus/multus-7m8fc" Nov 04 04:46:37.457697 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.456765 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4xvx\" (UniqueName: \"kubernetes.io/projected/58be7092-9156-4787-8f49-a173fa0bd5a3-kube-api-access-w4xvx\") pod \"multus-7m8fc\" (UID: \"58be7092-9156-4787-8f49-a173fa0bd5a3\") " pod="openshift-multus/multus-7m8fc" Nov 04 04:46:37.457697 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.456789 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/198e9ab0-f615-4686-9acf-a6242f7e3584-cnibin\") pod \"multus-additional-cni-plugins-cplsq\" (UID: \"198e9ab0-f615-4686-9acf-a6242f7e3584\") " pod="openshift-multus/multus-additional-cni-plugins-cplsq" Nov 04 04:46:37.457697 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.456837 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccfxs\" (UniqueName: \"kubernetes.io/projected/198e9ab0-f615-4686-9acf-a6242f7e3584-kube-api-access-ccfxs\") pod \"multus-additional-cni-plugins-cplsq\" (UID: \"198e9ab0-f615-4686-9acf-a6242f7e3584\") " pod="openshift-multus/multus-additional-cni-plugins-cplsq" Nov 04 04:46:37.457697 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.456864 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/844cc12f-a7f6-4327-968e-5e9b2da00ad0-ovnkube-script-lib\") pod \"ovnkube-node-dvc6z\" (UID: \"844cc12f-a7f6-4327-968e-5e9b2da00ad0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvc6z" Nov 04 04:46:37.457697 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.456887 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/22d31d0c-6edf-4351-8f41-a6dd4e02368b-etc-sysconfig\") pod \"tuned-c65v8\" (UID: \"22d31d0c-6edf-4351-8f41-a6dd4e02368b\") " pod="openshift-cluster-node-tuning-operator/tuned-c65v8" Nov 04 04:46:37.457697 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.456910 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/00e0471e-5fe8-4494-8f17-632aee4f6c30-kubelet-dir\") pod \"aws-ebs-csi-driver-node-f2zrp\" (UID: \"00e0471e-5fe8-4494-8f17-632aee4f6c30\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f2zrp" Nov 04 04:46:37.457697 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.456941 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4dff9faf-d91f-4ed5-be82-555b9a6a924e-metrics-certs\") pod \"network-metrics-daemon-25flt\" (UID: \"4dff9faf-d91f-4ed5-be82-555b9a6a924e\") " pod="openshift-multus/network-metrics-daemon-25flt" Nov 04 04:46:37.457697 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.456966 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/844cc12f-a7f6-4327-968e-5e9b2da00ad0-etc-openvswitch\") pod \"ovnkube-node-dvc6z\" (UID: \"844cc12f-a7f6-4327-968e-5e9b2da00ad0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvc6z" Nov 04 04:46:37.457697 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.456987 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/22d31d0c-6edf-4351-8f41-a6dd4e02368b-host\") pod \"tuned-c65v8\" (UID: \"22d31d0c-6edf-4351-8f41-a6dd4e02368b\") " pod="openshift-cluster-node-tuning-operator/tuned-c65v8" Nov 04 04:46:37.457697 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.457010 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/22d31d0c-6edf-4351-8f41-a6dd4e02368b-tmp\") pod \"tuned-c65v8\" (UID: \"22d31d0c-6edf-4351-8f41-a6dd4e02368b\") " pod="openshift-cluster-node-tuning-operator/tuned-c65v8" Nov 04 04:46:37.457697 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.457081 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47nf5\" (UniqueName: \"kubernetes.io/projected/5f6922fa-9ba7-4d02-b550-606731852fde-kube-api-access-47nf5\") pod \"network-check-target-8hsp8\" (UID: \"5f6922fa-9ba7-4d02-b550-606731852fde\") " pod="openshift-network-diagnostics/network-check-target-8hsp8" Nov 04 04:46:37.457697 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.457107 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/00e0471e-5fe8-4494-8f17-632aee4f6c30-registration-dir\") pod \"aws-ebs-csi-driver-node-f2zrp\" (UID: \"00e0471e-5fe8-4494-8f17-632aee4f6c30\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f2zrp" Nov 04 04:46:37.458595 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.457132 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gftxp\" (UniqueName: \"kubernetes.io/projected/00e0471e-5fe8-4494-8f17-632aee4f6c30-kube-api-access-gftxp\") pod \"aws-ebs-csi-driver-node-f2zrp\" (UID: \"00e0471e-5fe8-4494-8f17-632aee4f6c30\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f2zrp" Nov 04 04:46:37.458595 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.457158 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8446aa71-5638-4b23-8f5b-12283ede21d4-hosts-file\") pod \"node-resolver-gzctm\" (UID: \"8446aa71-5638-4b23-8f5b-12283ede21d4\") " pod="openshift-dns/node-resolver-gzctm" Nov 04 04:46:37.458595 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.457183 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d737ccc2-5686-4130-9d00-d2f21d5b417f-host-slash\") pod \"iptables-alerter-wdk7z\" (UID: \"d737ccc2-5686-4130-9d00-d2f21d5b417f\") " pod="openshift-network-operator/iptables-alerter-wdk7z" Nov 04 04:46:37.458595 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.457207 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/844cc12f-a7f6-4327-968e-5e9b2da00ad0-run-systemd\") pod \"ovnkube-node-dvc6z\" (UID: \"844cc12f-a7f6-4327-968e-5e9b2da00ad0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvc6z" Nov 04 04:46:37.458595 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.457230 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/844cc12f-a7f6-4327-968e-5e9b2da00ad0-run-ovn\") pod \"ovnkube-node-dvc6z\" (UID: \"844cc12f-a7f6-4327-968e-5e9b2da00ad0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvc6z" Nov 04 04:46:37.458595 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.457254 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/22d31d0c-6edf-4351-8f41-a6dd4e02368b-var-lib-kubelet\") pod \"tuned-c65v8\" (UID: \"22d31d0c-6edf-4351-8f41-a6dd4e02368b\") " pod="openshift-cluster-node-tuning-operator/tuned-c65v8" Nov 04 04:46:37.458595 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.457279 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/198e9ab0-f615-4686-9acf-a6242f7e3584-system-cni-dir\") pod \"multus-additional-cni-plugins-cplsq\" (UID: \"198e9ab0-f615-4686-9acf-a6242f7e3584\") " pod="openshift-multus/multus-additional-cni-plugins-cplsq" Nov 04 04:46:37.458595 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.457302 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/23ed4a39-18b5-4bc6-9c7b-8b60a7043ad2-agent-certs\") pod \"konnectivity-agent-7tw97\" (UID: \"23ed4a39-18b5-4bc6-9c7b-8b60a7043ad2\") " pod="kube-system/konnectivity-agent-7tw97" Nov 04 04:46:37.494955 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.494924 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-vmp7s\"" Nov 04 04:46:37.495093 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.494999 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-shz8v\"" Nov 04 04:46:37.495093 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.495051 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 04 04:46:37.495174 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.495142 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 04 04:46:37.495215 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.495175 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 04 04:46:37.495215 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.495189 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 04 04:46:37.495285 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.495272 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 04 04:46:37.495343 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.495279 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-zsr9v\"" Nov 04 04:46:37.496130 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.496116 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-mkkxt\"" Nov 04 04:46:37.497697 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.497685 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-tqbjc\"" Nov 04 04:46:37.498151 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.498132 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 04 04:46:37.498243 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.498164 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-kgq44\"" Nov 04 04:46:37.498243 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.498185 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 04 04:46:37.498403 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.498384 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 04 04:46:37.499242 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.499230 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 04 04:46:37.499304 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.499289 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 04 04:46:37.505609 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.505593 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 04 04:46:37.505609 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.505603 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 04 04:46:37.505748 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.505615 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 04 04:46:37.505748 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.505646 2579 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 04 04:46:37.505748 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.505656 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 04 04:46:37.505748 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.505696 2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-h6bkr" Nov 04 04:46:37.557823 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.557787 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c79eff94-6543-4313-b59b-502ca4089e84-dbus\") pod \"global-pull-secret-syncer-gfjqz\" (UID: \"c79eff94-6543-4313-b59b-502ca4089e84\") " pod="kube-system/global-pull-secret-syncer-gfjqz" Nov 04 04:46:37.557823 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.557825 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/198e9ab0-f615-4686-9acf-a6242f7e3584-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cplsq\" (UID: \"198e9ab0-f615-4686-9acf-a6242f7e3584\") " pod="openshift-multus/multus-additional-cni-plugins-cplsq" Nov 04 04:46:37.558052 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.557842 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/00e0471e-5fe8-4494-8f17-632aee4f6c30-device-dir\") pod \"aws-ebs-csi-driver-node-f2zrp\" (UID: \"00e0471e-5fe8-4494-8f17-632aee4f6c30\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f2zrp" Nov 04 04:46:37.558052 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.557864 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/00e0471e-5fe8-4494-8f17-632aee4f6c30-etc-selinux\") pod \"aws-ebs-csi-driver-node-f2zrp\" (UID: \"00e0471e-5fe8-4494-8f17-632aee4f6c30\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f2zrp" Nov 04 04:46:37.558052 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.557885 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/58be7092-9156-4787-8f49-a173fa0bd5a3-host-var-lib-kubelet\") pod \"multus-7m8fc\" (UID: \"58be7092-9156-4787-8f49-a173fa0bd5a3\") " pod="openshift-multus/multus-7m8fc" Nov 04 04:46:37.558052 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.557908 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c79eff94-6543-4313-b59b-502ca4089e84-kubelet-config\") pod \"global-pull-secret-syncer-gfjqz\" (UID: \"c79eff94-6543-4313-b59b-502ca4089e84\") " pod="kube-system/global-pull-secret-syncer-gfjqz" Nov 04 04:46:37.558052 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.557921 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c79eff94-6543-4313-b59b-502ca4089e84-dbus\") pod \"global-pull-secret-syncer-gfjqz\" (UID: \"c79eff94-6543-4313-b59b-502ca4089e84\") " pod="kube-system/global-pull-secret-syncer-gfjqz" Nov 04 04:46:37.558052 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.557951 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/00e0471e-5fe8-4494-8f17-632aee4f6c30-device-dir\") pod \"aws-ebs-csi-driver-node-f2zrp\" (UID: \"00e0471e-5fe8-4494-8f17-632aee4f6c30\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f2zrp" Nov 04 04:46:37.558052 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.557931 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c79eff94-6543-4313-b59b-502ca4089e84-original-pull-secret\") pod \"global-pull-secret-syncer-gfjqz\" (UID: \"c79eff94-6543-4313-b59b-502ca4089e84\") " pod="kube-system/global-pull-secret-syncer-gfjqz" Nov 04 04:46:37.558052 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.557966 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/58be7092-9156-4787-8f49-a173fa0bd5a3-host-var-lib-kubelet\") pod \"multus-7m8fc\" (UID: \"58be7092-9156-4787-8f49-a173fa0bd5a3\") " pod="openshift-multus/multus-7m8fc" Nov 04 04:46:37.558052 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.557996 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kz2hd\" (UniqueName: \"kubernetes.io/projected/d737ccc2-5686-4130-9d00-d2f21d5b417f-kube-api-access-kz2hd\") pod \"iptables-alerter-wdk7z\" (UID: \"d737ccc2-5686-4130-9d00-d2f21d5b417f\") " pod="openshift-network-operator/iptables-alerter-wdk7z" Nov 04 04:46:37.558052 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.558003 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/00e0471e-5fe8-4494-8f17-632aee4f6c30-etc-selinux\") pod \"aws-ebs-csi-driver-node-f2zrp\" (UID: \"00e0471e-5fe8-4494-8f17-632aee4f6c30\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f2zrp" Nov 04 04:46:37.558052 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.558013 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c79eff94-6543-4313-b59b-502ca4089e84-kubelet-config\") pod \"global-pull-secret-syncer-gfjqz\" (UID: \"c79eff94-6543-4313-b59b-502ca4089e84\") " pod="kube-system/global-pull-secret-syncer-gfjqz" Nov 04 04:46:37.558052 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.558021 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/844cc12f-a7f6-4327-968e-5e9b2da00ad0-systemd-units\") pod \"ovnkube-node-dvc6z\" (UID: \"844cc12f-a7f6-4327-968e-5e9b2da00ad0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvc6z" Nov 04 04:46:37.558052 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.558047 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/844cc12f-a7f6-4327-968e-5e9b2da00ad0-systemd-units\") pod \"ovnkube-node-dvc6z\" (UID: \"844cc12f-a7f6-4327-968e-5e9b2da00ad0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvc6z" Nov 04 04:46:37.558597 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.558070 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/844cc12f-a7f6-4327-968e-5e9b2da00ad0-host-run-netns\") pod \"ovnkube-node-dvc6z\" (UID: \"844cc12f-a7f6-4327-968e-5e9b2da00ad0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvc6z" Nov 04 04:46:37.558597 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.558100 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/22d31d0c-6edf-4351-8f41-a6dd4e02368b-etc-sysctl-d\") pod \"tuned-c65v8\" (UID: \"22d31d0c-6edf-4351-8f41-a6dd4e02368b\") " pod="openshift-cluster-node-tuning-operator/tuned-c65v8" Nov 04 04:46:37.558597 ip-10-0-133-180 kubenswrapper[2579]: E1104 04:46:37.558123 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 04 04:46:37.558597 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.558125 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/58be7092-9156-4787-8f49-a173fa0bd5a3-system-cni-dir\") pod \"multus-7m8fc\" (UID: \"58be7092-9156-4787-8f49-a173fa0bd5a3\") " pod="openshift-multus/multus-7m8fc" Nov 04 04:46:37.558597 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.558171 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/844cc12f-a7f6-4327-968e-5e9b2da00ad0-host-run-netns\") pod \"ovnkube-node-dvc6z\" (UID: \"844cc12f-a7f6-4327-968e-5e9b2da00ad0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvc6z" Nov 04 04:46:37.558597 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.558184 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/58be7092-9156-4787-8f49-a173fa0bd5a3-system-cni-dir\") pod \"multus-7m8fc\" (UID: \"58be7092-9156-4787-8f49-a173fa0bd5a3\") " pod="openshift-multus/multus-7m8fc" Nov 04 04:46:37.558597 ip-10-0-133-180 kubenswrapper[2579]: E1104 04:46:37.558191 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c79eff94-6543-4313-b59b-502ca4089e84-original-pull-secret podName:c79eff94-6543-4313-b59b-502ca4089e84 nodeName:}" failed. No retries permitted until 2026-11-04 04:46:38.058168661 +0000 UTC m=+2.273416583 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c79eff94-6543-4313-b59b-502ca4089e84-original-pull-secret") pod "global-pull-secret-syncer-gfjqz" (UID: "c79eff94-6543-4313-b59b-502ca4089e84") : object "kube-system"/"original-pull-secret" not registered Nov 04 04:46:37.558597 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.558220 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/58be7092-9156-4787-8f49-a173fa0bd5a3-cnibin\") pod \"multus-7m8fc\" (UID: \"58be7092-9156-4787-8f49-a173fa0bd5a3\") " pod="openshift-multus/multus-7m8fc" Nov 04 04:46:37.558597 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.558247 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/58be7092-9156-4787-8f49-a173fa0bd5a3-multus-conf-dir\") pod \"multus-7m8fc\" (UID: \"58be7092-9156-4787-8f49-a173fa0bd5a3\") " pod="openshift-multus/multus-7m8fc" Nov 04 04:46:37.558597 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.558267 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/22d31d0c-6edf-4351-8f41-a6dd4e02368b-lib-modules\") pod \"tuned-c65v8\" (UID: \"22d31d0c-6edf-4351-8f41-a6dd4e02368b\") " pod="openshift-cluster-node-tuning-operator/tuned-c65v8" Nov 04 04:46:37.558597 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.558280 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/22d31d0c-6edf-4351-8f41-a6dd4e02368b-etc-sysctl-d\") pod \"tuned-c65v8\" (UID: \"22d31d0c-6edf-4351-8f41-a6dd4e02368b\") " pod="openshift-cluster-node-tuning-operator/tuned-c65v8" Nov 04 04:46:37.558597 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.558313 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/58be7092-9156-4787-8f49-a173fa0bd5a3-host-run-k8s-cni-cncf-io\") pod \"multus-7m8fc\" (UID: \"58be7092-9156-4787-8f49-a173fa0bd5a3\") " pod="openshift-multus/multus-7m8fc" Nov 04 04:46:37.558597 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.558360 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/58be7092-9156-4787-8f49-a173fa0bd5a3-cnibin\") pod \"multus-7m8fc\" (UID: \"58be7092-9156-4787-8f49-a173fa0bd5a3\") " pod="openshift-multus/multus-7m8fc" Nov 04 04:46:37.558597 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.558374 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/58be7092-9156-4787-8f49-a173fa0bd5a3-host-run-k8s-cni-cncf-io\") pod \"multus-7m8fc\" (UID: \"58be7092-9156-4787-8f49-a173fa0bd5a3\") " pod="openshift-multus/multus-7m8fc" Nov 04 04:46:37.558597 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.558378 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/58be7092-9156-4787-8f49-a173fa0bd5a3-multus-conf-dir\") pod \"multus-7m8fc\" (UID: \"58be7092-9156-4787-8f49-a173fa0bd5a3\") " pod="openshift-multus/multus-7m8fc" Nov 04 04:46:37.558597 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.558401 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/844cc12f-a7f6-4327-968e-5e9b2da00ad0-host-slash\") pod \"ovnkube-node-dvc6z\" (UID: \"844cc12f-a7f6-4327-968e-5e9b2da00ad0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvc6z" Nov 04 04:46:37.558597 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.558442 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/844cc12f-a7f6-4327-968e-5e9b2da00ad0-host-slash\") pod \"ovnkube-node-dvc6z\" (UID: \"844cc12f-a7f6-4327-968e-5e9b2da00ad0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvc6z" Nov 04 04:46:37.558597 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.558457 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/198e9ab0-f615-4686-9acf-a6242f7e3584-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cplsq\" (UID: \"198e9ab0-f615-4686-9acf-a6242f7e3584\") " pod="openshift-multus/multus-additional-cni-plugins-cplsq" Nov 04 04:46:37.559310 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.558464 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/844cc12f-a7f6-4327-968e-5e9b2da00ad0-node-log\") pod \"ovnkube-node-dvc6z\" (UID: \"844cc12f-a7f6-4327-968e-5e9b2da00ad0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvc6z" Nov 04 04:46:37.559310 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.558495 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/844cc12f-a7f6-4327-968e-5e9b2da00ad0-node-log\") pod \"ovnkube-node-dvc6z\" (UID: \"844cc12f-a7f6-4327-968e-5e9b2da00ad0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvc6z" Nov 04 04:46:37.559310 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.558501 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/844cc12f-a7f6-4327-968e-5e9b2da00ad0-env-overrides\") pod \"ovnkube-node-dvc6z\" (UID: \"844cc12f-a7f6-4327-968e-5e9b2da00ad0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvc6z" Nov 04 04:46:37.559310 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.558528 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/22d31d0c-6edf-4351-8f41-a6dd4e02368b-lib-modules\") pod \"tuned-c65v8\" (UID: \"22d31d0c-6edf-4351-8f41-a6dd4e02368b\") " pod="openshift-cluster-node-tuning-operator/tuned-c65v8" Nov 04 04:46:37.559310 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.558531 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/58be7092-9156-4787-8f49-a173fa0bd5a3-multus-daemon-config\") pod \"multus-7m8fc\" (UID: \"58be7092-9156-4787-8f49-a173fa0bd5a3\") " pod="openshift-multus/multus-7m8fc" Nov 04 04:46:37.559310 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.558557 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d737ccc2-5686-4130-9d00-d2f21d5b417f-iptables-alerter-script\") pod \"iptables-alerter-wdk7z\" (UID: \"d737ccc2-5686-4130-9d00-d2f21d5b417f\") " pod="openshift-network-operator/iptables-alerter-wdk7z" Nov 04 04:46:37.559310 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.558599 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/844cc12f-a7f6-4327-968e-5e9b2da00ad0-host-run-ovn-kubernetes\") pod \"ovnkube-node-dvc6z\" (UID: \"844cc12f-a7f6-4327-968e-5e9b2da00ad0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvc6z" Nov 04 04:46:37.559310 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.558621 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/844cc12f-a7f6-4327-968e-5e9b2da00ad0-host-cni-bin\") pod \"ovnkube-node-dvc6z\" (UID: \"844cc12f-a7f6-4327-968e-5e9b2da00ad0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvc6z" Nov 04 04:46:37.559310 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.558643 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/58be7092-9156-4787-8f49-a173fa0bd5a3-multus-socket-dir-parent\") pod \"multus-7m8fc\" (UID: \"58be7092-9156-4787-8f49-a173fa0bd5a3\") " pod="openshift-multus/multus-7m8fc" Nov 04 04:46:37.559310 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.558652 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/844cc12f-a7f6-4327-968e-5e9b2da00ad0-host-run-ovn-kubernetes\") pod \"ovnkube-node-dvc6z\" (UID: \"844cc12f-a7f6-4327-968e-5e9b2da00ad0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvc6z" Nov 04 04:46:37.559310 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.558666 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/23ed4a39-18b5-4bc6-9c7b-8b60a7043ad2-konnectivity-ca\") pod \"konnectivity-agent-7tw97\" (UID: \"23ed4a39-18b5-4bc6-9c7b-8b60a7043ad2\") " pod="kube-system/konnectivity-agent-7tw97" Nov 04 04:46:37.559310 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.558685 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/844cc12f-a7f6-4327-968e-5e9b2da00ad0-host-cni-bin\") pod \"ovnkube-node-dvc6z\" (UID: \"844cc12f-a7f6-4327-968e-5e9b2da00ad0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvc6z" Nov 04 04:46:37.559310 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.558688 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/844cc12f-a7f6-4327-968e-5e9b2da00ad0-ovn-node-metrics-cert\") pod \"ovnkube-node-dvc6z\" (UID: \"844cc12f-a7f6-4327-968e-5e9b2da00ad0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvc6z" Nov 04 04:46:37.559310 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.558728 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbtxf\" (UniqueName: \"kubernetes.io/projected/22d31d0c-6edf-4351-8f41-a6dd4e02368b-kube-api-access-zbtxf\") pod \"tuned-c65v8\" (UID: \"22d31d0c-6edf-4351-8f41-a6dd4e02368b\") " pod="openshift-cluster-node-tuning-operator/tuned-c65v8" Nov 04 04:46:37.559310 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.558752 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/58be7092-9156-4787-8f49-a173fa0bd5a3-etc-kubernetes\") pod \"multus-7m8fc\" (UID: \"58be7092-9156-4787-8f49-a173fa0bd5a3\") " pod="openshift-multus/multus-7m8fc" Nov 04 04:46:37.559310 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.558773 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/22d31d0c-6edf-4351-8f41-a6dd4e02368b-etc-modprobe-d\") pod \"tuned-c65v8\" (UID: \"22d31d0c-6edf-4351-8f41-a6dd4e02368b\") " pod="openshift-cluster-node-tuning-operator/tuned-c65v8" Nov 04 04:46:37.559310 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.558800 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlz69\" (UniqueName: \"kubernetes.io/projected/9a06ec07-c72b-43af-aa27-7414aa9c9aba-kube-api-access-mlz69\") pod \"node-ca-4z89d\" (UID: \"9a06ec07-c72b-43af-aa27-7414aa9c9aba\") " pod="openshift-image-registry/node-ca-4z89d" Nov 04 04:46:37.560099 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.558805 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/58be7092-9156-4787-8f49-a173fa0bd5a3-multus-socket-dir-parent\") pod \"multus-7m8fc\" (UID: \"58be7092-9156-4787-8f49-a173fa0bd5a3\") " pod="openshift-multus/multus-7m8fc" Nov 04 04:46:37.560099 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.558848 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/58be7092-9156-4787-8f49-a173fa0bd5a3-multus-cni-dir\") pod \"multus-7m8fc\" (UID: \"58be7092-9156-4787-8f49-a173fa0bd5a3\") " pod="openshift-multus/multus-7m8fc" Nov 04 04:46:37.560099 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.558851 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/58be7092-9156-4787-8f49-a173fa0bd5a3-etc-kubernetes\") pod \"multus-7m8fc\" (UID: \"58be7092-9156-4787-8f49-a173fa0bd5a3\") " pod="openshift-multus/multus-7m8fc" Nov 04 04:46:37.560099 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.558874 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/58be7092-9156-4787-8f49-a173fa0bd5a3-host-run-multus-certs\") pod \"multus-7m8fc\" (UID: \"58be7092-9156-4787-8f49-a173fa0bd5a3\") " pod="openshift-multus/multus-7m8fc" Nov 04 04:46:37.560099 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.558896 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25g6k\" (UniqueName: \"kubernetes.io/projected/8446aa71-5638-4b23-8f5b-12283ede21d4-kube-api-access-25g6k\") pod \"node-resolver-gzctm\" (UID: \"8446aa71-5638-4b23-8f5b-12283ede21d4\") " pod="openshift-dns/node-resolver-gzctm" Nov 04 04:46:37.560099 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.558929 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/198e9ab0-f615-4686-9acf-a6242f7e3584-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cplsq\" (UID: \"198e9ab0-f615-4686-9acf-a6242f7e3584\") " pod="openshift-multus/multus-additional-cni-plugins-cplsq" Nov 04 04:46:37.560099 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.558944 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/198e9ab0-f615-4686-9acf-a6242f7e3584-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cplsq\" (UID: \"198e9ab0-f615-4686-9acf-a6242f7e3584\") " pod="openshift-multus/multus-additional-cni-plugins-cplsq" Nov 04 04:46:37.560099 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.558959 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/844cc12f-a7f6-4327-968e-5e9b2da00ad0-host-kubelet\") pod \"ovnkube-node-dvc6z\" (UID: \"844cc12f-a7f6-4327-968e-5e9b2da00ad0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvc6z" Nov 04 04:46:37.560099 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.558974 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/844cc12f-a7f6-4327-968e-5e9b2da00ad0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dvc6z\" (UID: \"844cc12f-a7f6-4327-968e-5e9b2da00ad0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvc6z" Nov 04 04:46:37.560099 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.558989 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/22d31d0c-6edf-4351-8f41-a6dd4e02368b-sys\") pod \"tuned-c65v8\" (UID: \"22d31d0c-6edf-4351-8f41-a6dd4e02368b\") " pod="openshift-cluster-node-tuning-operator/tuned-c65v8" Nov 04 04:46:37.560099 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.559003 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/58be7092-9156-4787-8f49-a173fa0bd5a3-host-var-lib-cni-multus\") pod \"multus-7m8fc\" (UID: \"58be7092-9156-4787-8f49-a173fa0bd5a3\") " pod="openshift-multus/multus-7m8fc" Nov 04 04:46:37.560099 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.559017 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4xvx\" (UniqueName: \"kubernetes.io/projected/58be7092-9156-4787-8f49-a173fa0bd5a3-kube-api-access-w4xvx\") pod \"multus-7m8fc\" (UID: \"58be7092-9156-4787-8f49-a173fa0bd5a3\") " pod="openshift-multus/multus-7m8fc" Nov 04 04:46:37.560099 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.559031 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/198e9ab0-f615-4686-9acf-a6242f7e3584-cnibin\") pod \"multus-additional-cni-plugins-cplsq\" (UID: \"198e9ab0-f615-4686-9acf-a6242f7e3584\") " pod="openshift-multus/multus-additional-cni-plugins-cplsq" Nov 04 04:46:37.560099 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.559045 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccfxs\" (UniqueName: \"kubernetes.io/projected/198e9ab0-f615-4686-9acf-a6242f7e3584-kube-api-access-ccfxs\") pod \"multus-additional-cni-plugins-cplsq\" (UID: \"198e9ab0-f615-4686-9acf-a6242f7e3584\") " pod="openshift-multus/multus-additional-cni-plugins-cplsq" Nov 04 04:46:37.560099 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.559059 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/844cc12f-a7f6-4327-968e-5e9b2da00ad0-ovnkube-script-lib\") pod \"ovnkube-node-dvc6z\" (UID: \"844cc12f-a7f6-4327-968e-5e9b2da00ad0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvc6z" Nov 04 04:46:37.560099 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.559072 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/22d31d0c-6edf-4351-8f41-a6dd4e02368b-etc-sysconfig\") pod \"tuned-c65v8\" (UID: \"22d31d0c-6edf-4351-8f41-a6dd4e02368b\") " pod="openshift-cluster-node-tuning-operator/tuned-c65v8" Nov 04 04:46:37.560099 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.559105 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/58be7092-9156-4787-8f49-a173fa0bd5a3-multus-daemon-config\") pod \"multus-7m8fc\" (UID: \"58be7092-9156-4787-8f49-a173fa0bd5a3\") " pod="openshift-multus/multus-7m8fc" Nov 04 04:46:37.560907 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.559166 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/844cc12f-a7f6-4327-968e-5e9b2da00ad0-host-kubelet\") pod \"ovnkube-node-dvc6z\" (UID: \"844cc12f-a7f6-4327-968e-5e9b2da00ad0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvc6z" Nov 04 04:46:37.560907 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.559254 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/22d31d0c-6edf-4351-8f41-a6dd4e02368b-etc-modprobe-d\") pod \"tuned-c65v8\" (UID: \"22d31d0c-6edf-4351-8f41-a6dd4e02368b\") " pod="openshift-cluster-node-tuning-operator/tuned-c65v8" Nov 04 04:46:37.560907 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.559268 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/23ed4a39-18b5-4bc6-9c7b-8b60a7043ad2-konnectivity-ca\") pod \"konnectivity-agent-7tw97\" (UID: \"23ed4a39-18b5-4bc6-9c7b-8b60a7043ad2\") " pod="kube-system/konnectivity-agent-7tw97" Nov 04 04:46:37.560907 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.559371 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/844cc12f-a7f6-4327-968e-5e9b2da00ad0-env-overrides\") pod \"ovnkube-node-dvc6z\" (UID: \"844cc12f-a7f6-4327-968e-5e9b2da00ad0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvc6z" Nov 04 04:46:37.560907 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.559461 2579 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 04 04:46:37.560907 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.559467 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/58be7092-9156-4787-8f49-a173fa0bd5a3-multus-cni-dir\") pod \"multus-7m8fc\" (UID: \"58be7092-9156-4787-8f49-a173fa0bd5a3\") " pod="openshift-multus/multus-7m8fc" Nov 04 04:46:37.560907 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.559504 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/58be7092-9156-4787-8f49-a173fa0bd5a3-host-run-multus-certs\") pod \"multus-7m8fc\" (UID: \"58be7092-9156-4787-8f49-a173fa0bd5a3\") " pod="openshift-multus/multus-7m8fc" Nov 04 04:46:37.560907 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.559549 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/198e9ab0-f615-4686-9acf-a6242f7e3584-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cplsq\" (UID: \"198e9ab0-f615-4686-9acf-a6242f7e3584\") " pod="openshift-multus/multus-additional-cni-plugins-cplsq" Nov 04 04:46:37.560907 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.559553 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/00e0471e-5fe8-4494-8f17-632aee4f6c30-kubelet-dir\") pod \"aws-ebs-csi-driver-node-f2zrp\" (UID: \"00e0471e-5fe8-4494-8f17-632aee4f6c30\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f2zrp" Nov 04 04:46:37.560907 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.559592 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4dff9faf-d91f-4ed5-be82-555b9a6a924e-metrics-certs\") pod \"network-metrics-daemon-25flt\" (UID: \"4dff9faf-d91f-4ed5-be82-555b9a6a924e\") " pod="openshift-multus/network-metrics-daemon-25flt" Nov 04 04:46:37.560907 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.559592 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d737ccc2-5686-4130-9d00-d2f21d5b417f-iptables-alerter-script\") pod \"iptables-alerter-wdk7z\" (UID: \"d737ccc2-5686-4130-9d00-d2f21d5b417f\") " pod="openshift-network-operator/iptables-alerter-wdk7z" Nov 04 04:46:37.560907 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.559614 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/844cc12f-a7f6-4327-968e-5e9b2da00ad0-etc-openvswitch\") pod \"ovnkube-node-dvc6z\" (UID: \"844cc12f-a7f6-4327-968e-5e9b2da00ad0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvc6z" Nov 04 04:46:37.560907 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.559613 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/00e0471e-5fe8-4494-8f17-632aee4f6c30-kubelet-dir\") pod \"aws-ebs-csi-driver-node-f2zrp\" (UID: \"00e0471e-5fe8-4494-8f17-632aee4f6c30\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f2zrp" Nov 04 04:46:37.560907 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.559623 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/22d31d0c-6edf-4351-8f41-a6dd4e02368b-sys\") pod \"tuned-c65v8\" (UID: \"22d31d0c-6edf-4351-8f41-a6dd4e02368b\") " pod="openshift-cluster-node-tuning-operator/tuned-c65v8" Nov 04 04:46:37.560907 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.559660 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/844cc12f-a7f6-4327-968e-5e9b2da00ad0-etc-openvswitch\") pod \"ovnkube-node-dvc6z\" (UID: \"844cc12f-a7f6-4327-968e-5e9b2da00ad0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvc6z" Nov 04 04:46:37.560907 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.559664 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/198e9ab0-f615-4686-9acf-a6242f7e3584-cnibin\") pod \"multus-additional-cni-plugins-cplsq\" (UID: \"198e9ab0-f615-4686-9acf-a6242f7e3584\") " pod="openshift-multus/multus-additional-cni-plugins-cplsq" Nov 04 04:46:37.560907 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.559680 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/844cc12f-a7f6-4327-968e-5e9b2da00ad0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dvc6z\" (UID: \"844cc12f-a7f6-4327-968e-5e9b2da00ad0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvc6z" Nov 04 04:46:37.560907 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.559684 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/58be7092-9156-4787-8f49-a173fa0bd5a3-host-var-lib-cni-multus\") pod \"multus-7m8fc\" (UID: \"58be7092-9156-4787-8f49-a173fa0bd5a3\") " pod="openshift-multus/multus-7m8fc" Nov 04 04:46:37.561637 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.559713 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/22d31d0c-6edf-4351-8f41-a6dd4e02368b-etc-sysconfig\") pod \"tuned-c65v8\" (UID: \"22d31d0c-6edf-4351-8f41-a6dd4e02368b\") " pod="openshift-cluster-node-tuning-operator/tuned-c65v8" Nov 04 04:46:37.561637 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.559724 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/22d31d0c-6edf-4351-8f41-a6dd4e02368b-host\") pod \"tuned-c65v8\" (UID: \"22d31d0c-6edf-4351-8f41-a6dd4e02368b\") " pod="openshift-cluster-node-tuning-operator/tuned-c65v8" Nov 04 04:46:37.561637 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.559759 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/22d31d0c-6edf-4351-8f41-a6dd4e02368b-host\") pod \"tuned-c65v8\" (UID: \"22d31d0c-6edf-4351-8f41-a6dd4e02368b\") " pod="openshift-cluster-node-tuning-operator/tuned-c65v8" Nov 04 04:46:37.561637 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.559762 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/22d31d0c-6edf-4351-8f41-a6dd4e02368b-tmp\") pod \"tuned-c65v8\" (UID: \"22d31d0c-6edf-4351-8f41-a6dd4e02368b\") " pod="openshift-cluster-node-tuning-operator/tuned-c65v8" Nov 04 04:46:37.561637 ip-10-0-133-180 kubenswrapper[2579]: E1104 04:46:37.559729 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 04:46:37.561637 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.559790 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47nf5\" (UniqueName: \"kubernetes.io/projected/5f6922fa-9ba7-4d02-b550-606731852fde-kube-api-access-47nf5\") pod \"network-check-target-8hsp8\" (UID: \"5f6922fa-9ba7-4d02-b550-606731852fde\") " pod="openshift-network-diagnostics/network-check-target-8hsp8" Nov 04 04:46:37.561637 ip-10-0-133-180 kubenswrapper[2579]: E1104 04:46:37.559816 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4dff9faf-d91f-4ed5-be82-555b9a6a924e-metrics-certs podName:4dff9faf-d91f-4ed5-be82-555b9a6a924e nodeName:}" failed. No retries permitted until 2026-11-04 04:46:38.059803303 +0000 UTC m=+2.275051221 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4dff9faf-d91f-4ed5-be82-555b9a6a924e-metrics-certs") pod "network-metrics-daemon-25flt" (UID: "4dff9faf-d91f-4ed5-be82-555b9a6a924e") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 04:46:37.561637 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.559830 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/00e0471e-5fe8-4494-8f17-632aee4f6c30-registration-dir\") pod \"aws-ebs-csi-driver-node-f2zrp\" (UID: \"00e0471e-5fe8-4494-8f17-632aee4f6c30\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f2zrp" Nov 04 04:46:37.561637 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.559850 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gftxp\" (UniqueName: \"kubernetes.io/projected/00e0471e-5fe8-4494-8f17-632aee4f6c30-kube-api-access-gftxp\") pod \"aws-ebs-csi-driver-node-f2zrp\" (UID: \"00e0471e-5fe8-4494-8f17-632aee4f6c30\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f2zrp" Nov 04 04:46:37.561637 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.559871 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8446aa71-5638-4b23-8f5b-12283ede21d4-hosts-file\") pod \"node-resolver-gzctm\" (UID: \"8446aa71-5638-4b23-8f5b-12283ede21d4\") " pod="openshift-dns/node-resolver-gzctm" Nov 04 04:46:37.561637 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.559872 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/198e9ab0-f615-4686-9acf-a6242f7e3584-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cplsq\" (UID: \"198e9ab0-f615-4686-9acf-a6242f7e3584\") " pod="openshift-multus/multus-additional-cni-plugins-cplsq" Nov 04 04:46:37.561637 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.559889 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d737ccc2-5686-4130-9d00-d2f21d5b417f-host-slash\") pod \"iptables-alerter-wdk7z\" (UID: \"d737ccc2-5686-4130-9d00-d2f21d5b417f\") " pod="openshift-network-operator/iptables-alerter-wdk7z" Nov 04 04:46:37.561637 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.559910 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/844cc12f-a7f6-4327-968e-5e9b2da00ad0-run-systemd\") pod \"ovnkube-node-dvc6z\" (UID: \"844cc12f-a7f6-4327-968e-5e9b2da00ad0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvc6z" Nov 04 04:46:37.561637 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.559915 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/00e0471e-5fe8-4494-8f17-632aee4f6c30-registration-dir\") pod \"aws-ebs-csi-driver-node-f2zrp\" (UID: \"00e0471e-5fe8-4494-8f17-632aee4f6c30\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f2zrp" Nov 04 04:46:37.561637 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.559929 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/844cc12f-a7f6-4327-968e-5e9b2da00ad0-run-ovn\") pod \"ovnkube-node-dvc6z\" (UID: \"844cc12f-a7f6-4327-968e-5e9b2da00ad0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvc6z" Nov 04 04:46:37.561637 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.559947 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/22d31d0c-6edf-4351-8f41-a6dd4e02368b-var-lib-kubelet\") pod \"tuned-c65v8\" (UID: \"22d31d0c-6edf-4351-8f41-a6dd4e02368b\") " pod="openshift-cluster-node-tuning-operator/tuned-c65v8" Nov 04 04:46:37.561637 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.559962 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d737ccc2-5686-4130-9d00-d2f21d5b417f-host-slash\") pod \"iptables-alerter-wdk7z\" (UID: \"d737ccc2-5686-4130-9d00-d2f21d5b417f\") " pod="openshift-network-operator/iptables-alerter-wdk7z" Nov 04 04:46:37.562113 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.559968 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/198e9ab0-f615-4686-9acf-a6242f7e3584-system-cni-dir\") pod \"multus-additional-cni-plugins-cplsq\" (UID: \"198e9ab0-f615-4686-9acf-a6242f7e3584\") " pod="openshift-multus/multus-additional-cni-plugins-cplsq" Nov 04 04:46:37.562113 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.559978 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/844cc12f-a7f6-4327-968e-5e9b2da00ad0-run-systemd\") pod \"ovnkube-node-dvc6z\" (UID: \"844cc12f-a7f6-4327-968e-5e9b2da00ad0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvc6z" Nov 04 04:46:37.562113 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.559988 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/23ed4a39-18b5-4bc6-9c7b-8b60a7043ad2-agent-certs\") pod \"konnectivity-agent-7tw97\" (UID: \"23ed4a39-18b5-4bc6-9c7b-8b60a7043ad2\") " pod="kube-system/konnectivity-agent-7tw97" Nov 04 04:46:37.562113 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.560027 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/198e9ab0-f615-4686-9acf-a6242f7e3584-system-cni-dir\") pod \"multus-additional-cni-plugins-cplsq\" (UID: \"198e9ab0-f615-4686-9acf-a6242f7e3584\") " pod="openshift-multus/multus-additional-cni-plugins-cplsq" Nov 04 04:46:37.562113 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.560026 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/844cc12f-a7f6-4327-968e-5e9b2da00ad0-run-ovn\") pod \"ovnkube-node-dvc6z\" (UID: \"844cc12f-a7f6-4327-968e-5e9b2da00ad0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvc6z" Nov 04 04:46:37.562113 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.560052 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/844cc12f-a7f6-4327-968e-5e9b2da00ad0-var-lib-openvswitch\") pod \"ovnkube-node-dvc6z\" (UID: \"844cc12f-a7f6-4327-968e-5e9b2da00ad0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvc6z" Nov 04 04:46:37.562113 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.560079 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/844cc12f-a7f6-4327-968e-5e9b2da00ad0-log-socket\") pod \"ovnkube-node-dvc6z\" (UID: \"844cc12f-a7f6-4327-968e-5e9b2da00ad0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvc6z" Nov 04 04:46:37.562113 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.560102 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/58be7092-9156-4787-8f49-a173fa0bd5a3-host-var-lib-cni-bin\") pod \"multus-7m8fc\" (UID: \"58be7092-9156-4787-8f49-a173fa0bd5a3\") " pod="openshift-multus/multus-7m8fc" Nov 04 04:46:37.562113 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.560102 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/844cc12f-a7f6-4327-968e-5e9b2da00ad0-var-lib-openvswitch\") pod \"ovnkube-node-dvc6z\" (UID: \"844cc12f-a7f6-4327-968e-5e9b2da00ad0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvc6z" Nov 04 04:46:37.562113 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.560137 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/58be7092-9156-4787-8f49-a173fa0bd5a3-host-var-lib-cni-bin\") pod \"multus-7m8fc\" (UID: \"58be7092-9156-4787-8f49-a173fa0bd5a3\") " pod="openshift-multus/multus-7m8fc" Nov 04 04:46:37.562113 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.560026 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/22d31d0c-6edf-4351-8f41-a6dd4e02368b-var-lib-kubelet\") pod \"tuned-c65v8\" (UID: \"22d31d0c-6edf-4351-8f41-a6dd4e02368b\") " pod="openshift-cluster-node-tuning-operator/tuned-c65v8" Nov 04 04:46:37.562113 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.560126 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8446aa71-5638-4b23-8f5b-12283ede21d4-tmp-dir\") pod \"node-resolver-gzctm\" (UID: \"8446aa71-5638-4b23-8f5b-12283ede21d4\") " pod="openshift-dns/node-resolver-gzctm" Nov 04 04:46:37.562113 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.560244 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nch9\" (UniqueName: \"kubernetes.io/projected/844cc12f-a7f6-4327-968e-5e9b2da00ad0-kube-api-access-6nch9\") pod \"ovnkube-node-dvc6z\" (UID: \"844cc12f-a7f6-4327-968e-5e9b2da00ad0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvc6z" Nov 04 04:46:37.562113 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.560271 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/22d31d0c-6edf-4351-8f41-a6dd4e02368b-etc-kubernetes\") pod \"tuned-c65v8\" (UID: \"22d31d0c-6edf-4351-8f41-a6dd4e02368b\") " pod="openshift-cluster-node-tuning-operator/tuned-c65v8" Nov 04 04:46:37.562113 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.560274 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/844cc12f-a7f6-4327-968e-5e9b2da00ad0-ovnkube-script-lib\") pod \"ovnkube-node-dvc6z\" (UID: \"844cc12f-a7f6-4327-968e-5e9b2da00ad0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvc6z" Nov 04 04:46:37.562113 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.560173 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/844cc12f-a7f6-4327-968e-5e9b2da00ad0-log-socket\") pod \"ovnkube-node-dvc6z\" (UID: \"844cc12f-a7f6-4327-968e-5e9b2da00ad0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvc6z" Nov 04 04:46:37.562113 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.559947 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8446aa71-5638-4b23-8f5b-12283ede21d4-hosts-file\") pod \"node-resolver-gzctm\" (UID: \"8446aa71-5638-4b23-8f5b-12283ede21d4\") " pod="openshift-dns/node-resolver-gzctm" Nov 04 04:46:37.562113 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.560354 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/22d31d0c-6edf-4351-8f41-a6dd4e02368b-etc-kubernetes\") pod \"tuned-c65v8\" (UID: \"22d31d0c-6edf-4351-8f41-a6dd4e02368b\") " pod="openshift-cluster-node-tuning-operator/tuned-c65v8" Nov 04 04:46:37.562593 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.560355 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/22d31d0c-6edf-4351-8f41-a6dd4e02368b-etc-systemd\") pod \"tuned-c65v8\" (UID: \"22d31d0c-6edf-4351-8f41-a6dd4e02368b\") " pod="openshift-cluster-node-tuning-operator/tuned-c65v8" Nov 04 04:46:37.562593 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.560381 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/198e9ab0-f615-4686-9acf-a6242f7e3584-cni-binary-copy\") pod \"multus-additional-cni-plugins-cplsq\" (UID: \"198e9ab0-f615-4686-9acf-a6242f7e3584\") " pod="openshift-multus/multus-additional-cni-plugins-cplsq" Nov 04 04:46:37.562593 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.560396 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/22d31d0c-6edf-4351-8f41-a6dd4e02368b-etc-systemd\") pod \"tuned-c65v8\" (UID: \"22d31d0c-6edf-4351-8f41-a6dd4e02368b\") " pod="openshift-cluster-node-tuning-operator/tuned-c65v8" Nov 04 04:46:37.562593 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.560403 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/844cc12f-a7f6-4327-968e-5e9b2da00ad0-host-cni-netd\") pod \"ovnkube-node-dvc6z\" (UID: \"844cc12f-a7f6-4327-968e-5e9b2da00ad0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvc6z" Nov 04 04:46:37.562593 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.560428 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8446aa71-5638-4b23-8f5b-12283ede21d4-tmp-dir\") pod \"node-resolver-gzctm\" (UID: \"8446aa71-5638-4b23-8f5b-12283ede21d4\") " pod="openshift-dns/node-resolver-gzctm" Nov 04 04:46:37.562593 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.560431 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/844cc12f-a7f6-4327-968e-5e9b2da00ad0-host-cni-netd\") pod \"ovnkube-node-dvc6z\" (UID: \"844cc12f-a7f6-4327-968e-5e9b2da00ad0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvc6z" Nov 04 04:46:37.562593 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.560434 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/844cc12f-a7f6-4327-968e-5e9b2da00ad0-ovnkube-config\") pod \"ovnkube-node-dvc6z\" (UID: \"844cc12f-a7f6-4327-968e-5e9b2da00ad0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvc6z" Nov 04 04:46:37.562593 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.560527 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/22d31d0c-6edf-4351-8f41-a6dd4e02368b-etc-sysctl-conf\") pod \"tuned-c65v8\" (UID: \"22d31d0c-6edf-4351-8f41-a6dd4e02368b\") " pod="openshift-cluster-node-tuning-operator/tuned-c65v8" Nov 04 04:46:37.562593 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.560552 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/22d31d0c-6edf-4351-8f41-a6dd4e02368b-run\") pod \"tuned-c65v8\" (UID: \"22d31d0c-6edf-4351-8f41-a6dd4e02368b\") " pod="openshift-cluster-node-tuning-operator/tuned-c65v8" Nov 04 04:46:37.562593 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.560579 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/22d31d0c-6edf-4351-8f41-a6dd4e02368b-etc-tuned\") pod \"tuned-c65v8\" (UID: \"22d31d0c-6edf-4351-8f41-a6dd4e02368b\") " pod="openshift-cluster-node-tuning-operator/tuned-c65v8" Nov 04 04:46:37.562593 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.560605 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/58be7092-9156-4787-8f49-a173fa0bd5a3-cni-binary-copy\") pod \"multus-7m8fc\" (UID: \"58be7092-9156-4787-8f49-a173fa0bd5a3\") " pod="openshift-multus/multus-7m8fc" Nov 04 04:46:37.562593 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.560630 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/58be7092-9156-4787-8f49-a173fa0bd5a3-hostroot\") pod \"multus-7m8fc\" (UID: \"58be7092-9156-4787-8f49-a173fa0bd5a3\") " pod="openshift-multus/multus-7m8fc" Nov 04 04:46:37.562593 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.560655 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9a06ec07-c72b-43af-aa27-7414aa9c9aba-host\") pod \"node-ca-4z89d\" (UID: \"9a06ec07-c72b-43af-aa27-7414aa9c9aba\") " pod="openshift-image-registry/node-ca-4z89d" Nov 04 04:46:37.562593 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.560682 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/00e0471e-5fe8-4494-8f17-632aee4f6c30-socket-dir\") pod \"aws-ebs-csi-driver-node-f2zrp\" (UID: \"00e0471e-5fe8-4494-8f17-632aee4f6c30\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f2zrp" Nov 04 04:46:37.562593 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.560712 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/00e0471e-5fe8-4494-8f17-632aee4f6c30-sys-fs\") pod \"aws-ebs-csi-driver-node-f2zrp\" (UID: \"00e0471e-5fe8-4494-8f17-632aee4f6c30\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f2zrp" Nov 04 04:46:37.562593 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.560737 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/844cc12f-a7f6-4327-968e-5e9b2da00ad0-run-openvswitch\") pod \"ovnkube-node-dvc6z\" (UID: \"844cc12f-a7f6-4327-968e-5e9b2da00ad0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvc6z" Nov 04 04:46:37.562593 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.560762 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9a06ec07-c72b-43af-aa27-7414aa9c9aba-serviceca\") pod \"node-ca-4z89d\" (UID: \"9a06ec07-c72b-43af-aa27-7414aa9c9aba\") " pod="openshift-image-registry/node-ca-4z89d" Nov 04 04:46:37.562593 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.560785 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/58be7092-9156-4787-8f49-a173fa0bd5a3-host-run-netns\") pod \"multus-7m8fc\" (UID: \"58be7092-9156-4787-8f49-a173fa0bd5a3\") " pod="openshift-multus/multus-7m8fc" Nov 04 04:46:37.563107 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.560812 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hx8dr\" (UniqueName: \"kubernetes.io/projected/4dff9faf-d91f-4ed5-be82-555b9a6a924e-kube-api-access-hx8dr\") pod \"network-metrics-daemon-25flt\" (UID: \"4dff9faf-d91f-4ed5-be82-555b9a6a924e\") " pod="openshift-multus/network-metrics-daemon-25flt" Nov 04 04:46:37.563107 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.560919 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/844cc12f-a7f6-4327-968e-5e9b2da00ad0-ovnkube-config\") pod \"ovnkube-node-dvc6z\" (UID: \"844cc12f-a7f6-4327-968e-5e9b2da00ad0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvc6z" Nov 04 04:46:37.563107 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.560981 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/22d31d0c-6edf-4351-8f41-a6dd4e02368b-run\") pod \"tuned-c65v8\" (UID: \"22d31d0c-6edf-4351-8f41-a6dd4e02368b\") " pod="openshift-cluster-node-tuning-operator/tuned-c65v8" Nov 04 04:46:37.563107 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.560682 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/22d31d0c-6edf-4351-8f41-a6dd4e02368b-etc-sysctl-conf\") pod \"tuned-c65v8\" (UID: \"22d31d0c-6edf-4351-8f41-a6dd4e02368b\") " pod="openshift-cluster-node-tuning-operator/tuned-c65v8" Nov 04 04:46:37.563107 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.561035 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/58be7092-9156-4787-8f49-a173fa0bd5a3-hostroot\") pod \"multus-7m8fc\" (UID: \"58be7092-9156-4787-8f49-a173fa0bd5a3\") " pod="openshift-multus/multus-7m8fc" Nov 04 04:46:37.563107 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.561072 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/00e0471e-5fe8-4494-8f17-632aee4f6c30-sys-fs\") pod \"aws-ebs-csi-driver-node-f2zrp\" (UID: \"00e0471e-5fe8-4494-8f17-632aee4f6c30\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f2zrp" Nov 04 04:46:37.563107 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.561075 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/198e9ab0-f615-4686-9acf-a6242f7e3584-cni-binary-copy\") pod \"multus-additional-cni-plugins-cplsq\" (UID: \"198e9ab0-f615-4686-9acf-a6242f7e3584\") " pod="openshift-multus/multus-additional-cni-plugins-cplsq" Nov 04 04:46:37.563107 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.561127 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/58be7092-9156-4787-8f49-a173fa0bd5a3-host-run-netns\") pod \"multus-7m8fc\" (UID: \"58be7092-9156-4787-8f49-a173fa0bd5a3\") " pod="openshift-multus/multus-7m8fc" Nov 04 04:46:37.563107 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.561078 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9a06ec07-c72b-43af-aa27-7414aa9c9aba-host\") pod \"node-ca-4z89d\" (UID: \"9a06ec07-c72b-43af-aa27-7414aa9c9aba\") " pod="openshift-image-registry/node-ca-4z89d" Nov 04 04:46:37.563107 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.561146 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/844cc12f-a7f6-4327-968e-5e9b2da00ad0-run-openvswitch\") pod \"ovnkube-node-dvc6z\" (UID: \"844cc12f-a7f6-4327-968e-5e9b2da00ad0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvc6z" Nov 04 04:46:37.563107 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.561169 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/58be7092-9156-4787-8f49-a173fa0bd5a3-cni-binary-copy\") pod \"multus-7m8fc\" (UID: \"58be7092-9156-4787-8f49-a173fa0bd5a3\") " pod="openshift-multus/multus-7m8fc" Nov 04 04:46:37.563107 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.561244 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/00e0471e-5fe8-4494-8f17-632aee4f6c30-socket-dir\") pod \"aws-ebs-csi-driver-node-f2zrp\" (UID: \"00e0471e-5fe8-4494-8f17-632aee4f6c30\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f2zrp" Nov 04 04:46:37.563107 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.561411 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9a06ec07-c72b-43af-aa27-7414aa9c9aba-serviceca\") pod \"node-ca-4z89d\" (UID: \"9a06ec07-c72b-43af-aa27-7414aa9c9aba\") " pod="openshift-image-registry/node-ca-4z89d" Nov 04 04:46:37.563107 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.563024 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/22d31d0c-6edf-4351-8f41-a6dd4e02368b-tmp\") pod \"tuned-c65v8\" (UID: \"22d31d0c-6edf-4351-8f41-a6dd4e02368b\") " pod="openshift-cluster-node-tuning-operator/tuned-c65v8" Nov 04 04:46:37.563589 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.563220 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/22d31d0c-6edf-4351-8f41-a6dd4e02368b-etc-tuned\") pod \"tuned-c65v8\" (UID: \"22d31d0c-6edf-4351-8f41-a6dd4e02368b\") " pod="openshift-cluster-node-tuning-operator/tuned-c65v8" Nov 04 04:46:37.563589 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.563304 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/844cc12f-a7f6-4327-968e-5e9b2da00ad0-ovn-node-metrics-cert\") pod \"ovnkube-node-dvc6z\" (UID: \"844cc12f-a7f6-4327-968e-5e9b2da00ad0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvc6z" Nov 04 04:46:37.563589 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.563426 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/23ed4a39-18b5-4bc6-9c7b-8b60a7043ad2-agent-certs\") pod \"konnectivity-agent-7tw97\" (UID: \"23ed4a39-18b5-4bc6-9c7b-8b60a7043ad2\") " pod="kube-system/konnectivity-agent-7tw97" Nov 04 04:46:37.614467 ip-10-0-133-180 kubenswrapper[2579]: E1104 04:46:37.614425 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 04 04:46:37.614467 ip-10-0-133-180 kubenswrapper[2579]: E1104 04:46:37.614456 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 04 04:46:37.614467 ip-10-0-133-180 kubenswrapper[2579]: E1104 04:46:37.614470 2579 projected.go:196] Error preparing data for projected volume kube-api-access-47nf5 for pod openshift-network-diagnostics/network-check-target-8hsp8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 04:46:37.614805 ip-10-0-133-180 kubenswrapper[2579]: E1104 04:46:37.614558 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5f6922fa-9ba7-4d02-b550-606731852fde-kube-api-access-47nf5 podName:5f6922fa-9ba7-4d02-b550-606731852fde nodeName:}" failed. No retries permitted until 2026-11-04 04:46:38.114536688 +0000 UTC m=+2.329784622 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-47nf5" (UniqueName: "kubernetes.io/projected/5f6922fa-9ba7-4d02-b550-606731852fde-kube-api-access-47nf5") pod "network-check-target-8hsp8" (UID: "5f6922fa-9ba7-4d02-b550-606731852fde") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 04:46:37.616481 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.616448 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kz2hd\" (UniqueName: \"kubernetes.io/projected/d737ccc2-5686-4130-9d00-d2f21d5b417f-kube-api-access-kz2hd\") pod \"iptables-alerter-wdk7z\" (UID: \"d737ccc2-5686-4130-9d00-d2f21d5b417f\") " pod="openshift-network-operator/iptables-alerter-wdk7z" Nov 04 04:46:37.617140 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.617107 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlz69\" (UniqueName: \"kubernetes.io/projected/9a06ec07-c72b-43af-aa27-7414aa9c9aba-kube-api-access-mlz69\") pod \"node-ca-4z89d\" (UID: \"9a06ec07-c72b-43af-aa27-7414aa9c9aba\") " pod="openshift-image-registry/node-ca-4z89d" Nov 04 04:46:37.617234 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.617142 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-25g6k\" (UniqueName: \"kubernetes.io/projected/8446aa71-5638-4b23-8f5b-12283ede21d4-kube-api-access-25g6k\") pod \"node-resolver-gzctm\" (UID: \"8446aa71-5638-4b23-8f5b-12283ede21d4\") " pod="openshift-dns/node-resolver-gzctm" Nov 04 04:46:37.617682 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.617661 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbtxf\" (UniqueName: \"kubernetes.io/projected/22d31d0c-6edf-4351-8f41-a6dd4e02368b-kube-api-access-zbtxf\") pod \"tuned-c65v8\" (UID: \"22d31d0c-6edf-4351-8f41-a6dd4e02368b\") " pod="openshift-cluster-node-tuning-operator/tuned-c65v8" Nov 04 04:46:37.618082 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.618062 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4xvx\" (UniqueName: \"kubernetes.io/projected/58be7092-9156-4787-8f49-a173fa0bd5a3-kube-api-access-w4xvx\") pod \"multus-7m8fc\" (UID: \"58be7092-9156-4787-8f49-a173fa0bd5a3\") " pod="openshift-multus/multus-7m8fc" Nov 04 04:46:37.618472 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.618448 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hx8dr\" (UniqueName: \"kubernetes.io/projected/4dff9faf-d91f-4ed5-be82-555b9a6a924e-kube-api-access-hx8dr\") pod \"network-metrics-daemon-25flt\" (UID: \"4dff9faf-d91f-4ed5-be82-555b9a6a924e\") " pod="openshift-multus/network-metrics-daemon-25flt" Nov 04 04:46:37.618742 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.618719 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nch9\" (UniqueName: \"kubernetes.io/projected/844cc12f-a7f6-4327-968e-5e9b2da00ad0-kube-api-access-6nch9\") pod \"ovnkube-node-dvc6z\" (UID: \"844cc12f-a7f6-4327-968e-5e9b2da00ad0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvc6z" Nov 04 04:46:37.618959 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.618940 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccfxs\" (UniqueName: \"kubernetes.io/projected/198e9ab0-f615-4686-9acf-a6242f7e3584-kube-api-access-ccfxs\") pod \"multus-additional-cni-plugins-cplsq\" (UID: \"198e9ab0-f615-4686-9acf-a6242f7e3584\") " pod="openshift-multus/multus-additional-cni-plugins-cplsq" Nov 04 04:46:37.619693 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.619670 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gftxp\" (UniqueName: \"kubernetes.io/projected/00e0471e-5fe8-4494-8f17-632aee4f6c30-kube-api-access-gftxp\") pod \"aws-ebs-csi-driver-node-f2zrp\" (UID: \"00e0471e-5fe8-4494-8f17-632aee4f6c30\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f2zrp" Nov 04 04:46:37.737919 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.737852 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-7tw97" Nov 04 04:46:37.743013 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.742990 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-cplsq" Nov 04 04:46:37.763783 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.763750 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f2zrp" Nov 04 04:46:37.774450 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.774425 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-wdk7z" Nov 04 04:46:37.791579 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.791554 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-dvc6z" Nov 04 04:46:37.810164 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.810146 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-c65v8" Nov 04 04:46:37.814157 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.814138 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-7m8fc" Nov 04 04:46:37.826047 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.826027 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-gzctm" Nov 04 04:46:37.826679 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:37.826655 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22d31d0c_6edf_4351_8f41_a6dd4e02368b.slice/crio-a3364a59938817d67809b86557a1cf7927287ef22809b4fc291bc1f04372ad72 WatchSource:0}: Error finding container a3364a59938817d67809b86557a1cf7927287ef22809b4fc291bc1f04372ad72: Status 404 returned error can't find the container with id a3364a59938817d67809b86557a1cf7927287ef22809b4fc291bc1f04372ad72 Nov 04 04:46:37.831731 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:37.831707 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6aa9201e961e4ed5a49af4c50453ef92.slice/crio-61c7ccd8c37f4b1903256dd5a442b7df3c9fe02b842028cc54b884875b60942a WatchSource:0}: Error finding container 61c7ccd8c37f4b1903256dd5a442b7df3c9fe02b842028cc54b884875b60942a: Status 404 returned error can't find the container with id 61c7ccd8c37f4b1903256dd5a442b7df3c9fe02b842028cc54b884875b60942a Nov 04 04:46:37.832014 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.831989 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-4z89d" Nov 04 04:46:37.835434 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:37.835415 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 04 04:46:37.840481 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:37.840454 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8446aa71_5638_4b23_8f5b_12283ede21d4.slice/crio-07780832d01bfc20e7d9d4e0447f6b33fa59657b03493e9cf6835dbff988e3a7 WatchSource:0}: Error finding container 07780832d01bfc20e7d9d4e0447f6b33fa59657b03493e9cf6835dbff988e3a7: Status 404 returned error can't find the container with id 07780832d01bfc20e7d9d4e0447f6b33fa59657b03493e9cf6835dbff988e3a7 Nov 04 04:46:37.843650 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:37.843626 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a06ec07_c72b_43af_aa27_7414aa9c9aba.slice/crio-699641a78e73f5bb420a0efb52bc044af718fdc4a940b1c5f2a5a01cd780941c WatchSource:0}: Error finding container 699641a78e73f5bb420a0efb52bc044af718fdc4a940b1c5f2a5a01cd780941c: Status 404 returned error can't find the container with id 699641a78e73f5bb420a0efb52bc044af718fdc4a940b1c5f2a5a01cd780941c Nov 04 04:46:38.013690 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:38.013607 2579 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 04 04:46:38.065577 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:38.065538 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4dff9faf-d91f-4ed5-be82-555b9a6a924e-metrics-certs\") pod \"network-metrics-daemon-25flt\" (UID: \"4dff9faf-d91f-4ed5-be82-555b9a6a924e\") " pod="openshift-multus/network-metrics-daemon-25flt" Nov 04 04:46:38.065751 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:38.065607 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c79eff94-6543-4313-b59b-502ca4089e84-original-pull-secret\") pod \"global-pull-secret-syncer-gfjqz\" (UID: \"c79eff94-6543-4313-b59b-502ca4089e84\") " pod="kube-system/global-pull-secret-syncer-gfjqz" Nov 04 04:46:38.065751 ip-10-0-133-180 kubenswrapper[2579]: E1104 04:46:38.065703 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 04 04:46:38.065751 ip-10-0-133-180 kubenswrapper[2579]: E1104 04:46:38.065703 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 04:46:38.065848 ip-10-0-133-180 kubenswrapper[2579]: E1104 04:46:38.065766 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c79eff94-6543-4313-b59b-502ca4089e84-original-pull-secret podName:c79eff94-6543-4313-b59b-502ca4089e84 nodeName:}" failed. No retries permitted until 2026-11-04 04:46:39.065747813 +0000 UTC m=+3.280995731 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c79eff94-6543-4313-b59b-502ca4089e84-original-pull-secret") pod "global-pull-secret-syncer-gfjqz" (UID: "c79eff94-6543-4313-b59b-502ca4089e84") : object "kube-system"/"original-pull-secret" not registered Nov 04 04:46:38.065848 ip-10-0-133-180 kubenswrapper[2579]: E1104 04:46:38.065782 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4dff9faf-d91f-4ed5-be82-555b9a6a924e-metrics-certs podName:4dff9faf-d91f-4ed5-be82-555b9a6a924e nodeName:}" failed. No retries permitted until 2026-11-04 04:46:39.065774944 +0000 UTC m=+3.281022861 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4dff9faf-d91f-4ed5-be82-555b9a6a924e-metrics-certs") pod "network-metrics-daemon-25flt" (UID: "4dff9faf-d91f-4ed5-be82-555b9a6a924e") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 04:46:38.166910 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:38.166871 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47nf5\" (UniqueName: \"kubernetes.io/projected/5f6922fa-9ba7-4d02-b550-606731852fde-kube-api-access-47nf5\") pod \"network-check-target-8hsp8\" (UID: \"5f6922fa-9ba7-4d02-b550-606731852fde\") " pod="openshift-network-diagnostics/network-check-target-8hsp8" Nov 04 04:46:38.167536 ip-10-0-133-180 kubenswrapper[2579]: E1104 04:46:38.167051 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 04 04:46:38.167536 ip-10-0-133-180 kubenswrapper[2579]: E1104 04:46:38.167074 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 04 04:46:38.167536 ip-10-0-133-180 kubenswrapper[2579]: E1104 04:46:38.167085 2579 projected.go:196] Error preparing data for projected volume kube-api-access-47nf5 for pod openshift-network-diagnostics/network-check-target-8hsp8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 04:46:38.167536 ip-10-0-133-180 kubenswrapper[2579]: E1104 04:46:38.167164 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5f6922fa-9ba7-4d02-b550-606731852fde-kube-api-access-47nf5 podName:5f6922fa-9ba7-4d02-b550-606731852fde nodeName:}" failed. No retries permitted until 2026-11-04 04:46:39.167129687 +0000 UTC m=+3.382377604 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-47nf5" (UniqueName: "kubernetes.io/projected/5f6922fa-9ba7-4d02-b550-606731852fde-kube-api-access-47nf5") pod "network-check-target-8hsp8" (UID: "5f6922fa-9ba7-4d02-b550-606731852fde") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 04:46:38.235938 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:38.235908 2579 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 04 04:46:38.293515 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:38.293411 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-4z89d" event={"ID":"9a06ec07-c72b-43af-aa27-7414aa9c9aba","Type":"ContainerStarted","Data":"699641a78e73f5bb420a0efb52bc044af718fdc4a940b1c5f2a5a01cd780941c"} Nov 04 04:46:38.294426 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:38.294391 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-gzctm" event={"ID":"8446aa71-5638-4b23-8f5b-12283ede21d4","Type":"ContainerStarted","Data":"07780832d01bfc20e7d9d4e0447f6b33fa59657b03493e9cf6835dbff988e3a7"} Nov 04 04:46:38.295440 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:38.295415 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-7m8fc" event={"ID":"58be7092-9156-4787-8f49-a173fa0bd5a3","Type":"ContainerStarted","Data":"319071750bc643809077f76db9d2220e29cc5c1a3924dd48f8a4608fb71cd798"} Nov 04 04:46:38.296433 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:38.296413 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-c65v8" event={"ID":"22d31d0c-6edf-4351-8f41-a6dd4e02368b","Type":"ContainerStarted","Data":"a3364a59938817d67809b86557a1cf7927287ef22809b4fc291bc1f04372ad72"} Nov 04 04:46:38.297518 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:38.297493 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-180.us-west-1.compute.internal" event={"ID":"46addbc2e24e99ce85548c61cc2707a9","Type":"ContainerStarted","Data":"9a9d05d8927347bce1aaee6783a92cb2e0d8550b2ab0cabbc0015124f74b024e"} Nov 04 04:46:38.298641 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:38.298613 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-180.us-west-1.compute.internal" event={"ID":"6aa9201e961e4ed5a49af4c50453ef92","Type":"ContainerStarted","Data":"61c7ccd8c37f4b1903256dd5a442b7df3c9fe02b842028cc54b884875b60942a"} Nov 04 04:46:38.390598 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:38.390562 2579 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 04 04:46:38.508451 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:38.508392 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-03 04:41:38 +0000 UTC" deadline="2028-06-06 16:15:16.767984632 +0000 UTC" Nov 04 04:46:38.508451 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:38.508441 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13931h28m38.259546643s" Nov 04 04:46:38.608036 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:38.607997 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00e0471e_5fe8_4494_8f17_632aee4f6c30.slice/crio-1e385cdda0d1bccd6f07d5afcdaee46c8a729a2ec16ed217e691159218ab711f WatchSource:0}: Error finding container 1e385cdda0d1bccd6f07d5afcdaee46c8a729a2ec16ed217e691159218ab711f: Status 404 returned error can't find the container with id 1e385cdda0d1bccd6f07d5afcdaee46c8a729a2ec16ed217e691159218ab711f Nov 04 04:46:38.652842 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:38.652808 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod198e9ab0_f615_4686_9acf_a6242f7e3584.slice/crio-08cbd96522b2f06bb323ee7dac9b9d159935e1ce665c5d9dd78e426e9a662fcf WatchSource:0}: Error finding container 08cbd96522b2f06bb323ee7dac9b9d159935e1ce665c5d9dd78e426e9a662fcf: Status 404 returned error can't find the container with id 08cbd96522b2f06bb323ee7dac9b9d159935e1ce665c5d9dd78e426e9a662fcf Nov 04 04:46:38.701293 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:38.701262 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod844cc12f_a7f6_4327_968e_5e9b2da00ad0.slice/crio-6d3be9a0db0ee308071b3422d53c6844098593216d463c6863ec775c3e47e798 WatchSource:0}: Error finding container 6d3be9a0db0ee308071b3422d53c6844098593216d463c6863ec775c3e47e798: Status 404 returned error can't find the container with id 6d3be9a0db0ee308071b3422d53c6844098593216d463c6863ec775c3e47e798 Nov 04 04:46:38.721622 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:38.721597 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23ed4a39_18b5_4bc6_9c7b_8b60a7043ad2.slice/crio-9025e93209d4a5f767ba5a97ab0f507331a3e676c4eb362351de7da833c8c582 WatchSource:0}: Error finding container 9025e93209d4a5f767ba5a97ab0f507331a3e676c4eb362351de7da833c8c582: Status 404 returned error can't find the container with id 9025e93209d4a5f767ba5a97ab0f507331a3e676c4eb362351de7da833c8c582 Nov 04 04:46:38.823579 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:46:38.823530 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd737ccc2_5686_4130_9d00_d2f21d5b417f.slice/crio-573341ae49ba363b5674428f3452af54a79baeec54fbc99e3391040e0213b64e WatchSource:0}: Error finding container 573341ae49ba363b5674428f3452af54a79baeec54fbc99e3391040e0213b64e: Status 404 returned error can't find the container with id 573341ae49ba363b5674428f3452af54a79baeec54fbc99e3391040e0213b64e Nov 04 04:46:39.075991 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:39.075665 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4dff9faf-d91f-4ed5-be82-555b9a6a924e-metrics-certs\") pod \"network-metrics-daemon-25flt\" (UID: \"4dff9faf-d91f-4ed5-be82-555b9a6a924e\") " pod="openshift-multus/network-metrics-daemon-25flt" Nov 04 04:46:39.075991 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:39.075746 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c79eff94-6543-4313-b59b-502ca4089e84-original-pull-secret\") pod \"global-pull-secret-syncer-gfjqz\" (UID: \"c79eff94-6543-4313-b59b-502ca4089e84\") " pod="kube-system/global-pull-secret-syncer-gfjqz" Nov 04 04:46:39.075991 ip-10-0-133-180 kubenswrapper[2579]: E1104 04:46:39.075855 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 04 04:46:39.075991 ip-10-0-133-180 kubenswrapper[2579]: E1104 04:46:39.075906 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c79eff94-6543-4313-b59b-502ca4089e84-original-pull-secret podName:c79eff94-6543-4313-b59b-502ca4089e84 nodeName:}" failed. No retries permitted until 2026-11-04 04:46:41.075892154 +0000 UTC m=+5.291140087 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c79eff94-6543-4313-b59b-502ca4089e84-original-pull-secret") pod "global-pull-secret-syncer-gfjqz" (UID: "c79eff94-6543-4313-b59b-502ca4089e84") : object "kube-system"/"original-pull-secret" not registered Nov 04 04:46:39.075991 ip-10-0-133-180 kubenswrapper[2579]: E1104 04:46:39.075951 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 04:46:39.075991 ip-10-0-133-180 kubenswrapper[2579]: E1104 04:46:39.075973 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4dff9faf-d91f-4ed5-be82-555b9a6a924e-metrics-certs podName:4dff9faf-d91f-4ed5-be82-555b9a6a924e nodeName:}" failed. No retries permitted until 2026-11-04 04:46:41.075965103 +0000 UTC m=+5.291213021 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4dff9faf-d91f-4ed5-be82-555b9a6a924e-metrics-certs") pod "network-metrics-daemon-25flt" (UID: "4dff9faf-d91f-4ed5-be82-555b9a6a924e") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 04:46:39.176176 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:39.176079 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47nf5\" (UniqueName: \"kubernetes.io/projected/5f6922fa-9ba7-4d02-b550-606731852fde-kube-api-access-47nf5\") pod \"network-check-target-8hsp8\" (UID: \"5f6922fa-9ba7-4d02-b550-606731852fde\") " pod="openshift-network-diagnostics/network-check-target-8hsp8" Nov 04 04:46:39.176927 ip-10-0-133-180 kubenswrapper[2579]: E1104 04:46:39.176316 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 04 04:46:39.176927 ip-10-0-133-180 kubenswrapper[2579]: E1104 04:46:39.176334 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 04 04:46:39.176927 ip-10-0-133-180 kubenswrapper[2579]: E1104 04:46:39.176347 2579 projected.go:196] Error preparing data for projected volume kube-api-access-47nf5 for pod openshift-network-diagnostics/network-check-target-8hsp8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 04:46:39.176927 ip-10-0-133-180 kubenswrapper[2579]: E1104 04:46:39.176430 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5f6922fa-9ba7-4d02-b550-606731852fde-kube-api-access-47nf5 podName:5f6922fa-9ba7-4d02-b550-606731852fde nodeName:}" failed. No retries permitted until 2026-11-04 04:46:41.176408648 +0000 UTC m=+5.391656577 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-47nf5" (UniqueName: "kubernetes.io/projected/5f6922fa-9ba7-4d02-b550-606731852fde-kube-api-access-47nf5") pod "network-check-target-8hsp8" (UID: "5f6922fa-9ba7-4d02-b550-606731852fde") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 04:46:39.289743 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:39.289173 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8hsp8" Nov 04 04:46:39.289743 ip-10-0-133-180 kubenswrapper[2579]: E1104 04:46:39.289320 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8hsp8" podUID="5f6922fa-9ba7-4d02-b550-606731852fde" Nov 04 04:46:39.289743 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:39.289430 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-25flt" Nov 04 04:46:39.289743 ip-10-0-133-180 kubenswrapper[2579]: E1104 04:46:39.289550 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-25flt" podUID="4dff9faf-d91f-4ed5-be82-555b9a6a924e" Nov 04 04:46:39.289743 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:39.289608 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gfjqz" Nov 04 04:46:39.289743 ip-10-0-133-180 kubenswrapper[2579]: E1104 04:46:39.289683 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gfjqz" podUID="c79eff94-6543-4313-b59b-502ca4089e84" Nov 04 04:46:39.309221 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:39.309140 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-wdk7z" event={"ID":"d737ccc2-5686-4130-9d00-d2f21d5b417f","Type":"ContainerStarted","Data":"573341ae49ba363b5674428f3452af54a79baeec54fbc99e3391040e0213b64e"} Nov 04 04:46:39.311632 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:39.311543 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-7tw97" event={"ID":"23ed4a39-18b5-4bc6-9c7b-8b60a7043ad2","Type":"ContainerStarted","Data":"9025e93209d4a5f767ba5a97ab0f507331a3e676c4eb362351de7da833c8c582"} Nov 04 04:46:39.313813 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:39.313737 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cplsq" event={"ID":"198e9ab0-f615-4686-9acf-a6242f7e3584","Type":"ContainerStarted","Data":"08cbd96522b2f06bb323ee7dac9b9d159935e1ce665c5d9dd78e426e9a662fcf"} Nov 04 04:46:39.321361 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:39.321212 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dvc6z" event={"ID":"844cc12f-a7f6-4327-968e-5e9b2da00ad0","Type":"ContainerStarted","Data":"6d3be9a0db0ee308071b3422d53c6844098593216d463c6863ec775c3e47e798"} Nov 04 04:46:39.324426 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:39.324393 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f2zrp" event={"ID":"00e0471e-5fe8-4494-8f17-632aee4f6c30","Type":"ContainerStarted","Data":"1e385cdda0d1bccd6f07d5afcdaee46c8a729a2ec16ed217e691159218ab711f"} Nov 04 04:46:39.509503 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:39.509424 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-03 04:41:38 +0000 UTC" deadline="2028-07-06 12:38:27.278194463 +0000 UTC" Nov 04 04:46:39.509503 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:39.509491 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14647h51m47.76870782s" Nov 04 04:46:41.091559 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:41.091518 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c79eff94-6543-4313-b59b-502ca4089e84-original-pull-secret\") pod \"global-pull-secret-syncer-gfjqz\" (UID: \"c79eff94-6543-4313-b59b-502ca4089e84\") " pod="kube-system/global-pull-secret-syncer-gfjqz" Nov 04 04:46:41.092243 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:41.091598 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4dff9faf-d91f-4ed5-be82-555b9a6a924e-metrics-certs\") pod \"network-metrics-daemon-25flt\" (UID: \"4dff9faf-d91f-4ed5-be82-555b9a6a924e\") " pod="openshift-multus/network-metrics-daemon-25flt" Nov 04 04:46:41.092243 ip-10-0-133-180 kubenswrapper[2579]: E1104 04:46:41.091773 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 04:46:41.092243 ip-10-0-133-180 kubenswrapper[2579]: E1104 04:46:41.091839 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4dff9faf-d91f-4ed5-be82-555b9a6a924e-metrics-certs podName:4dff9faf-d91f-4ed5-be82-555b9a6a924e nodeName:}" failed. No retries permitted until 2026-11-04 04:46:45.091819945 +0000 UTC m=+9.307067881 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4dff9faf-d91f-4ed5-be82-555b9a6a924e-metrics-certs") pod "network-metrics-daemon-25flt" (UID: "4dff9faf-d91f-4ed5-be82-555b9a6a924e") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 04:46:41.092243 ip-10-0-133-180 kubenswrapper[2579]: E1104 04:46:41.091903 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 04 04:46:41.092243 ip-10-0-133-180 kubenswrapper[2579]: E1104 04:46:41.091934 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c79eff94-6543-4313-b59b-502ca4089e84-original-pull-secret podName:c79eff94-6543-4313-b59b-502ca4089e84 nodeName:}" failed. No retries permitted until 2026-11-04 04:46:45.091923074 +0000 UTC m=+9.307170992 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c79eff94-6543-4313-b59b-502ca4089e84-original-pull-secret") pod "global-pull-secret-syncer-gfjqz" (UID: "c79eff94-6543-4313-b59b-502ca4089e84") : object "kube-system"/"original-pull-secret" not registered Nov 04 04:46:41.192816 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:41.192775 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47nf5\" (UniqueName: \"kubernetes.io/projected/5f6922fa-9ba7-4d02-b550-606731852fde-kube-api-access-47nf5\") pod \"network-check-target-8hsp8\" (UID: \"5f6922fa-9ba7-4d02-b550-606731852fde\") " pod="openshift-network-diagnostics/network-check-target-8hsp8" Nov 04 04:46:41.193030 ip-10-0-133-180 kubenswrapper[2579]: E1104 04:46:41.192984 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 04 04:46:41.193030 ip-10-0-133-180 kubenswrapper[2579]: E1104 04:46:41.193005 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 04 04:46:41.193030 ip-10-0-133-180 kubenswrapper[2579]: E1104 04:46:41.193030 2579 projected.go:196] Error preparing data for projected volume kube-api-access-47nf5 for pod openshift-network-diagnostics/network-check-target-8hsp8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 04:46:41.193208 ip-10-0-133-180 kubenswrapper[2579]: E1104 04:46:41.193090 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5f6922fa-9ba7-4d02-b550-606731852fde-kube-api-access-47nf5 podName:5f6922fa-9ba7-4d02-b550-606731852fde nodeName:}" failed. No retries permitted until 2026-11-04 04:46:45.193071346 +0000 UTC m=+9.408319278 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-47nf5" (UniqueName: "kubernetes.io/projected/5f6922fa-9ba7-4d02-b550-606731852fde-kube-api-access-47nf5") pod "network-check-target-8hsp8" (UID: "5f6922fa-9ba7-4d02-b550-606731852fde") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 04:46:41.289685 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:41.289215 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gfjqz" Nov 04 04:46:41.289685 ip-10-0-133-180 kubenswrapper[2579]: E1104 04:46:41.289382 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gfjqz" podUID="c79eff94-6543-4313-b59b-502ca4089e84" Nov 04 04:46:41.289685 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:41.289450 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8hsp8" Nov 04 04:46:41.289685 ip-10-0-133-180 kubenswrapper[2579]: E1104 04:46:41.289518 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8hsp8" podUID="5f6922fa-9ba7-4d02-b550-606731852fde" Nov 04 04:46:41.289685 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:41.289567 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-25flt" Nov 04 04:46:41.289685 ip-10-0-133-180 kubenswrapper[2579]: E1104 04:46:41.289645 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-25flt" podUID="4dff9faf-d91f-4ed5-be82-555b9a6a924e" Nov 04 04:46:43.292323 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:43.292265 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8hsp8" Nov 04 04:46:43.292892 ip-10-0-133-180 kubenswrapper[2579]: E1104 04:46:43.292436 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8hsp8" podUID="5f6922fa-9ba7-4d02-b550-606731852fde" Nov 04 04:46:43.292892 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:43.292524 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-25flt" Nov 04 04:46:43.292892 ip-10-0-133-180 kubenswrapper[2579]: E1104 04:46:43.292615 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-25flt" podUID="4dff9faf-d91f-4ed5-be82-555b9a6a924e" Nov 04 04:46:43.292892 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:43.292670 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gfjqz" Nov 04 04:46:43.292892 ip-10-0-133-180 kubenswrapper[2579]: E1104 04:46:43.292740 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gfjqz" podUID="c79eff94-6543-4313-b59b-502ca4089e84" Nov 04 04:46:43.357512 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:43.357470 2579 generic.go:356] "Generic (PLEG): container finished" podID="198e9ab0-f615-4686-9acf-a6242f7e3584" containerID="68a3bfbe4c7af69a9d5b7f06c7eb1b4d323f201b91a0b6fc2d4b24cf335028cb" exitCode=0 Nov 04 04:46:43.357698 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:43.357576 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cplsq" event={"ID":"198e9ab0-f615-4686-9acf-a6242f7e3584","Type":"ContainerDied","Data":"68a3bfbe4c7af69a9d5b7f06c7eb1b4d323f201b91a0b6fc2d4b24cf335028cb"} Nov 04 04:46:43.373045 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:43.372270 2579 generic.go:356] "Generic (PLEG): container finished" podID="46addbc2e24e99ce85548c61cc2707a9" containerID="17a36ef60eab503b266d126674f81064202a9c2fe4867d3aba215a0eaced37eb" exitCode=0 Nov 04 04:46:43.373045 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:43.372355 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-180.us-west-1.compute.internal" event={"ID":"46addbc2e24e99ce85548c61cc2707a9","Type":"ContainerDied","Data":"17a36ef60eab503b266d126674f81064202a9c2fe4867d3aba215a0eaced37eb"} Nov 04 04:46:43.382373 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:43.382338 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-180.us-west-1.compute.internal" event={"ID":"6aa9201e961e4ed5a49af4c50453ef92","Type":"ContainerStarted","Data":"12ad69f5d7bf82e2961b5d2ff6f0d403750c49e510a905db052fee4e2272789a"} Nov 04 04:46:44.388658 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:44.388550 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-7tw97" event={"ID":"23ed4a39-18b5-4bc6-9c7b-8b60a7043ad2","Type":"ContainerStarted","Data":"55dac150764656d132a13233d066fb49c206fab4a6f8f7199dbdef775b5e212b"} Nov 04 04:46:44.394638 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:44.394088 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-4z89d" event={"ID":"9a06ec07-c72b-43af-aa27-7414aa9c9aba","Type":"ContainerStarted","Data":"b7a78acb2b461ff4929a3e8a65d4d173153c1e279ab2f9a791927858d833521b"} Nov 04 04:46:44.398246 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:44.398194 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-180.us-west-1.compute.internal" event={"ID":"46addbc2e24e99ce85548c61cc2707a9","Type":"ContainerStarted","Data":"8d3db7d02613eb50a6b709814f2b10edb68088541dbcf0f26579d907c4033b2f"} Nov 04 04:46:44.510854 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:44.510791 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-180.us-west-1.compute.internal" podStartSLOduration=7.510770455 podStartE2EDuration="7.510770455s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 04:46:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:46:43.735628841 +0000 UTC m=+7.950876785" watchObservedRunningTime="2026-11-04 04:46:44.510770455 +0000 UTC m=+8.726018401" Nov 04 04:46:44.638492 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:44.637572 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-7tw97" podStartSLOduration=2.549370561 podStartE2EDuration="7.637548096s" totalImagesPullingTime="5.088177535s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 04:46:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:46:44.51150234 +0000 UTC m=+8.726750283" watchObservedRunningTime="2026-11-04 04:46:44.637548096 +0000 UTC m=+8.852796040" Nov 04 04:46:44.760439 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:44.760299 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-4z89d" podStartSLOduration=1.97566945 podStartE2EDuration="7.760275606s" totalImagesPullingTime="5.784606156s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 04:46:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:46:44.638739658 +0000 UTC m=+8.853987602" watchObservedRunningTime="2026-11-04 04:46:44.760275606 +0000 UTC m=+8.975523554" Nov 04 04:46:44.760662 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:44.760441 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-180.us-west-1.compute.internal" podStartSLOduration=7.760432791 podStartE2EDuration="7.760432791s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 04:46:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:46:44.759303429 +0000 UTC m=+8.974551371" watchObservedRunningTime="2026-11-04 04:46:44.760432791 +0000 UTC m=+8.975680736" Nov 04 04:46:45.126561 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:45.126460 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c79eff94-6543-4313-b59b-502ca4089e84-original-pull-secret\") pod \"global-pull-secret-syncer-gfjqz\" (UID: \"c79eff94-6543-4313-b59b-502ca4089e84\") " pod="kube-system/global-pull-secret-syncer-gfjqz" Nov 04 04:46:45.126732 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:45.126581 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4dff9faf-d91f-4ed5-be82-555b9a6a924e-metrics-certs\") pod \"network-metrics-daemon-25flt\" (UID: \"4dff9faf-d91f-4ed5-be82-555b9a6a924e\") " pod="openshift-multus/network-metrics-daemon-25flt" Nov 04 04:46:45.126798 ip-10-0-133-180 kubenswrapper[2579]: E1104 04:46:45.126735 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 04:46:45.126798 ip-10-0-133-180 kubenswrapper[2579]: E1104 04:46:45.126797 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4dff9faf-d91f-4ed5-be82-555b9a6a924e-metrics-certs podName:4dff9faf-d91f-4ed5-be82-555b9a6a924e nodeName:}" failed. No retries permitted until 2026-11-04 04:46:53.126777172 +0000 UTC m=+17.342025093 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4dff9faf-d91f-4ed5-be82-555b9a6a924e-metrics-certs") pod "network-metrics-daemon-25flt" (UID: "4dff9faf-d91f-4ed5-be82-555b9a6a924e") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 04:46:45.127369 ip-10-0-133-180 kubenswrapper[2579]: E1104 04:46:45.127253 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 04 04:46:45.127369 ip-10-0-133-180 kubenswrapper[2579]: E1104 04:46:45.127306 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c79eff94-6543-4313-b59b-502ca4089e84-original-pull-secret podName:c79eff94-6543-4313-b59b-502ca4089e84 nodeName:}" failed. No retries permitted until 2026-11-04 04:46:53.127290485 +0000 UTC m=+17.342538408 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c79eff94-6543-4313-b59b-502ca4089e84-original-pull-secret") pod "global-pull-secret-syncer-gfjqz" (UID: "c79eff94-6543-4313-b59b-502ca4089e84") : object "kube-system"/"original-pull-secret" not registered Nov 04 04:46:45.227204 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:45.227167 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47nf5\" (UniqueName: \"kubernetes.io/projected/5f6922fa-9ba7-4d02-b550-606731852fde-kube-api-access-47nf5\") pod \"network-check-target-8hsp8\" (UID: \"5f6922fa-9ba7-4d02-b550-606731852fde\") " pod="openshift-network-diagnostics/network-check-target-8hsp8" Nov 04 04:46:45.227423 ip-10-0-133-180 kubenswrapper[2579]: E1104 04:46:45.227330 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 04 04:46:45.227423 ip-10-0-133-180 kubenswrapper[2579]: E1104 04:46:45.227381 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 04 04:46:45.227423 ip-10-0-133-180 kubenswrapper[2579]: E1104 04:46:45.227394 2579 projected.go:196] Error preparing data for projected volume kube-api-access-47nf5 for pod openshift-network-diagnostics/network-check-target-8hsp8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 04:46:45.227594 ip-10-0-133-180 kubenswrapper[2579]: E1104 04:46:45.227453 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5f6922fa-9ba7-4d02-b550-606731852fde-kube-api-access-47nf5 podName:5f6922fa-9ba7-4d02-b550-606731852fde nodeName:}" failed. No retries permitted until 2026-11-04 04:46:53.227433755 +0000 UTC m=+17.442681679 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-47nf5" (UniqueName: "kubernetes.io/projected/5f6922fa-9ba7-4d02-b550-606731852fde-kube-api-access-47nf5") pod "network-check-target-8hsp8" (UID: "5f6922fa-9ba7-4d02-b550-606731852fde") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 04:46:45.289071 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:45.289034 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gfjqz" Nov 04 04:46:45.289255 ip-10-0-133-180 kubenswrapper[2579]: E1104 04:46:45.289180 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gfjqz" podUID="c79eff94-6543-4313-b59b-502ca4089e84" Nov 04 04:46:45.289255 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:45.289239 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8hsp8" Nov 04 04:46:45.289382 ip-10-0-133-180 kubenswrapper[2579]: E1104 04:46:45.289311 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8hsp8" podUID="5f6922fa-9ba7-4d02-b550-606731852fde" Nov 04 04:46:45.289468 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:45.289357 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-25flt" Nov 04 04:46:45.289528 ip-10-0-133-180 kubenswrapper[2579]: E1104 04:46:45.289477 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-25flt" podUID="4dff9faf-d91f-4ed5-be82-555b9a6a924e" Nov 04 04:46:46.047872 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:46.047799 2579 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-7tw97" Nov 04 04:46:46.048470 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:46.048446 2579 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-7tw97" Nov 04 04:46:46.406915 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:46.406837 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f2zrp" event={"ID":"00e0471e-5fe8-4494-8f17-632aee4f6c30","Type":"ContainerStarted","Data":"810030737f2b7426799a5b9338c8290e2c6ab1ea9254a97a83b2f9235328ac19"} Nov 04 04:46:47.289491 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:47.289412 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8hsp8" Nov 04 04:46:47.289985 ip-10-0-133-180 kubenswrapper[2579]: E1104 04:46:47.289559 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8hsp8" podUID="5f6922fa-9ba7-4d02-b550-606731852fde" Nov 04 04:46:47.289985 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:47.289977 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-25flt" Nov 04 04:46:47.290129 ip-10-0-133-180 kubenswrapper[2579]: E1104 04:46:47.290100 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-25flt" podUID="4dff9faf-d91f-4ed5-be82-555b9a6a924e" Nov 04 04:46:47.290180 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:47.290162 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gfjqz" Nov 04 04:46:47.290254 ip-10-0-133-180 kubenswrapper[2579]: E1104 04:46:47.290236 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gfjqz" podUID="c79eff94-6543-4313-b59b-502ca4089e84" Nov 04 04:46:47.413817 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:47.413784 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-gzctm" event={"ID":"8446aa71-5638-4b23-8f5b-12283ede21d4","Type":"ContainerStarted","Data":"63a4891010e443f50f782ce6eb0a2f8952cabaed030d0cf28cd1c54db619be71"} Nov 04 04:46:47.416964 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:47.416942 2579 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 04 04:46:47.417614 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:47.417577 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-7m8fc" event={"ID":"58be7092-9156-4787-8f49-a173fa0bd5a3","Type":"ContainerStarted","Data":"459f9b24d757a9578a7ed999988aae667bc3eaf7101c467b11eb4e3cb844dbcd"} Nov 04 04:46:47.667092 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:47.667033 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-gzctm" podStartSLOduration=1.619616331 podStartE2EDuration="10.667018243s" totalImagesPullingTime="9.047401912s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 04:46:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:46:47.54903961 +0000 UTC m=+11.764287548" watchObservedRunningTime="2026-11-04 04:46:47.667018243 +0000 UTC m=+11.882266183" Nov 04 04:46:47.667665 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:47.667630 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-7m8fc" podStartSLOduration=1.840631389 podStartE2EDuration="10.667619542s" totalImagesPullingTime="8.826988153s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 04:46:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:46:47.666857124 +0000 UTC m=+11.882105064" watchObservedRunningTime="2026-11-04 04:46:47.667619542 +0000 UTC m=+11.882867482" Nov 04 04:46:47.820869 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:47.820730 2579 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 04 04:46:48.217345 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:48.217144 2579 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-04T04:46:47.820751941Z","UUID":"4c1905b0-689d-476d-aafa-5a020dce2b37","Handler":null,"Name":"","Endpoint":""} Nov 04 04:46:48.218930 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:48.218912 2579 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 04 04:46:48.219069 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:48.218950 2579 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 04 04:46:48.421990 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:48.421947 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f2zrp" event={"ID":"00e0471e-5fe8-4494-8f17-632aee4f6c30","Type":"ContainerStarted","Data":"07531be308c9d14afa0fe0f7b84bdb2e814007ac34aa9adeab8ce5599c51bdef"} Nov 04 04:46:48.424478 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:48.424447 2579 generic.go:356] "Generic (PLEG): container finished" podID="198e9ab0-f615-4686-9acf-a6242f7e3584" containerID="590d8d4be63d5ccffc60b4f1f9153c9b75ffb08cea1c4afacabd879c04ba5bfa" exitCode=0 Nov 04 04:46:48.424629 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:48.424581 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cplsq" event={"ID":"198e9ab0-f615-4686-9acf-a6242f7e3584","Type":"ContainerDied","Data":"590d8d4be63d5ccffc60b4f1f9153c9b75ffb08cea1c4afacabd879c04ba5bfa"} Nov 04 04:46:49.289544 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:49.289511 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gfjqz" Nov 04 04:46:49.289715 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:49.289603 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-25flt" Nov 04 04:46:49.289715 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:49.289626 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8hsp8" Nov 04 04:46:49.289826 ip-10-0-133-180 kubenswrapper[2579]: E1104 04:46:49.289730 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gfjqz" podUID="c79eff94-6543-4313-b59b-502ca4089e84" Nov 04 04:46:49.289907 ip-10-0-133-180 kubenswrapper[2579]: E1104 04:46:49.289886 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-25flt" podUID="4dff9faf-d91f-4ed5-be82-555b9a6a924e" Nov 04 04:46:49.290001 ip-10-0-133-180 kubenswrapper[2579]: E1104 04:46:49.289978 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8hsp8" podUID="5f6922fa-9ba7-4d02-b550-606731852fde" Nov 04 04:46:49.428686 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:49.428646 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-wdk7z" event={"ID":"d737ccc2-5686-4130-9d00-d2f21d5b417f","Type":"ContainerStarted","Data":"2d16b67e2177f68a8f425dbd5442ab0c819b47da9fa5db779fe3a46b028546c9"} Nov 04 04:46:50.432920 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:50.432721 2579 generic.go:356] "Generic (PLEG): container finished" podID="198e9ab0-f615-4686-9acf-a6242f7e3584" containerID="323fc8e1cff69036edddfb491d7c584c0c28de977220a248a1c7aad5548e2ac7" exitCode=0 Nov 04 04:46:50.433398 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:50.433215 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cplsq" event={"ID":"198e9ab0-f615-4686-9acf-a6242f7e3584","Type":"ContainerDied","Data":"323fc8e1cff69036edddfb491d7c584c0c28de977220a248a1c7aad5548e2ac7"} Nov 04 04:46:50.434764 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:50.434738 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-c65v8" event={"ID":"22d31d0c-6edf-4351-8f41-a6dd4e02368b","Type":"ContainerStarted","Data":"723cfafc5236a5426d4d421728a5c9c7db6621d23b6666cfba1bf549c5feb4c0"} Nov 04 04:46:50.437009 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:50.436974 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f2zrp" event={"ID":"00e0471e-5fe8-4494-8f17-632aee4f6c30","Type":"ContainerStarted","Data":"74b9c96f024dd545635e160d5765c947d4c48f61e9c2dd9680e24de3a02d47ca"} Nov 04 04:46:50.649850 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:50.649793 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-wdk7z" podStartSLOduration=4.883370192 podStartE2EDuration="13.649779596s" totalImagesPullingTime="8.766409404s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 04:46:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:46:49.540498468 +0000 UTC m=+13.755746407" watchObservedRunningTime="2026-11-04 04:46:50.649779596 +0000 UTC m=+14.865027534" Nov 04 04:46:50.715772 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:50.715670 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-c65v8" podStartSLOduration=2.01452823 podStartE2EDuration="13.715652114s" totalImagesPullingTime="11.701123884s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 04:46:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:46:50.714829613 +0000 UTC m=+14.930077551" watchObservedRunningTime="2026-11-04 04:46:50.715652114 +0000 UTC m=+14.930900052" Nov 04 04:46:50.843485 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:50.843428 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f2zrp" podStartSLOduration=3.0655103 podStartE2EDuration="13.843414875s" totalImagesPullingTime="10.777904575s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 04:46:37 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:46:50.843163418 +0000 UTC m=+15.058411356" watchObservedRunningTime="2026-11-04 04:46:50.843414875 +0000 UTC m=+15.058662852" Nov 04 04:46:51.289913 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:51.289882 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gfjqz" Nov 04 04:46:51.290140 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:51.289923 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-25flt" Nov 04 04:46:51.290140 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:51.289880 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8hsp8" Nov 04 04:46:51.290140 ip-10-0-133-180 kubenswrapper[2579]: E1104 04:46:51.290003 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gfjqz" podUID="c79eff94-6543-4313-b59b-502ca4089e84" Nov 04 04:46:51.290140 ip-10-0-133-180 kubenswrapper[2579]: E1104 04:46:51.290078 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-25flt" podUID="4dff9faf-d91f-4ed5-be82-555b9a6a924e" Nov 04 04:46:51.290140 ip-10-0-133-180 kubenswrapper[2579]: E1104 04:46:51.290123 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8hsp8" podUID="5f6922fa-9ba7-4d02-b550-606731852fde" Nov 04 04:46:51.576868 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:51.576788 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-7tw97" Nov 04 04:46:51.577409 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:51.576921 2579 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 04 04:46:51.577472 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:51.577453 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-7tw97" Nov 04 04:46:52.445344 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:52.445288 2579 generic.go:356] "Generic (PLEG): container finished" podID="198e9ab0-f615-4686-9acf-a6242f7e3584" containerID="32038a768b343a887e01933064b0baa519cc33e054200b177d1088225b8efd75" exitCode=0 Nov 04 04:46:52.445522 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:52.445380 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cplsq" event={"ID":"198e9ab0-f615-4686-9acf-a6242f7e3584","Type":"ContainerDied","Data":"32038a768b343a887e01933064b0baa519cc33e054200b177d1088225b8efd75"} Nov 04 04:46:53.188761 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:53.188724 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4dff9faf-d91f-4ed5-be82-555b9a6a924e-metrics-certs\") pod \"network-metrics-daemon-25flt\" (UID: \"4dff9faf-d91f-4ed5-be82-555b9a6a924e\") " pod="openshift-multus/network-metrics-daemon-25flt" Nov 04 04:46:53.189490 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:53.188803 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c79eff94-6543-4313-b59b-502ca4089e84-original-pull-secret\") pod \"global-pull-secret-syncer-gfjqz\" (UID: \"c79eff94-6543-4313-b59b-502ca4089e84\") " pod="kube-system/global-pull-secret-syncer-gfjqz" Nov 04 04:46:53.189490 ip-10-0-133-180 kubenswrapper[2579]: E1104 04:46:53.188882 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 04:46:53.189490 ip-10-0-133-180 kubenswrapper[2579]: E1104 04:46:53.188906 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 04 04:46:53.189490 ip-10-0-133-180 kubenswrapper[2579]: E1104 04:46:53.188948 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4dff9faf-d91f-4ed5-be82-555b9a6a924e-metrics-certs podName:4dff9faf-d91f-4ed5-be82-555b9a6a924e nodeName:}" failed. No retries permitted until 2026-11-04 04:47:09.188931958 +0000 UTC m=+33.404179874 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4dff9faf-d91f-4ed5-be82-555b9a6a924e-metrics-certs") pod "network-metrics-daemon-25flt" (UID: "4dff9faf-d91f-4ed5-be82-555b9a6a924e") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 04:46:53.189490 ip-10-0-133-180 kubenswrapper[2579]: E1104 04:46:53.188967 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c79eff94-6543-4313-b59b-502ca4089e84-original-pull-secret podName:c79eff94-6543-4313-b59b-502ca4089e84 nodeName:}" failed. No retries permitted until 2026-11-04 04:47:09.188959415 +0000 UTC m=+33.404207332 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c79eff94-6543-4313-b59b-502ca4089e84-original-pull-secret") pod "global-pull-secret-syncer-gfjqz" (UID: "c79eff94-6543-4313-b59b-502ca4089e84") : object "kube-system"/"original-pull-secret" not registered Nov 04 04:46:53.289422 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:53.289388 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gfjqz" Nov 04 04:46:53.289422 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:53.289402 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47nf5\" (UniqueName: \"kubernetes.io/projected/5f6922fa-9ba7-4d02-b550-606731852fde-kube-api-access-47nf5\") pod \"network-check-target-8hsp8\" (UID: \"5f6922fa-9ba7-4d02-b550-606731852fde\") " pod="openshift-network-diagnostics/network-check-target-8hsp8" Nov 04 04:46:53.289628 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:53.289426 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8hsp8" Nov 04 04:46:53.289628 ip-10-0-133-180 kubenswrapper[2579]: E1104 04:46:53.289525 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gfjqz" podUID="c79eff94-6543-4313-b59b-502ca4089e84" Nov 04 04:46:53.289628 ip-10-0-133-180 kubenswrapper[2579]: E1104 04:46:53.289564 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 04 04:46:53.289628 ip-10-0-133-180 kubenswrapper[2579]: E1104 04:46:53.289605 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8hsp8" podUID="5f6922fa-9ba7-4d02-b550-606731852fde" Nov 04 04:46:53.289628 ip-10-0-133-180 kubenswrapper[2579]: E1104 04:46:53.289614 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 04 04:46:53.289628 ip-10-0-133-180 kubenswrapper[2579]: E1104 04:46:53.289631 2579 projected.go:196] Error preparing data for projected volume kube-api-access-47nf5 for pod openshift-network-diagnostics/network-check-target-8hsp8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 04:46:53.289841 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:53.289650 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-25flt" Nov 04 04:46:53.289841 ip-10-0-133-180 kubenswrapper[2579]: E1104 04:46:53.289672 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5f6922fa-9ba7-4d02-b550-606731852fde-kube-api-access-47nf5 podName:5f6922fa-9ba7-4d02-b550-606731852fde nodeName:}" failed. No retries permitted until 2026-11-04 04:47:09.289655694 +0000 UTC m=+33.504903627 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-47nf5" (UniqueName: "kubernetes.io/projected/5f6922fa-9ba7-4d02-b550-606731852fde-kube-api-access-47nf5") pod "network-check-target-8hsp8" (UID: "5f6922fa-9ba7-4d02-b550-606731852fde") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 04:46:53.289841 ip-10-0-133-180 kubenswrapper[2579]: E1104 04:46:53.289719 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-25flt" podUID="4dff9faf-d91f-4ed5-be82-555b9a6a924e" Nov 04 04:46:54.454057 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:54.453857 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cplsq" event={"ID":"198e9ab0-f615-4686-9acf-a6242f7e3584","Type":"ContainerStarted","Data":"84838fa07477f4eb2ad0a3809e89ad1fceee70bee3f05d41d9e53ca23d6438e5"} Nov 04 04:46:54.457218 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:54.457181 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dvc6z" event={"ID":"844cc12f-a7f6-4327-968e-5e9b2da00ad0","Type":"ContainerStarted","Data":"e26b238da9aeea072f9886f14a73029391f057637221c68c54b09dbefe59939a"} Nov 04 04:46:54.457425 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:54.457258 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dvc6z" event={"ID":"844cc12f-a7f6-4327-968e-5e9b2da00ad0","Type":"ContainerStarted","Data":"abcf3aadef437bb4b105dd93290b3b393d4e740de9a8573253849cfb0fa24cfb"} Nov 04 04:46:54.457425 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:54.457300 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dvc6z" event={"ID":"844cc12f-a7f6-4327-968e-5e9b2da00ad0","Type":"ContainerStarted","Data":"742d3eb317d3447dfb0645b518e7364976d403af5d5086b1d588339c2846498e"} Nov 04 04:46:54.457425 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:54.457361 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dvc6z" event={"ID":"844cc12f-a7f6-4327-968e-5e9b2da00ad0","Type":"ContainerStarted","Data":"31f70bed64f8d144c3cd87b6606a337bc1906466b85846dacf03c7ed3d31fb3c"} Nov 04 04:46:54.457425 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:54.457404 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dvc6z" event={"ID":"844cc12f-a7f6-4327-968e-5e9b2da00ad0","Type":"ContainerStarted","Data":"1438c20db605bca65221c505548a167b0fd9c3872bc4d28ffb1f2be7b96a65ab"} Nov 04 04:46:54.457577 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:54.457448 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dvc6z" event={"ID":"844cc12f-a7f6-4327-968e-5e9b2da00ad0","Type":"ContainerStarted","Data":"c41fba25d500702a4322c478374594eff9e699b4142021986eaf4bb15828b7a4"} Nov 04 04:46:55.289654 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:55.289452 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gfjqz" Nov 04 04:46:55.289806 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:55.289512 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8hsp8" Nov 04 04:46:55.289806 ip-10-0-133-180 kubenswrapper[2579]: E1104 04:46:55.289728 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gfjqz" podUID="c79eff94-6543-4313-b59b-502ca4089e84" Nov 04 04:46:55.289806 ip-10-0-133-180 kubenswrapper[2579]: E1104 04:46:55.289798 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8hsp8" podUID="5f6922fa-9ba7-4d02-b550-606731852fde" Nov 04 04:46:55.289920 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:55.289512 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-25flt" Nov 04 04:46:55.289920 ip-10-0-133-180 kubenswrapper[2579]: E1104 04:46:55.289869 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-25flt" podUID="4dff9faf-d91f-4ed5-be82-555b9a6a924e" Nov 04 04:46:55.461449 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:55.461413 2579 generic.go:356] "Generic (PLEG): container finished" podID="198e9ab0-f615-4686-9acf-a6242f7e3584" containerID="84838fa07477f4eb2ad0a3809e89ad1fceee70bee3f05d41d9e53ca23d6438e5" exitCode=0 Nov 04 04:46:55.461786 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:55.461459 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cplsq" event={"ID":"198e9ab0-f615-4686-9acf-a6242f7e3584","Type":"ContainerDied","Data":"84838fa07477f4eb2ad0a3809e89ad1fceee70bee3f05d41d9e53ca23d6438e5"} Nov 04 04:46:56.467259 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:56.467223 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dvc6z" event={"ID":"844cc12f-a7f6-4327-968e-5e9b2da00ad0","Type":"ContainerStarted","Data":"c22759f0297054d1e680bc8ab219e9e004e0120093f3f7bbeee03e2a5a903c43"} Nov 04 04:46:56.470975 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:56.470953 2579 generic.go:356] "Generic (PLEG): container finished" podID="198e9ab0-f615-4686-9acf-a6242f7e3584" containerID="54f3cbda9d6ee2f42e6a7cd241272b78ca6c284da961c54d1674c33833a011aa" exitCode=0 Nov 04 04:46:56.471071 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:56.470981 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cplsq" event={"ID":"198e9ab0-f615-4686-9acf-a6242f7e3584","Type":"ContainerDied","Data":"54f3cbda9d6ee2f42e6a7cd241272b78ca6c284da961c54d1674c33833a011aa"} Nov 04 04:46:57.289814 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:57.289791 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gfjqz" Nov 04 04:46:57.289814 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:57.289822 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-25flt" Nov 04 04:46:57.289814 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:57.289820 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8hsp8" Nov 04 04:46:57.290021 ip-10-0-133-180 kubenswrapper[2579]: E1104 04:46:57.289903 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gfjqz" podUID="c79eff94-6543-4313-b59b-502ca4089e84" Nov 04 04:46:57.290021 ip-10-0-133-180 kubenswrapper[2579]: E1104 04:46:57.289953 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-25flt" podUID="4dff9faf-d91f-4ed5-be82-555b9a6a924e" Nov 04 04:46:57.290099 ip-10-0-133-180 kubenswrapper[2579]: E1104 04:46:57.290037 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8hsp8" podUID="5f6922fa-9ba7-4d02-b550-606731852fde" Nov 04 04:46:57.479281 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:57.479250 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cplsq" event={"ID":"198e9ab0-f615-4686-9acf-a6242f7e3584","Type":"ContainerStarted","Data":"458ac1f5b44d65a130f06f2cb3c01c00d3a6bff786a6af04556aa933f31f368e"} Nov 04 04:46:57.596887 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:57.596787 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-cplsq" podStartSLOduration=8.001682357 podStartE2EDuration="20.596774598s" totalImagesPullingTime="12.595092241s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 04:46:37 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:46:57.596281971 +0000 UTC m=+21.811529911" watchObservedRunningTime="2026-11-04 04:46:57.596774598 +0000 UTC m=+21.812022537" Nov 04 04:46:59.289671 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:59.289640 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8hsp8" Nov 04 04:46:59.290375 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:59.289758 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-25flt" Nov 04 04:46:59.290375 ip-10-0-133-180 kubenswrapper[2579]: E1104 04:46:59.289769 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8hsp8" podUID="5f6922fa-9ba7-4d02-b550-606731852fde" Nov 04 04:46:59.290375 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:59.289787 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gfjqz" Nov 04 04:46:59.290375 ip-10-0-133-180 kubenswrapper[2579]: E1104 04:46:59.289863 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-25flt" podUID="4dff9faf-d91f-4ed5-be82-555b9a6a924e" Nov 04 04:46:59.290375 ip-10-0-133-180 kubenswrapper[2579]: E1104 04:46:59.289919 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gfjqz" podUID="c79eff94-6543-4313-b59b-502ca4089e84" Nov 04 04:46:59.486907 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:59.486871 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dvc6z" event={"ID":"844cc12f-a7f6-4327-968e-5e9b2da00ad0","Type":"ContainerStarted","Data":"061a8871db3113d62ca3b8e57c0d0c7dcbd106cda380d9dd72e367fe50444184"} Nov 04 04:46:59.487280 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:59.487250 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-dvc6z" Nov 04 04:46:59.487422 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:59.487348 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-dvc6z" Nov 04 04:46:59.503039 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:59.503018 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dvc6z" Nov 04 04:46:59.503203 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:59.503187 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dvc6z" Nov 04 04:46:59.757215 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:46:59.757164 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-dvc6z" podStartSLOduration=7.443299954 podStartE2EDuration="22.757150035s" totalImagesPullingTime="15.313850081s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 04:46:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:46:59.707073497 +0000 UTC m=+23.922321436" watchObservedRunningTime="2026-11-04 04:46:59.757150035 +0000 UTC m=+23.972397973" Nov 04 04:47:00.489085 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:47:00.489058 2579 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 04 04:47:01.289504 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:47:01.289290 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8hsp8" Nov 04 04:47:01.289678 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:47:01.289391 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gfjqz" Nov 04 04:47:01.289678 ip-10-0-133-180 kubenswrapper[2579]: E1104 04:47:01.289594 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8hsp8" podUID="5f6922fa-9ba7-4d02-b550-606731852fde" Nov 04 04:47:01.289678 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:47:01.289400 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-25flt" Nov 04 04:47:01.289678 ip-10-0-133-180 kubenswrapper[2579]: E1104 04:47:01.289640 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gfjqz" podUID="c79eff94-6543-4313-b59b-502ca4089e84" Nov 04 04:47:01.289872 ip-10-0-133-180 kubenswrapper[2579]: E1104 04:47:01.289714 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-25flt" podUID="4dff9faf-d91f-4ed5-be82-555b9a6a924e" Nov 04 04:47:01.295828 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:47:01.295800 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-gfjqz"] Nov 04 04:47:01.298763 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:47:01.298731 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-8hsp8"] Nov 04 04:47:01.345491 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:47:01.345455 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-25flt"] Nov 04 04:47:01.490942 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:47:01.490917 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8hsp8" Nov 04 04:47:01.491428 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:47:01.490918 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-25flt" Nov 04 04:47:01.491428 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:47:01.491009 2579 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 04 04:47:01.491428 ip-10-0-133-180 kubenswrapper[2579]: E1104 04:47:01.491015 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8hsp8" podUID="5f6922fa-9ba7-4d02-b550-606731852fde" Nov 04 04:47:01.491428 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:47:01.490918 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gfjqz" Nov 04 04:47:01.491428 ip-10-0-133-180 kubenswrapper[2579]: E1104 04:47:01.491095 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-25flt" podUID="4dff9faf-d91f-4ed5-be82-555b9a6a924e" Nov 04 04:47:01.491870 ip-10-0-133-180 kubenswrapper[2579]: E1104 04:47:01.491555 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gfjqz" podUID="c79eff94-6543-4313-b59b-502ca4089e84" Nov 04 04:47:03.289882 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:47:03.289849 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gfjqz" Nov 04 04:47:03.290381 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:47:03.289900 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-25flt" Nov 04 04:47:03.290381 ip-10-0-133-180 kubenswrapper[2579]: E1104 04:47:03.289981 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gfjqz" podUID="c79eff94-6543-4313-b59b-502ca4089e84" Nov 04 04:47:03.290381 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:47:03.290051 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8hsp8" Nov 04 04:47:03.290381 ip-10-0-133-180 kubenswrapper[2579]: E1104 04:47:03.290149 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8hsp8" podUID="5f6922fa-9ba7-4d02-b550-606731852fde" Nov 04 04:47:03.290381 ip-10-0-133-180 kubenswrapper[2579]: E1104 04:47:03.290244 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-25flt" podUID="4dff9faf-d91f-4ed5-be82-555b9a6a924e" Nov 04 04:47:04.440225 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:47:04.440168 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-dvc6z" Nov 04 04:47:04.440671 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:47:04.440439 2579 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 04 04:47:04.452065 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:47:04.452026 2579 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-dvc6z" podUID="844cc12f-a7f6-4327-968e-5e9b2da00ad0" containerName="ovnkube-controller" probeResult="failure" output="" Nov 04 04:47:04.461193 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:47:04.461163 2579 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-dvc6z" podUID="844cc12f-a7f6-4327-968e-5e9b2da00ad0" containerName="ovnkube-controller" probeResult="failure" output="" Nov 04 04:47:05.289605 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:47:05.289572 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8hsp8" Nov 04 04:47:05.289777 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:47:05.289674 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gfjqz" Nov 04 04:47:05.289777 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:47:05.289690 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-25flt" Nov 04 04:47:05.289875 ip-10-0-133-180 kubenswrapper[2579]: E1104 04:47:05.289773 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8hsp8" podUID="5f6922fa-9ba7-4d02-b550-606731852fde" Nov 04 04:47:05.289875 ip-10-0-133-180 kubenswrapper[2579]: E1104 04:47:05.289859 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gfjqz" podUID="c79eff94-6543-4313-b59b-502ca4089e84" Nov 04 04:47:05.290154 ip-10-0-133-180 kubenswrapper[2579]: E1104 04:47:05.290131 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-25flt" podUID="4dff9faf-d91f-4ed5-be82-555b9a6a924e" Nov 04 04:47:06.345707 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:47:06.345681 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-gzctm_8446aa71-5638-4b23-8f5b-12283ede21d4/dns-node-resolver/0.log" Nov 04 04:47:06.520944 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:47:06.520923 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-180.us-west-1.compute.internal" event="NodeReady" Nov 04 04:47:06.521037 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:47:06.521018 2579 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 04 04:47:06.674936 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:47:06.674853 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-4xxrr"] Nov 04 04:47:06.677981 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:47:06.677959 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-dmdhl"] Nov 04 04:47:06.678102 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:47:06.678063 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-4xxrr" Nov 04 04:47:06.681177 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:47:06.681156 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-dmdhl" Nov 04 04:47:06.682417 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:47:06.682400 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-mt4dj"] Nov 04 04:47:06.685530 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:47:06.685517 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-mt4dj" Nov 04 04:47:06.697749 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:47:06.697681 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-4xxrr"] Nov 04 04:47:06.699753 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:47:06.699732 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-dmdhl"] Nov 04 04:47:06.730369 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:47:06.730343 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 04 04:47:06.730466 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:47:06.730449 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 04 04:47:06.730561 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:47:06.730547 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-6sxpx\"" Nov 04 04:47:06.732626 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:47:06.732609 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 04 04:47:06.732814 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:47:06.732791 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 04 04:47:06.732904 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:47:06.732826 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 04 04:47:06.732904 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:47:06.732835 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 04 04:47:06.732983 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:47:06.732916 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-jvrn2\"" Nov 04 04:47:06.737070 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:47:06.737053 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 04 04:47:06.737135 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:47:06.737102 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-q5r74\"" Nov 04 04:47:06.737135 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:47:06.737105 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 04 04:47:06.737221 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:47:06.737154 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 04 04:47:06.757171 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:47:06.757149 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-mt4dj"] Nov 04 04:47:06.779087 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:47:06.779068 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/21f0e203-fdf7-42cf-9c0c-4103e6601020-cert\") pod \"ingress-canary-mt4dj\" (UID: \"21f0e203-fdf7-42cf-9c0c-4103e6601020\") " pod="openshift-ingress-canary/ingress-canary-mt4dj" Nov 04 04:47:06.779166 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:47:06.779094 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cgfm\" (UniqueName: \"kubernetes.io/projected/cce8fa86-4f61-4dfc-b36c-2cefb7ecaa03-kube-api-access-7cgfm\") pod \"insights-runtime-extractor-dmdhl\" (UID: \"cce8fa86-4f61-4dfc-b36c-2cefb7ecaa03\") " pod="openshift-insights/insights-runtime-extractor-dmdhl" Nov 04 04:47:06.779166 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:47:06.779112 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/cce8fa86-4f61-4dfc-b36c-2cefb7ecaa03-data-volume\") pod \"insights-runtime-extractor-dmdhl\" (UID: \"cce8fa86-4f61-4dfc-b36c-2cefb7ecaa03\") " pod="openshift-insights/insights-runtime-extractor-dmdhl" Nov 04 04:47:06.779166 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:47:06.779130 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/cce8fa86-4f61-4dfc-b36c-2cefb7ecaa03-crio-socket\") pod \"insights-runtime-extractor-dmdhl\" (UID: \"cce8fa86-4f61-4dfc-b36c-2cefb7ecaa03\") " pod="openshift-insights/insights-runtime-extractor-dmdhl" Nov 04 04:47:06.779166 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:47:06.779147 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cce8fa86-4f61-4dfc-b36c-2cefb7ecaa03-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dmdhl\" (UID: \"cce8fa86-4f61-4dfc-b36c-2cefb7ecaa03\") " pod="openshift-insights/insights-runtime-extractor-dmdhl" Nov 04 04:47:06.779369 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:47:06.779352 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd0aaef0-d114-43b4-9ce4-135e818e4df3-tmp-dir\") pod \"dns-default-4xxrr\" (UID: \"dd0aaef0-d114-43b4-9ce4-135e818e4df3\") " pod="openshift-dns/dns-default-4xxrr" Nov 04 04:47:06.779428 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:47:06.779412 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2qjt\" (UniqueName: \"kubernetes.io/projected/21f0e203-fdf7-42cf-9c0c-4103e6601020-kube-api-access-x2qjt\") pod \"ingress-canary-mt4dj\" (UID: \"21f0e203-fdf7-42cf-9c0c-4103e6601020\") " pod="openshift-ingress-canary/ingress-canary-mt4dj" Nov 04 04:47:06.779478 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:47:06.779439 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/dd0aaef0-d114-43b4-9ce4-135e818e4df3-config-volume\") pod \"dns-default-4xxrr\" (UID: \"dd0aaef0-d114-43b4-9ce4-135e818e4df3\") " pod="openshift-dns/dns-default-4xxrr" Nov 04 04:47:06.779478 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:47:06.779468 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dd0aaef0-d114-43b4-9ce4-135e818e4df3-metrics-tls\") pod \"dns-default-4xxrr\" (UID: \"dd0aaef0-d114-43b4-9ce4-135e818e4df3\") " pod="openshift-dns/dns-default-4xxrr" Nov 04 04:47:06.779553 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:47:06.779524 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7l4mv\" (UniqueName: \"kubernetes.io/projected/dd0aaef0-d114-43b4-9ce4-135e818e4df3-kube-api-access-7l4mv\") pod \"dns-default-4xxrr\" (UID: \"dd0aaef0-d114-43b4-9ce4-135e818e4df3\") " pod="openshift-dns/dns-default-4xxrr" Nov 04 04:47:06.779553 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:47:06.779548 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cce8fa86-4f61-4dfc-b36c-2cefb7ecaa03-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dmdhl\" (UID: \"cce8fa86-4f61-4dfc-b36c-2cefb7ecaa03\") " pod="openshift-insights/insights-runtime-extractor-dmdhl" Nov 04 04:47:06.880793 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:47:06.880764 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/21f0e203-fdf7-42cf-9c0c-4103e6601020-cert\") pod \"ingress-canary-mt4dj\" (UID: \"21f0e203-fdf7-42cf-9c0c-4103e6601020\") " pod="openshift-ingress-canary/ingress-canary-mt4dj" Nov 04 04:47:06.880793 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:47:06.880792 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cgfm\" (UniqueName: \"kubernetes.io/projected/cce8fa86-4f61-4dfc-b36c-2cefb7ecaa03-kube-api-access-7cgfm\") pod \"insights-runtime-extractor-dmdhl\" (UID: \"cce8fa86-4f61-4dfc-b36c-2cefb7ecaa03\") " pod="openshift-insights/insights-runtime-extractor-dmdhl" Nov 04 04:47:06.880988 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:47:06.880813 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/cce8fa86-4f61-4dfc-b36c-2cefb7ecaa03-data-volume\") pod \"insights-runtime-extractor-dmdhl\" (UID: \"cce8fa86-4f61-4dfc-b36c-2cefb7ecaa03\") " pod="openshift-insights/insights-runtime-extractor-dmdhl" Nov 04 04:47:06.880988 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:47:06.880921 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/cce8fa86-4f61-4dfc-b36c-2cefb7ecaa03-crio-socket\") pod \"insights-runtime-extractor-dmdhl\" (UID: \"cce8fa86-4f61-4dfc-b36c-2cefb7ecaa03\") " pod="openshift-insights/insights-runtime-extractor-dmdhl" Nov 04 04:47:06.880988 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:47:06.880944 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cce8fa86-4f61-4dfc-b36c-2cefb7ecaa03-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dmdhl\" (UID: \"cce8fa86-4f61-4dfc-b36c-2cefb7ecaa03\") " pod="openshift-insights/insights-runtime-extractor-dmdhl" Nov 04 04:47:06.881138 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:47:06.880993 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd0aaef0-d114-43b4-9ce4-135e818e4df3-tmp-dir\") pod \"dns-default-4xxrr\" (UID: \"dd0aaef0-d114-43b4-9ce4-135e818e4df3\") " pod="openshift-dns/dns-default-4xxrr" Nov 04 04:47:06.881138 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:47:06.881039 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2qjt\" (UniqueName: \"kubernetes.io/projected/21f0e203-fdf7-42cf-9c0c-4103e6601020-kube-api-access-x2qjt\") pod \"ingress-canary-mt4dj\" (UID: \"21f0e203-fdf7-42cf-9c0c-4103e6601020\") " pod="openshift-ingress-canary/ingress-canary-mt4dj" Nov 04 04:47:06.881138 ip-10-0-133-180 kubenswrapper[2579]: E1104 04:47:06.881073 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 04 04:47:06.881138 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:47:06.881095 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/dd0aaef0-d114-43b4-9ce4-135e818e4df3-config-volume\") pod \"dns-default-4xxrr\" (UID: \"dd0aaef0-d114-43b4-9ce4-135e818e4df3\") " pod="openshift-dns/dns-default-4xxrr" Nov 04 04:47:06.881351 ip-10-0-133-180 kubenswrapper[2579]: E1104 04:47:06.881157 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cce8fa86-4f61-4dfc-b36c-2cefb7ecaa03-kube-rbac-proxy-cm podName:cce8fa86-4f61-4dfc-b36c-2cefb7ecaa03 nodeName:}" failed. No retries permitted until 2026-11-04 04:47:07.381138867 +0000 UTC m=+31.596386820 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/cce8fa86-4f61-4dfc-b36c-2cefb7ecaa03-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dmdhl" (UID: "cce8fa86-4f61-4dfc-b36c-2cefb7ecaa03") : configmap "kube-rbac-proxy" not found Nov 04 04:47:06.881351 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:47:06.881180 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dd0aaef0-d114-43b4-9ce4-135e818e4df3-metrics-tls\") pod \"dns-default-4xxrr\" (UID: \"dd0aaef0-d114-43b4-9ce4-135e818e4df3\") " pod="openshift-dns/dns-default-4xxrr" Nov 04 04:47:06.881351 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:47:06.881221 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7l4mv\" (UniqueName: \"kubernetes.io/projected/dd0aaef0-d114-43b4-9ce4-135e818e4df3-kube-api-access-7l4mv\") pod \"dns-default-4xxrr\" (UID: \"dd0aaef0-d114-43b4-9ce4-135e818e4df3\") " pod="openshift-dns/dns-default-4xxrr" Nov 04 04:47:06.881351 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:47:06.881250 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cce8fa86-4f61-4dfc-b36c-2cefb7ecaa03-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dmdhl\" (UID: \"cce8fa86-4f61-4dfc-b36c-2cefb7ecaa03\") " pod="openshift-insights/insights-runtime-extractor-dmdhl" Nov 04 04:47:06.881351 ip-10-0-133-180 kubenswrapper[2579]: E1104 04:47:06.881279 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 04 04:47:06.881351 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:47:06.881156 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/cce8fa86-4f61-4dfc-b36c-2cefb7ecaa03-crio-socket\") pod \"insights-runtime-extractor-dmdhl\" (UID: \"cce8fa86-4f61-4dfc-b36c-2cefb7ecaa03\") " pod="openshift-insights/insights-runtime-extractor-dmdhl" Nov 04 04:47:06.881662 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:47:06.881367 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/cce8fa86-4f61-4dfc-b36c-2cefb7ecaa03-data-volume\") pod \"insights-runtime-extractor-dmdhl\" (UID: \"cce8fa86-4f61-4dfc-b36c-2cefb7ecaa03\") " pod="openshift-insights/insights-runtime-extractor-dmdhl" Nov 04 04:47:06.881662 ip-10-0-133-180 kubenswrapper[2579]: E1104 04:47:06.881378 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd0aaef0-d114-43b4-9ce4-135e818e4df3-metrics-tls podName:dd0aaef0-d114-43b4-9ce4-135e818e4df3 nodeName:}" failed. No retries permitted until 2026-11-04 04:47:07.381358427 +0000 UTC m=+31.596606345 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/dd0aaef0-d114-43b4-9ce4-135e818e4df3-metrics-tls") pod "dns-default-4xxrr" (UID: "dd0aaef0-d114-43b4-9ce4-135e818e4df3") : secret "dns-default-metrics-tls" not found Nov 04 04:47:06.881662 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:47:06.881367 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd0aaef0-d114-43b4-9ce4-135e818e4df3-tmp-dir\") pod \"dns-default-4xxrr\" (UID: \"dd0aaef0-d114-43b4-9ce4-135e818e4df3\") " pod="openshift-dns/dns-default-4xxrr" Nov 04 04:47:06.881662 ip-10-0-133-180 kubenswrapper[2579]: E1104 04:47:06.881415 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 04 04:47:06.881662 ip-10-0-133-180 kubenswrapper[2579]: E1104 04:47:06.881506 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cce8fa86-4f61-4dfc-b36c-2cefb7ecaa03-insights-runtime-extractor-tls podName:cce8fa86-4f61-4dfc-b36c-2cefb7ecaa03 nodeName:}" failed. No retries permitted until 2026-11-04 04:47:07.381488738 +0000 UTC m=+31.596736654 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/cce8fa86-4f61-4dfc-b36c-2cefb7ecaa03-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dmdhl" (UID: "cce8fa86-4f61-4dfc-b36c-2cefb7ecaa03") : secret "insights-runtime-extractor-tls" not found Nov 04 04:47:06.881954 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:47:06.881933 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/dd0aaef0-d114-43b4-9ce4-135e818e4df3-config-volume\") pod \"dns-default-4xxrr\" (UID: \"dd0aaef0-d114-43b4-9ce4-135e818e4df3\") " pod="openshift-dns/dns-default-4xxrr" Nov 04 04:47:06.884842 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:47:06.884816 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/21f0e203-fdf7-42cf-9c0c-4103e6601020-cert\") pod \"ingress-canary-mt4dj\" (UID: \"21f0e203-fdf7-42cf-9c0c-4103e6601020\") " pod="openshift-ingress-canary/ingress-canary-mt4dj" Nov 04 04:47:06.937500 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:47:06.937446 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cgfm\" (UniqueName: \"kubernetes.io/projected/cce8fa86-4f61-4dfc-b36c-2cefb7ecaa03-kube-api-access-7cgfm\") pod \"insights-runtime-extractor-dmdhl\" (UID: \"cce8fa86-4f61-4dfc-b36c-2cefb7ecaa03\") " pod="openshift-insights/insights-runtime-extractor-dmdhl" Nov 04 04:47:06.937751 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:47:06.937731 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7l4mv\" (UniqueName: \"kubernetes.io/projected/dd0aaef0-d114-43b4-9ce4-135e818e4df3-kube-api-access-7l4mv\") pod \"dns-default-4xxrr\" (UID: \"dd0aaef0-d114-43b4-9ce4-135e818e4df3\") " pod="openshift-dns/dns-default-4xxrr" Nov 04 04:47:06.937811 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:47:06.937802 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2qjt\" (UniqueName: \"kubernetes.io/projected/21f0e203-fdf7-42cf-9c0c-4103e6601020-kube-api-access-x2qjt\") pod \"ingress-canary-mt4dj\" (UID: \"21f0e203-fdf7-42cf-9c0c-4103e6601020\") " pod="openshift-ingress-canary/ingress-canary-mt4dj" Nov 04 04:47:07.011762 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:47:07.011742 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-mt4dj" Nov 04 04:47:07.183314 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:47:07.183289 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-4z89d_9a06ec07-c72b-43af-aa27-7414aa9c9aba/node-ca/0.log" Nov 04 04:47:07.289608 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:47:07.289581 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8hsp8" Nov 04 04:47:07.289768 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:47:07.289665 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gfjqz" Nov 04 04:47:07.289768 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:47:07.289679 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-25flt" Nov 04 04:47:07.367768 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:47:07.367740 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 04 04:47:07.367768 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:47:07.367768 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 04 04:47:07.368194 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:47:07.368142 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-w4xfx\"" Nov 04 04:47:07.368254 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:47:07.368244 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-n7w7t\"" Nov 04 04:47:07.368368 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:47:07.368221 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 04 04:47:07.368368 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:47:07.368270 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 04 04:47:07.368368 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:47:07.368315 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-mrmbg\"" Nov 04 04:47:07.384740 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:47:07.384716 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dd0aaef0-d114-43b4-9ce4-135e818e4df3-metrics-tls\") pod \"dns-default-4xxrr\" (UID: \"dd0aaef0-d114-43b4-9ce4-135e818e4df3\") " pod="openshift-dns/dns-default-4xxrr" Nov 04 04:47:07.384835 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:47:07.384751 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cce8fa86-4f61-4dfc-b36c-2cefb7ecaa03-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dmdhl\" (UID: \"cce8fa86-4f61-4dfc-b36c-2cefb7ecaa03\") " pod="openshift-insights/insights-runtime-extractor-dmdhl" Nov 04 04:47:07.384835 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:47:07.384780 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cce8fa86-4f61-4dfc-b36c-2cefb7ecaa03-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dmdhl\" (UID: \"cce8fa86-4f61-4dfc-b36c-2cefb7ecaa03\") " pod="openshift-insights/insights-runtime-extractor-dmdhl" Nov 04 04:47:07.384910 ip-10-0-133-180 kubenswrapper[2579]: E1104 04:47:07.384884 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 04 04:47:07.384910 ip-10-0-133-180 kubenswrapper[2579]: E1104 04:47:07.384885 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 04 04:47:07.384977 ip-10-0-133-180 kubenswrapper[2579]: E1104 04:47:07.384918 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 04 04:47:07.384977 ip-10-0-133-180 kubenswrapper[2579]: E1104 04:47:07.384941 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd0aaef0-d114-43b4-9ce4-135e818e4df3-metrics-tls podName:dd0aaef0-d114-43b4-9ce4-135e818e4df3 nodeName:}" failed. No retries permitted until 2026-11-04 04:47:08.384926507 +0000 UTC m=+32.600174423 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/dd0aaef0-d114-43b4-9ce4-135e818e4df3-metrics-tls") pod "dns-default-4xxrr" (UID: "dd0aaef0-d114-43b4-9ce4-135e818e4df3") : secret "dns-default-metrics-tls" not found Nov 04 04:47:07.384977 ip-10-0-133-180 kubenswrapper[2579]: E1104 04:47:07.384961 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cce8fa86-4f61-4dfc-b36c-2cefb7ecaa03-insights-runtime-extractor-tls podName:cce8fa86-4f61-4dfc-b36c-2cefb7ecaa03 nodeName:}" failed. No retries permitted until 2026-11-04 04:47:08.38495273 +0000 UTC m=+32.600200650 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/cce8fa86-4f61-4dfc-b36c-2cefb7ecaa03-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dmdhl" (UID: "cce8fa86-4f61-4dfc-b36c-2cefb7ecaa03") : secret "insights-runtime-extractor-tls" not found Nov 04 04:47:07.384977 ip-10-0-133-180 kubenswrapper[2579]: E1104 04:47:07.384971 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cce8fa86-4f61-4dfc-b36c-2cefb7ecaa03-kube-rbac-proxy-cm podName:cce8fa86-4f61-4dfc-b36c-2cefb7ecaa03 nodeName:}" failed. No retries permitted until 2026-11-04 04:47:08.384966294 +0000 UTC m=+32.600214210 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/cce8fa86-4f61-4dfc-b36c-2cefb7ecaa03-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dmdhl" (UID: "cce8fa86-4f61-4dfc-b36c-2cefb7ecaa03") : configmap "kube-rbac-proxy" not found Nov 04 04:47:07.388738 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:47:07.388715 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-mt4dj"] Nov 04 04:47:07.392019 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:47:07.391999 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21f0e203_fdf7_42cf_9c0c_4103e6601020.slice/crio-32ef11380c8ec7870a0852b0413a2c0c70f4ea455b4889ff21c914c5c95ace32 WatchSource:0}: Error finding container 32ef11380c8ec7870a0852b0413a2c0c70f4ea455b4889ff21c914c5c95ace32: Status 404 returned error can't find the container with id 32ef11380c8ec7870a0852b0413a2c0c70f4ea455b4889ff21c914c5c95ace32 Nov 04 04:47:07.503216 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:47:07.503185 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-mt4dj" event={"ID":"21f0e203-fdf7-42cf-9c0c-4103e6601020","Type":"ContainerStarted","Data":"32ef11380c8ec7870a0852b0413a2c0c70f4ea455b4889ff21c914c5c95ace32"} Nov 04 04:47:08.392781 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:47:08.392743 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dd0aaef0-d114-43b4-9ce4-135e818e4df3-metrics-tls\") pod \"dns-default-4xxrr\" (UID: \"dd0aaef0-d114-43b4-9ce4-135e818e4df3\") " pod="openshift-dns/dns-default-4xxrr" Nov 04 04:47:08.392781 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:47:08.392782 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cce8fa86-4f61-4dfc-b36c-2cefb7ecaa03-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dmdhl\" (UID: \"cce8fa86-4f61-4dfc-b36c-2cefb7ecaa03\") " pod="openshift-insights/insights-runtime-extractor-dmdhl" Nov 04 04:47:08.393227 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:47:08.392810 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cce8fa86-4f61-4dfc-b36c-2cefb7ecaa03-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dmdhl\" (UID: \"cce8fa86-4f61-4dfc-b36c-2cefb7ecaa03\") " pod="openshift-insights/insights-runtime-extractor-dmdhl" Nov 04 04:47:08.393227 ip-10-0-133-180 kubenswrapper[2579]: E1104 04:47:08.392917 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 04 04:47:08.393227 ip-10-0-133-180 kubenswrapper[2579]: E1104 04:47:08.392985 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cce8fa86-4f61-4dfc-b36c-2cefb7ecaa03-insights-runtime-extractor-tls podName:cce8fa86-4f61-4dfc-b36c-2cefb7ecaa03 nodeName:}" failed. No retries permitted until 2026-11-04 04:47:10.392970789 +0000 UTC m=+34.608218707 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/cce8fa86-4f61-4dfc-b36c-2cefb7ecaa03-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dmdhl" (UID: "cce8fa86-4f61-4dfc-b36c-2cefb7ecaa03") : secret "insights-runtime-extractor-tls" not found Nov 04 04:47:08.393227 ip-10-0-133-180 kubenswrapper[2579]: E1104 04:47:08.392917 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 04 04:47:08.393227 ip-10-0-133-180 kubenswrapper[2579]: E1104 04:47:08.393050 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd0aaef0-d114-43b4-9ce4-135e818e4df3-metrics-tls podName:dd0aaef0-d114-43b4-9ce4-135e818e4df3 nodeName:}" failed. No retries permitted until 2026-11-04 04:47:10.393039486 +0000 UTC m=+34.608287417 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/dd0aaef0-d114-43b4-9ce4-135e818e4df3-metrics-tls") pod "dns-default-4xxrr" (UID: "dd0aaef0-d114-43b4-9ce4-135e818e4df3") : secret "dns-default-metrics-tls" not found Nov 04 04:47:08.393429 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:47:08.393283 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cce8fa86-4f61-4dfc-b36c-2cefb7ecaa03-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dmdhl\" (UID: \"cce8fa86-4f61-4dfc-b36c-2cefb7ecaa03\") " pod="openshift-insights/insights-runtime-extractor-dmdhl" Nov 04 04:47:09.200354 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:47:09.200249 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c79eff94-6543-4313-b59b-502ca4089e84-original-pull-secret\") pod \"global-pull-secret-syncer-gfjqz\" (UID: \"c79eff94-6543-4313-b59b-502ca4089e84\") " pod="kube-system/global-pull-secret-syncer-gfjqz" Nov 04 04:47:09.200541 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:47:09.200459 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4dff9faf-d91f-4ed5-be82-555b9a6a924e-metrics-certs\") pod \"network-metrics-daemon-25flt\" (UID: \"4dff9faf-d91f-4ed5-be82-555b9a6a924e\") " pod="openshift-multus/network-metrics-daemon-25flt" Nov 04 04:47:09.203174 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:47:09.203152 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c79eff94-6543-4313-b59b-502ca4089e84-original-pull-secret\") pod \"global-pull-secret-syncer-gfjqz\" (UID: \"c79eff94-6543-4313-b59b-502ca4089e84\") " pod="kube-system/global-pull-secret-syncer-gfjqz" Nov 04 04:47:09.203174 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:47:09.203167 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4dff9faf-d91f-4ed5-be82-555b9a6a924e-metrics-certs\") pod \"network-metrics-daemon-25flt\" (UID: \"4dff9faf-d91f-4ed5-be82-555b9a6a924e\") " pod="openshift-multus/network-metrics-daemon-25flt" Nov 04 04:47:09.301688 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:47:09.301651 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47nf5\" (UniqueName: \"kubernetes.io/projected/5f6922fa-9ba7-4d02-b550-606731852fde-kube-api-access-47nf5\") pod \"network-check-target-8hsp8\" (UID: \"5f6922fa-9ba7-4d02-b550-606731852fde\") " pod="openshift-network-diagnostics/network-check-target-8hsp8" Nov 04 04:47:09.304821 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:47:09.304793 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47nf5\" (UniqueName: \"kubernetes.io/projected/5f6922fa-9ba7-4d02-b550-606731852fde-kube-api-access-47nf5\") pod \"network-check-target-8hsp8\" (UID: \"5f6922fa-9ba7-4d02-b550-606731852fde\") " pod="openshift-network-diagnostics/network-check-target-8hsp8" Nov 04 04:47:09.405653 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:47:09.405619 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8hsp8" Nov 04 04:47:09.410842 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:47:09.410818 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-25flt" Nov 04 04:47:09.413973 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:47:09.413950 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gfjqz" Nov 04 04:47:09.812125 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:47:09.812039 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-8hsp8"] Nov 04 04:47:09.856700 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:47:09.856671 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f6922fa_9ba7_4d02_b550_606731852fde.slice/crio-39a9f52ed78a8bc4487051e839eee3354641e60a5732aede57702167ec597ace WatchSource:0}: Error finding container 39a9f52ed78a8bc4487051e839eee3354641e60a5732aede57702167ec597ace: Status 404 returned error can't find the container with id 39a9f52ed78a8bc4487051e839eee3354641e60a5732aede57702167ec597ace Nov 04 04:47:09.862508 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:47:09.862482 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-gfjqz"] Nov 04 04:47:09.866647 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:47:09.866614 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-25flt"] Nov 04 04:47:09.867899 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:47:09.867870 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc79eff94_6543_4313_b59b_502ca4089e84.slice/crio-861a0607ea89dfcd8263350e4f58f1cfa403d43419d2e64f3d1a7227d12e34fc WatchSource:0}: Error finding container 861a0607ea89dfcd8263350e4f58f1cfa403d43419d2e64f3d1a7227d12e34fc: Status 404 returned error can't find the container with id 861a0607ea89dfcd8263350e4f58f1cfa403d43419d2e64f3d1a7227d12e34fc Nov 04 04:47:09.870126 ip-10-0-133-180 kubenswrapper[2579]: W1104 04:47:09.870098 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4dff9faf_d91f_4ed5_be82_555b9a6a924e.slice/crio-b3dc53f0a90e45b5e090ed31fa76115def5e68ba143668be0c4aeae7f2177c08 WatchSource:0}: Error finding container b3dc53f0a90e45b5e090ed31fa76115def5e68ba143668be0c4aeae7f2177c08: Status 404 returned error can't find the container with id b3dc53f0a90e45b5e090ed31fa76115def5e68ba143668be0c4aeae7f2177c08 Nov 04 04:47:10.414401 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:47:10.414372 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dd0aaef0-d114-43b4-9ce4-135e818e4df3-metrics-tls\") pod \"dns-default-4xxrr\" (UID: \"dd0aaef0-d114-43b4-9ce4-135e818e4df3\") " pod="openshift-dns/dns-default-4xxrr" Nov 04 04:47:10.414401 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:47:10.414408 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cce8fa86-4f61-4dfc-b36c-2cefb7ecaa03-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dmdhl\" (UID: \"cce8fa86-4f61-4dfc-b36c-2cefb7ecaa03\") " pod="openshift-insights/insights-runtime-extractor-dmdhl" Nov 04 04:47:10.414932 ip-10-0-133-180 kubenswrapper[2579]: E1104 04:47:10.414525 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 04 04:47:10.414932 ip-10-0-133-180 kubenswrapper[2579]: E1104 04:47:10.414596 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cce8fa86-4f61-4dfc-b36c-2cefb7ecaa03-insights-runtime-extractor-tls podName:cce8fa86-4f61-4dfc-b36c-2cefb7ecaa03 nodeName:}" failed. No retries permitted until 2026-11-04 04:47:14.414576904 +0000 UTC m=+38.629824820 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/cce8fa86-4f61-4dfc-b36c-2cefb7ecaa03-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dmdhl" (UID: "cce8fa86-4f61-4dfc-b36c-2cefb7ecaa03") : secret "insights-runtime-extractor-tls" not found Nov 04 04:47:10.417707 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:47:10.417679 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dd0aaef0-d114-43b4-9ce4-135e818e4df3-metrics-tls\") pod \"dns-default-4xxrr\" (UID: \"dd0aaef0-d114-43b4-9ce4-135e818e4df3\") " pod="openshift-dns/dns-default-4xxrr" Nov 04 04:47:10.517837 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:47:10.517809 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-gfjqz" event={"ID":"c79eff94-6543-4313-b59b-502ca4089e84","Type":"ContainerStarted","Data":"861a0607ea89dfcd8263350e4f58f1cfa403d43419d2e64f3d1a7227d12e34fc"} Nov 04 04:47:10.518805 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:47:10.518787 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-8hsp8" event={"ID":"5f6922fa-9ba7-4d02-b550-606731852fde","Type":"ContainerStarted","Data":"39a9f52ed78a8bc4487051e839eee3354641e60a5732aede57702167ec597ace"} Nov 04 04:47:10.519716 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:47:10.519698 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-25flt" event={"ID":"4dff9faf-d91f-4ed5-be82-555b9a6a924e","Type":"ContainerStarted","Data":"b3dc53f0a90e45b5e090ed31fa76115def5e68ba143668be0c4aeae7f2177c08"} Nov 04 04:47:10.595830 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:47:10.595811 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-4xxrr" Nov 04 04:47:10.977265 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:47:10.977229 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-4xxrr"] Nov 04 04:47:11.525063 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:47:11.525025 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4xxrr" event={"ID":"dd0aaef0-d114-43b4-9ce4-135e818e4df3","Type":"ContainerStarted","Data":"10a3c53136d5e01c314b91221a5a31445eaf198fc677922193fc805f9c033083"} Nov 04 04:47:11.526868 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:47:11.526836 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-mt4dj" event={"ID":"21f0e203-fdf7-42cf-9c0c-4103e6601020","Type":"ContainerStarted","Data":"ea476d154490bff85ffb2a45e998e1710beb7751d6204f78671a002e2dc52d36"} Nov 04 04:47:11.639777 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:47:11.639725 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-mt4dj" podStartSLOduration=2.211207416 podStartE2EDuration="5.639710904s" totalImagesPullingTime="3.428503488s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 04:47:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:47:11.639094884 +0000 UTC m=+35.854342822" watchObservedRunningTime="2026-11-04 04:47:11.639710904 +0000 UTC m=+35.854958841" Nov 04 04:47:12.533401 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:47:12.533363 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-25flt" event={"ID":"4dff9faf-d91f-4ed5-be82-555b9a6a924e","Type":"ContainerStarted","Data":"677721b4540d01f55e49ea7fc3705fd1efdd9052cd88cac26ea87040df79dbe7"} Nov 04 04:47:12.533829 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:47:12.533423 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-25flt" event={"ID":"4dff9faf-d91f-4ed5-be82-555b9a6a924e","Type":"ContainerStarted","Data":"7f09c01185a81341c4b3f5fbcb2433dfdc8c41c4dc18fa654e700769a586731c"} Nov 04 04:47:12.648479 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:47:12.648412 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-25flt" podStartSLOduration=33.599598327 podStartE2EDuration="35.648392624s" totalImagesPullingTime="2.048794297s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 04:46:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:47:12.647900127 +0000 UTC m=+36.863148056" watchObservedRunningTime="2026-11-04 04:47:12.648392624 +0000 UTC m=+36.863640563" Nov 04 04:47:14.445637 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:47:14.445594 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cce8fa86-4f61-4dfc-b36c-2cefb7ecaa03-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dmdhl\" (UID: \"cce8fa86-4f61-4dfc-b36c-2cefb7ecaa03\") " pod="openshift-insights/insights-runtime-extractor-dmdhl" Nov 04 04:47:14.446097 ip-10-0-133-180 kubenswrapper[2579]: E1104 04:47:14.445738 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 04 04:47:14.446097 ip-10-0-133-180 kubenswrapper[2579]: E1104 04:47:14.445811 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cce8fa86-4f61-4dfc-b36c-2cefb7ecaa03-insights-runtime-extractor-tls podName:cce8fa86-4f61-4dfc-b36c-2cefb7ecaa03 nodeName:}" failed. No retries permitted until 2026-11-04 04:47:22.445791741 +0000 UTC m=+46.661039664 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/cce8fa86-4f61-4dfc-b36c-2cefb7ecaa03-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dmdhl" (UID: "cce8fa86-4f61-4dfc-b36c-2cefb7ecaa03") : secret "insights-runtime-extractor-tls" not found Nov 04 04:47:14.543252 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:47:14.543160 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4xxrr" event={"ID":"dd0aaef0-d114-43b4-9ce4-135e818e4df3","Type":"ContainerStarted","Data":"b0c35b1cfe4310bca155d6cf4704a230227c6d9cdcb855f18b0acf5ca2987c6b"} Nov 04 04:47:14.543252 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:47:14.543219 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4xxrr" event={"ID":"dd0aaef0-d114-43b4-9ce4-135e818e4df3","Type":"ContainerStarted","Data":"0146629d936b3098fbab9dc438527fb0038afe4000d99d7211a3c91326f3f458"} Nov 04 04:47:14.543482 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:47:14.543281 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-4xxrr" Nov 04 04:47:14.544955 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:47:14.544926 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-8hsp8" event={"ID":"5f6922fa-9ba7-4d02-b550-606731852fde","Type":"ContainerStarted","Data":"faaf64ad57e2ed5cca51e6aa969fd27d4b939b3145bd28c9deab591eec3207e1"} Nov 04 04:47:14.545079 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:47:14.545067 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-8hsp8" Nov 04 04:47:14.663600 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:47:14.663538 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-4xxrr" podStartSLOduration=5.633727237 podStartE2EDuration="8.663519185s" totalImagesPullingTime="3.029791948s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 04:47:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:47:14.662833635 +0000 UTC m=+38.878081570" watchObservedRunningTime="2026-11-04 04:47:14.663519185 +0000 UTC m=+38.878767127" Nov 04 04:47:14.774944 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:47:14.774886 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-8hsp8" podStartSLOduration=33.405184173 podStartE2EDuration="37.774867075s" totalImagesPullingTime="4.369682902s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 04:46:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:47:14.774659897 +0000 UTC m=+38.989907846" watchObservedRunningTime="2026-11-04 04:47:14.774867075 +0000 UTC m=+38.990115015" Nov 04 04:47:16.059933 ip-10-0-133-180 kubenswrapper[2579]: I1104 04:47:16.059836 2579 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 04 04:47:16.059996 ip-10-0-133-180 systemd[1]: Stopping Kubernetes Kubelet... Nov 04 04:47:16.074843 ip-10-0-133-180 systemd[1]: kubelet.service: Deactivated successfully. Nov 04 04:47:16.075019 ip-10-0-133-180 systemd[1]: Stopped Kubernetes Kubelet. Nov 04 04:47:16.075071 ip-10-0-133-180 systemd[1]: kubelet.service: Consumed 2.324s CPU time. Nov 04 04:47:16.090018 ip-10-0-133-180 systemd[1]: Starting Kubernetes Kubelet... Nov 04 04:47:16.179019 ip-10-0-133-180 kubenswrapper[4509]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 04 04:47:16.179019 ip-10-0-133-180 kubenswrapper[4509]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 04 04:47:16.179019 ip-10-0-133-180 kubenswrapper[4509]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 04 04:47:16.179019 ip-10-0-133-180 kubenswrapper[4509]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 04 04:47:16.181218 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.181127 4509 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 04:47:16.181218 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.181220 4509 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 04:47:16.181301 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.181224 4509 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 04:47:16.181301 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.181227 4509 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 04:47:16.181301 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.181230 4509 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 04:47:16.181301 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.181233 4509 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 04:47:16.181301 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.181236 4509 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 04:47:16.181301 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.181238 4509 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 04:47:16.181301 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.181241 4509 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 04:47:16.181301 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.181244 4509 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 04:47:16.181301 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.181247 4509 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 04:47:16.181301 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.181249 4509 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 04:47:16.181301 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.181252 4509 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 04:47:16.181301 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.181255 4509 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 04:47:16.181301 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.181257 4509 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 04:47:16.181301 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.181260 4509 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 04:47:16.181301 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.181262 4509 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 04:47:16.181301 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.181264 4509 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 04:47:16.181301 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.181267 4509 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 04:47:16.181301 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.181270 4509 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 04:47:16.181301 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.181280 4509 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 04:47:16.181836 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.181283 4509 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 04:47:16.181836 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.181285 4509 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 04:47:16.181836 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.181288 4509 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 04:47:16.181836 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.181291 4509 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 04:47:16.181836 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.181294 4509 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 04:47:16.181836 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.181297 4509 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 04:47:16.181836 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.181300 4509 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 04:47:16.181836 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.181302 4509 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 04:47:16.181836 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.181305 4509 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 04:47:16.181836 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.181308 4509 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 04:47:16.181836 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.181310 4509 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 04:47:16.181836 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.181313 4509 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 04:47:16.181836 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.181316 4509 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 04:47:16.181836 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.181338 4509 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 04:47:16.181836 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.181342 4509 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 04:47:16.181836 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.181344 4509 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 04:47:16.181836 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.181347 4509 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 04:47:16.181836 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.181349 4509 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 04:47:16.181836 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.181355 4509 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 04:47:16.181836 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.181357 4509 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 04:47:16.182354 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.181360 4509 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 04:47:16.182354 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.181363 4509 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 04:47:16.182354 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.181366 4509 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 04:47:16.182354 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.181368 4509 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 04:47:16.182354 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.181371 4509 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 04:47:16.182354 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.181374 4509 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 04:47:16.182354 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.181376 4509 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 04:47:16.182354 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.181379 4509 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 04:47:16.182354 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.181381 4509 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 04:47:16.182354 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.181384 4509 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 04:47:16.182354 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.181386 4509 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 04:47:16.182354 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.181389 4509 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 04:47:16.182354 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.181392 4509 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 04:47:16.182354 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.181394 4509 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 04:47:16.182354 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.181397 4509 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 04:47:16.182354 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.181399 4509 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 04:47:16.182354 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.181402 4509 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 04:47:16.182354 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.181405 4509 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 04:47:16.182354 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.181407 4509 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 04:47:16.182354 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.181410 4509 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 04:47:16.182835 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.181412 4509 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 04:47:16.182835 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.181415 4509 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 04:47:16.182835 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.181418 4509 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 04:47:16.182835 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.181420 4509 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 04:47:16.182835 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.181423 4509 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 04:47:16.182835 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.181426 4509 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 04:47:16.182835 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.181429 4509 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 04:47:16.182835 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.181431 4509 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 04:47:16.182835 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.181434 4509 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 04:47:16.182835 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.181449 4509 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 04:47:16.182835 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.181453 4509 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 04:47:16.182835 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.181456 4509 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 04:47:16.182835 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.181459 4509 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 04:47:16.182835 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.181461 4509 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 04:47:16.182835 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.181464 4509 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 04:47:16.182835 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.181467 4509 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 04:47:16.182835 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.181469 4509 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 04:47:16.182835 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.181472 4509 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 04:47:16.182835 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.181474 4509 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 04:47:16.182835 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.181476 4509 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 04:47:16.183314 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.181479 4509 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 04:47:16.183314 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.181483 4509 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 04:47:16.183314 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.181485 4509 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 04:47:16.183314 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.181488 4509 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 04:47:16.183314 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.181493 4509 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 04:47:16.183314 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.181498 4509 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 04:47:16.183314 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.181502 4509 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 04:47:16.183314 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.181505 4509 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 04:47:16.183314 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.181508 4509 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 04:47:16.183314 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.181512 4509 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 04:47:16.183314 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.181515 4509 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 04:47:16.183314 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.181517 4509 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 04:47:16.183314 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.181520 4509 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 04:47:16.183314 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.181523 4509 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 04:47:16.183314 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.181525 4509 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 04:47:16.183314 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.181528 4509 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 04:47:16.183314 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.181530 4509 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 04:47:16.183314 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.181533 4509 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 04:47:16.183314 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.181535 4509 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 04:47:16.183889 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.181538 4509 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 04:47:16.183889 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.181541 4509 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 04:47:16.183889 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.181543 4509 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 04:47:16.183889 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.181546 4509 feature_gate.go:386] unrecognized feature gate: Example Nov 04 04:47:16.183889 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.181548 4509 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 04:47:16.183889 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.181554 4509 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 04:47:16.183889 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.182334 4509 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 04:47:16.183889 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.182342 4509 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 04:47:16.183889 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.182345 4509 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 04:47:16.183889 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.182347 4509 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 04:47:16.183889 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.182350 4509 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 04:47:16.183889 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.182353 4509 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 04:47:16.183889 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.182355 4509 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 04:47:16.183889 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.182359 4509 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 04:47:16.183889 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.182361 4509 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 04:47:16.183889 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.182364 4509 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 04:47:16.183889 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.182367 4509 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 04:47:16.183889 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.182370 4509 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 04:47:16.183889 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.182374 4509 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 04:47:16.184478 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.182377 4509 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 04:47:16.184478 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.182380 4509 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 04:47:16.184478 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.182383 4509 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 04:47:16.184478 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.182385 4509 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 04:47:16.184478 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.182388 4509 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 04:47:16.184478 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.182391 4509 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 04:47:16.184478 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.182393 4509 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 04:47:16.184478 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.182395 4509 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 04:47:16.184478 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.182398 4509 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 04:47:16.184478 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.182401 4509 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 04:47:16.184478 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.182403 4509 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 04:47:16.184478 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.182406 4509 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 04:47:16.184478 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.182408 4509 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 04:47:16.184478 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.182411 4509 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 04:47:16.184478 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.182413 4509 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 04:47:16.184478 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.182416 4509 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 04:47:16.184478 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.182418 4509 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 04:47:16.184478 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.182421 4509 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 04:47:16.184478 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.182423 4509 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 04:47:16.184953 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.182426 4509 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 04:47:16.184953 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.182429 4509 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 04:47:16.184953 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.182431 4509 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 04:47:16.184953 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.182433 4509 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 04:47:16.184953 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.182436 4509 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 04:47:16.184953 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.182439 4509 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 04:47:16.184953 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.182441 4509 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 04:47:16.184953 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.182443 4509 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 04:47:16.184953 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.182446 4509 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 04:47:16.184953 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.182448 4509 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 04:47:16.184953 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.182451 4509 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 04:47:16.184953 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.182453 4509 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 04:47:16.184953 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.182456 4509 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 04:47:16.184953 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.182459 4509 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 04:47:16.184953 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.182461 4509 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 04:47:16.184953 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.182463 4509 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 04:47:16.184953 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.182466 4509 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 04:47:16.184953 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.182468 4509 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 04:47:16.184953 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.182471 4509 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 04:47:16.184953 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.182473 4509 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 04:47:16.185484 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.182476 4509 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 04:47:16.185484 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.182479 4509 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 04:47:16.185484 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.182481 4509 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 04:47:16.185484 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.182484 4509 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 04:47:16.185484 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.182487 4509 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 04:47:16.185484 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.182489 4509 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 04:47:16.185484 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.182492 4509 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 04:47:16.185484 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.182494 4509 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 04:47:16.185484 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.182497 4509 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 04:47:16.185484 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.182499 4509 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 04:47:16.185484 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.182502 4509 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 04:47:16.185484 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.182504 4509 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 04:47:16.185484 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.182507 4509 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 04:47:16.185484 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.182511 4509 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 04:47:16.185484 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.182513 4509 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 04:47:16.185484 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.182516 4509 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 04:47:16.185484 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.182518 4509 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 04:47:16.185484 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.182521 4509 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 04:47:16.185484 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.182523 4509 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 04:47:16.185953 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.182526 4509 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 04:47:16.185953 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.182529 4509 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 04:47:16.185953 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.182531 4509 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 04:47:16.185953 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.182534 4509 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 04:47:16.185953 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.182537 4509 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 04:47:16.185953 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.182539 4509 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 04:47:16.185953 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.182542 4509 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 04:47:16.185953 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.182544 4509 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 04:47:16.185953 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.182547 4509 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 04:47:16.185953 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.182549 4509 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 04:47:16.185953 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.182552 4509 feature_gate.go:386] unrecognized feature gate: Example Nov 04 04:47:16.185953 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.182554 4509 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 04:47:16.185953 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.182557 4509 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 04:47:16.185953 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.182559 4509 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 04:47:16.185953 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.182563 4509 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 04:47:16.185953 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.182567 4509 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 04:47:16.185953 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.182571 4509 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 04:47:16.185953 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.182574 4509 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 04:47:16.185953 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.182577 4509 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 04:47:16.186424 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.182579 4509 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 04:47:16.186424 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.182582 4509 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 04:47:16.186424 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.182584 4509 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 04:47:16.186424 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.182587 4509 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 04:47:16.186424 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.182589 4509 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 04:47:16.186424 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.182592 4509 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 04:47:16.186424 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.182594 4509 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 04:47:16.186424 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.182614 4509 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 04:47:16.186424 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.182618 4509 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 04:47:16.186424 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.182622 4509 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 04:47:16.186424 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.182624 4509 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 04:47:16.186424 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.182627 4509 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 04:47:16.186424 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.182630 4509 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 04:47:16.186424 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.182632 4509 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 04:47:16.186424 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.182635 4509 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 04:47:16.186424 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.182638 4509 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 04:47:16.186424 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.183117 4509 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 04:47:16.186424 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.183122 4509 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 04:47:16.186424 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.183125 4509 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 04:47:16.186424 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.183127 4509 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 04:47:16.186900 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.183130 4509 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 04:47:16.186900 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.183133 4509 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 04:47:16.186900 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.183136 4509 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 04:47:16.186900 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.183138 4509 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 04:47:16.186900 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.183141 4509 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 04:47:16.186900 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.183143 4509 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 04:47:16.186900 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.183146 4509 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 04:47:16.186900 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.183149 4509 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 04:47:16.186900 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.183151 4509 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 04:47:16.186900 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.183153 4509 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 04:47:16.186900 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.183156 4509 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 04:47:16.186900 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.183158 4509 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 04:47:16.186900 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.183161 4509 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 04:47:16.186900 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.183163 4509 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 04:47:16.186900 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.183166 4509 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 04:47:16.186900 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.183168 4509 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 04:47:16.186900 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.183171 4509 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 04:47:16.186900 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.183173 4509 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 04:47:16.186900 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.183176 4509 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 04:47:16.186900 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.183178 4509 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 04:47:16.187408 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.183182 4509 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 04:47:16.187408 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.183184 4509 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 04:47:16.187408 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.183192 4509 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 04:47:16.187408 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.183194 4509 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 04:47:16.187408 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.183197 4509 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 04:47:16.187408 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.183199 4509 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 04:47:16.187408 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.183202 4509 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 04:47:16.187408 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.183204 4509 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 04:47:16.187408 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.183206 4509 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 04:47:16.187408 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.183209 4509 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 04:47:16.187408 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.183211 4509 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 04:47:16.187408 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.183214 4509 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 04:47:16.187408 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.183216 4509 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 04:47:16.187408 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.183219 4509 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 04:47:16.187408 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.183221 4509 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 04:47:16.187408 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.183224 4509 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 04:47:16.187408 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.183226 4509 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 04:47:16.187408 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.183229 4509 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 04:47:16.187408 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.183232 4509 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 04:47:16.187408 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.183234 4509 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 04:47:16.187886 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.183237 4509 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 04:47:16.187886 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.183239 4509 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 04:47:16.187886 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.183241 4509 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 04:47:16.187886 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.183244 4509 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 04:47:16.187886 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.183246 4509 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 04:47:16.187886 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.183248 4509 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 04:47:16.187886 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.183251 4509 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 04:47:16.187886 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.183253 4509 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 04:47:16.187886 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.183256 4509 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 04:47:16.187886 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.183258 4509 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 04:47:16.187886 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.183260 4509 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 04:47:16.187886 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.183263 4509 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 04:47:16.187886 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.183265 4509 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 04:47:16.187886 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.183268 4509 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 04:47:16.187886 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.183272 4509 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 04:47:16.187886 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.183275 4509 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 04:47:16.187886 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.183277 4509 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 04:47:16.187886 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.183280 4509 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 04:47:16.187886 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.183282 4509 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 04:47:16.187886 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.183285 4509 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 04:47:16.188424 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.183288 4509 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 04:47:16.188424 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.183290 4509 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 04:47:16.188424 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.183292 4509 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 04:47:16.188424 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.183295 4509 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 04:47:16.188424 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.183297 4509 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 04:47:16.188424 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.183300 4509 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 04:47:16.188424 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.183302 4509 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 04:47:16.188424 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.183305 4509 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 04:47:16.188424 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.183309 4509 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 04:47:16.188424 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.183314 4509 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 04:47:16.188424 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.183332 4509 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 04:47:16.188424 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.183335 4509 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 04:47:16.188424 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.183338 4509 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 04:47:16.188424 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.183341 4509 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 04:47:16.188424 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.183344 4509 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 04:47:16.188424 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.183347 4509 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 04:47:16.188424 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.183349 4509 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 04:47:16.188424 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.183352 4509 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 04:47:16.188424 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.183354 4509 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 04:47:16.188424 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.183357 4509 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 04:47:16.188910 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.183360 4509 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 04:47:16.188910 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.183362 4509 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 04:47:16.188910 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.183365 4509 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 04:47:16.188910 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.183367 4509 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 04:47:16.188910 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.183370 4509 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 04:47:16.188910 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.183372 4509 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 04:47:16.188910 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.183377 4509 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 04:47:16.188910 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.183380 4509 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 04:47:16.188910 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.183383 4509 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 04:47:16.188910 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.183385 4509 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 04:47:16.188910 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.183388 4509 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 04:47:16.188910 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.183391 4509 feature_gate.go:386] unrecognized feature gate: Example Nov 04 04:47:16.188910 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.183393 4509 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 04:47:16.188910 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.183396 4509 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 04:47:16.188910 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.183398 4509 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 04:47:16.188910 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.183401 4509 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 04:47:16.188910 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.183405 4509 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 04:47:16.188910 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.183409 4509 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 04:47:16.188910 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.183413 4509 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 04:47:16.189406 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.183416 4509 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 04:47:16.189406 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.183419 4509 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 04:47:16.189406 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.183422 4509 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 04:47:16.189406 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.183516 4509 flags.go:64] FLAG: --address="0.0.0.0" Nov 04 04:47:16.189406 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.183524 4509 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 04 04:47:16.189406 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.183531 4509 flags.go:64] FLAG: --anonymous-auth="true" Nov 04 04:47:16.189406 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.183535 4509 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 04 04:47:16.189406 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.183540 4509 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 04 04:47:16.189406 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.183543 4509 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 04 04:47:16.189406 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.183548 4509 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 04 04:47:16.189406 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.183552 4509 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 04 04:47:16.189406 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.183555 4509 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 04 04:47:16.189406 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.183558 4509 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 04 04:47:16.189406 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.183562 4509 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 04 04:47:16.189406 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.183565 4509 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 04 04:47:16.189406 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.183568 4509 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 04 04:47:16.189406 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.183571 4509 flags.go:64] FLAG: --cgroup-root="" Nov 04 04:47:16.189406 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.183574 4509 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 04 04:47:16.189406 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.183577 4509 flags.go:64] FLAG: --client-ca-file="" Nov 04 04:47:16.189406 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.183581 4509 flags.go:64] FLAG: --cloud-provider="external" Nov 04 04:47:16.189406 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.183585 4509 flags.go:64] FLAG: --cluster-dns="[]" Nov 04 04:47:16.189406 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.183590 4509 flags.go:64] FLAG: --cluster-domain="" Nov 04 04:47:16.189406 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.183592 4509 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 04 04:47:16.190012 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.183595 4509 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 04 04:47:16.190012 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.183598 4509 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 04 04:47:16.190012 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.183602 4509 flags.go:64] FLAG: --container-log-max-files="5" Nov 04 04:47:16.190012 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.183606 4509 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 04 04:47:16.190012 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.183609 4509 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 04 04:47:16.190012 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.183612 4509 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 04 04:47:16.190012 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.183616 4509 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 04 04:47:16.190012 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.183619 4509 flags.go:64] FLAG: --contention-profiling="false" Nov 04 04:47:16.190012 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.183622 4509 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 04 04:47:16.190012 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.183625 4509 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 04 04:47:16.190012 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.183629 4509 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 04 04:47:16.190012 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.183632 4509 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 04 04:47:16.190012 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.183636 4509 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 04 04:47:16.190012 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.183639 4509 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 04 04:47:16.190012 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.183642 4509 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 04 04:47:16.190012 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.183645 4509 flags.go:64] FLAG: --enable-load-reader="false" Nov 04 04:47:16.190012 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.183648 4509 flags.go:64] FLAG: --enable-server="true" Nov 04 04:47:16.190012 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.183651 4509 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 04 04:47:16.190012 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.184017 4509 flags.go:64] FLAG: --event-burst="100" Nov 04 04:47:16.190012 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.184021 4509 flags.go:64] FLAG: --event-qps="50" Nov 04 04:47:16.190012 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.184024 4509 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 04 04:47:16.190012 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.184027 4509 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 04 04:47:16.190012 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.184030 4509 flags.go:64] FLAG: --eviction-hard="" Nov 04 04:47:16.190012 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.184035 4509 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 04 04:47:16.190993 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.184038 4509 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 04 04:47:16.190993 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.184041 4509 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 04 04:47:16.190993 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.184044 4509 flags.go:64] FLAG: --eviction-soft="" Nov 04 04:47:16.190993 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.184047 4509 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 04 04:47:16.190993 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.184051 4509 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 04 04:47:16.190993 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.184055 4509 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 04 04:47:16.190993 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.184058 4509 flags.go:64] FLAG: --experimental-mounter-path="" Nov 04 04:47:16.190993 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.184061 4509 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 04 04:47:16.190993 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.184064 4509 flags.go:64] FLAG: --fail-swap-on="true" Nov 04 04:47:16.190993 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.184067 4509 flags.go:64] FLAG: --feature-gates="" Nov 04 04:47:16.190993 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.184071 4509 flags.go:64] FLAG: --file-check-frequency="20s" Nov 04 04:47:16.190993 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.184074 4509 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 04 04:47:16.190993 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.184077 4509 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 04 04:47:16.190993 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.184080 4509 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 04 04:47:16.190993 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.184084 4509 flags.go:64] FLAG: --healthz-port="10248" Nov 04 04:47:16.190993 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.184088 4509 flags.go:64] FLAG: --help="false" Nov 04 04:47:16.190993 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.184090 4509 flags.go:64] FLAG: --hostname-override="ip-10-0-133-180.us-west-1.compute.internal" Nov 04 04:47:16.190993 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.184094 4509 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 04 04:47:16.190993 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.184097 4509 flags.go:64] FLAG: --http-check-frequency="20s" Nov 04 04:47:16.190993 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.184100 4509 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 04 04:47:16.190993 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.184103 4509 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 04 04:47:16.190993 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.184106 4509 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 04 04:47:16.190993 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.184109 4509 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 04 04:47:16.191787 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.184112 4509 flags.go:64] FLAG: --image-service-endpoint="" Nov 04 04:47:16.191787 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.184115 4509 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 04 04:47:16.191787 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.184118 4509 flags.go:64] FLAG: --kube-api-burst="100" Nov 04 04:47:16.191787 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.184121 4509 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 04 04:47:16.191787 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.184124 4509 flags.go:64] FLAG: --kube-api-qps="50" Nov 04 04:47:16.191787 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.184126 4509 flags.go:64] FLAG: --kube-reserved="" Nov 04 04:47:16.191787 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.184129 4509 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 04 04:47:16.191787 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.184132 4509 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 04 04:47:16.191787 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.184135 4509 flags.go:64] FLAG: --kubelet-cgroups="" Nov 04 04:47:16.191787 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.184138 4509 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 04 04:47:16.191787 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.184140 4509 flags.go:64] FLAG: --lock-file="" Nov 04 04:47:16.191787 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.184143 4509 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 04 04:47:16.191787 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.184146 4509 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 04 04:47:16.191787 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.184150 4509 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 04 04:47:16.191787 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.184156 4509 flags.go:64] FLAG: --log-json-split-stream="false" Nov 04 04:47:16.191787 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.184159 4509 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 04 04:47:16.191787 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.184162 4509 flags.go:64] FLAG: --log-text-split-stream="false" Nov 04 04:47:16.191787 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.184165 4509 flags.go:64] FLAG: --logging-format="text" Nov 04 04:47:16.191787 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.184168 4509 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 04 04:47:16.191787 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.184171 4509 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 04 04:47:16.191787 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.184174 4509 flags.go:64] FLAG: --manifest-url="" Nov 04 04:47:16.191787 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.184177 4509 flags.go:64] FLAG: --manifest-url-header="" Nov 04 04:47:16.191787 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.184181 4509 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 04 04:47:16.191787 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.184185 4509 flags.go:64] FLAG: --max-open-files="1000000" Nov 04 04:47:16.191787 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.184189 4509 flags.go:64] FLAG: --max-pods="110" Nov 04 04:47:16.192447 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.184192 4509 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 04 04:47:16.192447 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.184195 4509 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 04 04:47:16.192447 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.184198 4509 flags.go:64] FLAG: --memory-manager-policy="None" Nov 04 04:47:16.192447 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.184201 4509 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 04 04:47:16.192447 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.184204 4509 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 04 04:47:16.192447 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.184207 4509 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 04 04:47:16.192447 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.184210 4509 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 04 04:47:16.192447 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.184218 4509 flags.go:64] FLAG: --node-status-max-images="50" Nov 04 04:47:16.192447 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.184221 4509 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 04 04:47:16.192447 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.184224 4509 flags.go:64] FLAG: --oom-score-adj="-999" Nov 04 04:47:16.192447 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.184227 4509 flags.go:64] FLAG: --pod-cidr="" Nov 04 04:47:16.192447 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.184230 4509 flags.go:64] FLAG: --pod-manifest-path="" Nov 04 04:47:16.192447 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.184233 4509 flags.go:64] FLAG: --pod-max-pids="-1" Nov 04 04:47:16.192447 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.184236 4509 flags.go:64] FLAG: --pods-per-core="0" Nov 04 04:47:16.192447 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.184239 4509 flags.go:64] FLAG: --port="10250" Nov 04 04:47:16.192447 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.184242 4509 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 04 04:47:16.192447 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.184245 4509 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-0e23cc09baf76542f" Nov 04 04:47:16.192447 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.184248 4509 flags.go:64] FLAG: --qos-reserved="" Nov 04 04:47:16.192447 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.184251 4509 flags.go:64] FLAG: --read-only-port="10255" Nov 04 04:47:16.192447 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.184254 4509 flags.go:64] FLAG: --register-node="true" Nov 04 04:47:16.192447 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.184257 4509 flags.go:64] FLAG: --register-with-taints="" Nov 04 04:47:16.192447 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.184261 4509 flags.go:64] FLAG: --registry-burst="10" Nov 04 04:47:16.192447 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.184264 4509 flags.go:64] FLAG: --registry-qps="5" Nov 04 04:47:16.192447 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.184267 4509 flags.go:64] FLAG: --reserved-cpus="" Nov 04 04:47:16.192447 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.184270 4509 flags.go:64] FLAG: --reserved-memory="" Nov 04 04:47:16.192447 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.184274 4509 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 04 04:47:16.193124 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.184277 4509 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 04 04:47:16.193124 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.184280 4509 flags.go:64] FLAG: --rotate-certificates="false" Nov 04 04:47:16.193124 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.184283 4509 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 04 04:47:16.193124 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.184286 4509 flags.go:64] FLAG: --runonce="false" Nov 04 04:47:16.193124 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.184290 4509 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 04 04:47:16.193124 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.184304 4509 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 04 04:47:16.193124 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.184307 4509 flags.go:64] FLAG: --seccomp-default="false" Nov 04 04:47:16.193124 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.184310 4509 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 04 04:47:16.193124 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.184313 4509 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 04 04:47:16.193124 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.184316 4509 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 04 04:47:16.193124 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.184351 4509 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 04 04:47:16.193124 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.184355 4509 flags.go:64] FLAG: --storage-driver-password="root" Nov 04 04:47:16.193124 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.184358 4509 flags.go:64] FLAG: --storage-driver-secure="false" Nov 04 04:47:16.193124 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.184361 4509 flags.go:64] FLAG: --storage-driver-table="stats" Nov 04 04:47:16.193124 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.184364 4509 flags.go:64] FLAG: --storage-driver-user="root" Nov 04 04:47:16.193124 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.184367 4509 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 04 04:47:16.193124 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.184370 4509 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 04 04:47:16.193124 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.184373 4509 flags.go:64] FLAG: --system-cgroups="" Nov 04 04:47:16.193124 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.184376 4509 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 04 04:47:16.193124 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.184382 4509 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 04 04:47:16.193124 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.184385 4509 flags.go:64] FLAG: --tls-cert-file="" Nov 04 04:47:16.193124 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.184387 4509 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 04 04:47:16.193124 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.184391 4509 flags.go:64] FLAG: --tls-min-version="" Nov 04 04:47:16.193124 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.184394 4509 flags.go:64] FLAG: --tls-private-key-file="" Nov 04 04:47:16.193124 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.184396 4509 flags.go:64] FLAG: --topology-manager-policy="none" Nov 04 04:47:16.193778 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.184399 4509 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 04 04:47:16.193778 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.184403 4509 flags.go:64] FLAG: --topology-manager-scope="container" Nov 04 04:47:16.193778 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.184407 4509 flags.go:64] FLAG: --v="2" Nov 04 04:47:16.193778 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.184411 4509 flags.go:64] FLAG: --version="false" Nov 04 04:47:16.193778 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.184415 4509 flags.go:64] FLAG: --vmodule="" Nov 04 04:47:16.193778 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.184419 4509 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 04 04:47:16.193778 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.184423 4509 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 04 04:47:16.193778 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.184561 4509 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 04:47:16.193778 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.184565 4509 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 04:47:16.193778 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.184568 4509 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 04:47:16.193778 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.184571 4509 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 04:47:16.193778 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.184574 4509 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 04:47:16.193778 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.184577 4509 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 04:47:16.193778 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.184579 4509 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 04:47:16.193778 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.184582 4509 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 04:47:16.193778 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.184584 4509 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 04:47:16.193778 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.184587 4509 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 04:47:16.193778 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.184590 4509 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 04:47:16.193778 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.184592 4509 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 04:47:16.193778 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.184595 4509 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 04:47:16.193778 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.184598 4509 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 04:47:16.193778 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.184600 4509 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 04:47:16.194390 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.184603 4509 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 04:47:16.194390 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.184605 4509 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 04:47:16.194390 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.184608 4509 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 04:47:16.194390 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.184610 4509 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 04:47:16.194390 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.184613 4509 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 04:47:16.194390 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.184615 4509 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 04:47:16.194390 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.184618 4509 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 04:47:16.194390 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.184620 4509 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 04:47:16.194390 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.184623 4509 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 04:47:16.194390 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.184625 4509 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 04:47:16.194390 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.184628 4509 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 04:47:16.194390 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.184632 4509 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 04:47:16.194390 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.184634 4509 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 04:47:16.194390 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.184637 4509 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 04:47:16.194390 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.184640 4509 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 04:47:16.194390 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.184642 4509 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 04:47:16.194390 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.184645 4509 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 04:47:16.194390 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.184648 4509 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 04:47:16.194390 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.184651 4509 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 04:47:16.194390 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.184654 4509 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 04:47:16.194914 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.184656 4509 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 04:47:16.194914 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.184659 4509 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 04:47:16.194914 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.184661 4509 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 04:47:16.194914 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.184664 4509 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 04:47:16.194914 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.184666 4509 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 04:47:16.194914 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.184669 4509 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 04:47:16.194914 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.184673 4509 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 04:47:16.194914 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.184676 4509 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 04:47:16.194914 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.184680 4509 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 04:47:16.194914 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.184683 4509 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 04:47:16.194914 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.184685 4509 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 04:47:16.194914 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.184688 4509 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 04:47:16.194914 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.184690 4509 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 04:47:16.194914 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.184693 4509 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 04:47:16.194914 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.184695 4509 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 04:47:16.194914 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.184698 4509 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 04:47:16.194914 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.184701 4509 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 04:47:16.194914 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.184703 4509 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 04:47:16.194914 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.184706 4509 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 04:47:16.195598 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.184708 4509 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 04:47:16.195598 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.184711 4509 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 04:47:16.195598 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.184713 4509 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 04:47:16.195598 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.184716 4509 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 04:47:16.195598 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.184719 4509 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 04:47:16.195598 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.184722 4509 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 04:47:16.195598 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.184724 4509 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 04:47:16.195598 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.184727 4509 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 04:47:16.195598 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.184729 4509 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 04:47:16.195598 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.184732 4509 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 04:47:16.195598 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.184735 4509 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 04:47:16.195598 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.184740 4509 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 04:47:16.195598 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.184743 4509 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 04:47:16.195598 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.184745 4509 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 04:47:16.195598 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.184748 4509 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 04:47:16.195598 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.184750 4509 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 04:47:16.195598 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.184753 4509 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 04:47:16.195598 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.184755 4509 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 04:47:16.195598 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.184758 4509 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 04:47:16.196214 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.184760 4509 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 04:47:16.196214 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.184762 4509 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 04:47:16.196214 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.184765 4509 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 04:47:16.196214 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.184768 4509 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 04:47:16.196214 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.184770 4509 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 04:47:16.196214 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.184773 4509 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 04:47:16.196214 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.184775 4509 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 04:47:16.196214 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.184778 4509 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 04:47:16.196214 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.184781 4509 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 04:47:16.196214 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.184783 4509 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 04:47:16.196214 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.184786 4509 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 04:47:16.196214 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.184788 4509 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 04:47:16.196214 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.184791 4509 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 04:47:16.196214 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.184793 4509 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 04:47:16.196214 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.184796 4509 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 04:47:16.196214 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.184798 4509 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 04:47:16.196214 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.184801 4509 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 04:47:16.196214 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.184805 4509 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 04:47:16.196214 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.184808 4509 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 04:47:16.196897 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.184810 4509 feature_gate.go:386] unrecognized feature gate: Example Nov 04 04:47:16.196897 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.184813 4509 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 04:47:16.196897 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.184816 4509 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 04:47:16.196897 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.184818 4509 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 04:47:16.196897 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.184822 4509 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 04:47:16.196897 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.184825 4509 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 04:47:16.196897 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.184829 4509 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 04:47:16.196897 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.184831 4509 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 04:47:16.196897 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.184834 4509 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 04:47:16.196897 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.184836 4509 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 04:47:16.196897 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.184839 4509 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 04:47:16.196897 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.184841 4509 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 04:47:16.196897 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.184844 4509 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 04:47:16.196897 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.184847 4509 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 04:47:16.196897 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.184872 4509 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 04 04:47:16.196897 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.184916 4509 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 04:47:16.197378 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.184921 4509 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 04:47:16.197378 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.184924 4509 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 04:47:16.197378 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.184926 4509 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 04:47:16.197378 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.184929 4509 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 04:47:16.197378 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.184937 4509 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 04:47:16.197378 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.184940 4509 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 04:47:16.197378 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.184943 4509 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 04:47:16.197378 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.184945 4509 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 04:47:16.197378 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.184948 4509 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 04:47:16.197378 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.184950 4509 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 04:47:16.197378 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.184953 4509 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 04:47:16.197378 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.184956 4509 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 04:47:16.197378 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.184959 4509 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 04:47:16.197378 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.184961 4509 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 04:47:16.197378 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.184964 4509 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 04:47:16.197378 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.184968 4509 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 04:47:16.197378 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.184971 4509 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 04:47:16.197378 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.184974 4509 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 04:47:16.197378 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.184976 4509 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 04:47:16.197378 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.184979 4509 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 04:47:16.197927 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.184981 4509 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 04:47:16.197927 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.184984 4509 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 04:47:16.197927 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.184986 4509 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 04:47:16.197927 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.184989 4509 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 04:47:16.197927 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.184992 4509 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 04:47:16.197927 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.184994 4509 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 04:47:16.197927 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.184997 4509 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 04:47:16.197927 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.184999 4509 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 04:47:16.197927 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.185002 4509 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 04:47:16.197927 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.185005 4509 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 04:47:16.197927 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.185008 4509 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 04:47:16.197927 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.185010 4509 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 04:47:16.197927 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.185013 4509 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 04:47:16.197927 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.185015 4509 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 04:47:16.197927 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.185017 4509 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 04:47:16.197927 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.185020 4509 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 04:47:16.197927 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.185025 4509 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 04:47:16.197927 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.185030 4509 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 04:47:16.197927 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.185033 4509 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 04:47:16.198485 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.185036 4509 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 04:47:16.198485 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.185038 4509 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 04:47:16.198485 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.185041 4509 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 04:47:16.198485 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.185043 4509 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 04:47:16.198485 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.185046 4509 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 04:47:16.198485 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.185048 4509 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 04:47:16.198485 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.185051 4509 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 04:47:16.198485 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.185053 4509 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 04:47:16.198485 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.185057 4509 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 04:47:16.198485 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.185060 4509 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 04:47:16.198485 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.185062 4509 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 04:47:16.198485 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.185065 4509 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 04:47:16.198485 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.185067 4509 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 04:47:16.198485 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.185070 4509 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 04:47:16.198485 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.185072 4509 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 04:47:16.198485 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.185075 4509 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 04:47:16.198485 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.185077 4509 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 04:47:16.198485 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.185080 4509 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 04:47:16.198485 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.185083 4509 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 04:47:16.199017 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.185085 4509 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 04:47:16.199017 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.185088 4509 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 04:47:16.199017 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.185090 4509 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 04:47:16.199017 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.185093 4509 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 04:47:16.199017 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.185095 4509 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 04:47:16.199017 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.185098 4509 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 04:47:16.199017 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.185100 4509 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 04:47:16.199017 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.185103 4509 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 04:47:16.199017 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.185105 4509 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 04:47:16.199017 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.185108 4509 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 04:47:16.199017 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.185111 4509 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 04:47:16.199017 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.185114 4509 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 04:47:16.199017 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.185116 4509 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 04:47:16.199017 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.185119 4509 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 04:47:16.199017 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.185121 4509 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 04:47:16.199017 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.185124 4509 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 04:47:16.199017 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.185126 4509 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 04:47:16.199017 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.185129 4509 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 04:47:16.199017 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.185131 4509 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 04:47:16.199484 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.185134 4509 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 04:47:16.199484 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.185136 4509 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 04:47:16.199484 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.185140 4509 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 04:47:16.199484 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.185143 4509 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 04:47:16.199484 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.185147 4509 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 04:47:16.199484 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.185149 4509 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 04:47:16.199484 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.185152 4509 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 04:47:16.199484 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.185154 4509 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 04:47:16.199484 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.185157 4509 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 04:47:16.199484 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.185159 4509 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 04:47:16.199484 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.185162 4509 feature_gate.go:386] unrecognized feature gate: Example Nov 04 04:47:16.199484 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.185164 4509 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 04:47:16.199484 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.185167 4509 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 04:47:16.199484 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.185170 4509 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 04:47:16.199484 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.185174 4509 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 04:47:16.199484 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.185178 4509 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 04:47:16.199484 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.185180 4509 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 04:47:16.199484 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.185183 4509 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 04:47:16.199484 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.185186 4509 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 04:47:16.199923 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.185188 4509 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 04:47:16.199923 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.185191 4509 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 04:47:16.199923 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.185193 4509 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 04:47:16.199923 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.185196 4509 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 04:47:16.199923 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.185200 4509 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 04:47:16.199923 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.185202 4509 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 04:47:16.199923 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.185205 4509 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 04:47:16.199923 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.185207 4509 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 04:47:16.199923 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.185210 4509 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 04:47:16.199923 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.189187 4509 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 04 04:47:16.199923 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.189206 4509 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 04 04:47:16.199923 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.189225 4509 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 04 04:47:16.199923 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.189231 4509 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 04 04:47:16.199923 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189294 4509 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 04:47:16.199923 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189299 4509 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 04:47:16.199923 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189303 4509 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 04:47:16.199923 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189307 4509 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 04:47:16.199923 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189310 4509 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 04:47:16.199923 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189313 4509 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 04:47:16.199923 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189316 4509 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 04:47:16.200447 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189336 4509 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 04:47:16.200447 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189339 4509 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 04:47:16.200447 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189342 4509 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 04:47:16.200447 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189345 4509 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 04:47:16.200447 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189348 4509 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 04:47:16.200447 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189350 4509 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 04:47:16.200447 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189353 4509 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 04:47:16.200447 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189356 4509 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 04:47:16.200447 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189358 4509 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 04:47:16.200447 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189361 4509 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 04:47:16.200447 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189363 4509 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 04:47:16.200447 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189366 4509 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 04:47:16.200447 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189369 4509 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 04:47:16.200447 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189371 4509 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 04:47:16.200447 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189374 4509 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 04:47:16.200447 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189377 4509 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 04:47:16.200447 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189381 4509 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 04:47:16.200447 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189385 4509 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 04:47:16.200447 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189388 4509 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 04:47:16.201013 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189391 4509 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 04:47:16.201013 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189394 4509 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 04:47:16.201013 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189396 4509 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 04:47:16.201013 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189400 4509 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 04:47:16.201013 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189403 4509 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 04:47:16.201013 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189406 4509 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 04:47:16.201013 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189408 4509 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 04:47:16.201013 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189411 4509 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 04:47:16.201013 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189414 4509 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 04:47:16.201013 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189418 4509 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 04:47:16.201013 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189420 4509 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 04:47:16.201013 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189423 4509 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 04:47:16.201013 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189427 4509 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 04:47:16.201013 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189431 4509 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 04:47:16.201013 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189435 4509 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 04:47:16.201013 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189451 4509 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 04:47:16.201013 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189454 4509 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 04:47:16.201013 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189457 4509 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 04:47:16.201013 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189460 4509 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 04:47:16.201722 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189464 4509 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 04:47:16.201722 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189467 4509 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 04:47:16.201722 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189470 4509 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 04:47:16.201722 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189472 4509 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 04:47:16.201722 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189475 4509 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 04:47:16.201722 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189477 4509 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 04:47:16.201722 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189480 4509 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 04:47:16.201722 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189482 4509 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 04:47:16.201722 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189484 4509 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 04:47:16.201722 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189487 4509 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 04:47:16.201722 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189489 4509 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 04:47:16.201722 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189492 4509 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 04:47:16.201722 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189495 4509 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 04:47:16.201722 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189497 4509 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 04:47:16.201722 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189500 4509 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 04:47:16.201722 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189502 4509 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 04:47:16.201722 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189505 4509 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 04:47:16.201722 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189508 4509 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 04:47:16.201722 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189510 4509 feature_gate.go:386] unrecognized feature gate: Example Nov 04 04:47:16.201722 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189513 4509 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 04:47:16.202454 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189515 4509 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 04:47:16.202454 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189518 4509 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 04:47:16.202454 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189520 4509 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 04:47:16.202454 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189523 4509 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 04:47:16.202454 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189525 4509 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 04:47:16.202454 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189527 4509 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 04:47:16.202454 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189530 4509 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 04:47:16.202454 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189532 4509 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 04:47:16.202454 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189535 4509 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 04:47:16.202454 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189537 4509 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 04:47:16.202454 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189540 4509 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 04:47:16.202454 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189542 4509 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 04:47:16.202454 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189545 4509 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 04:47:16.202454 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189547 4509 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 04:47:16.202454 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189550 4509 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 04:47:16.202454 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189552 4509 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 04:47:16.202454 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189555 4509 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 04:47:16.202454 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189557 4509 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 04:47:16.202454 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189559 4509 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 04:47:16.202454 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189562 4509 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 04:47:16.202917 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189564 4509 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 04:47:16.202917 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189566 4509 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 04:47:16.202917 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189568 4509 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 04:47:16.202917 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189571 4509 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 04:47:16.202917 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189574 4509 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 04:47:16.202917 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189576 4509 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 04:47:16.202917 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189578 4509 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 04:47:16.202917 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189581 4509 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 04:47:16.202917 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189583 4509 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 04:47:16.202917 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189586 4509 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 04:47:16.202917 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189589 4509 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 04:47:16.202917 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189591 4509 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 04:47:16.202917 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189594 4509 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 04:47:16.202917 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189596 4509 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 04:47:16.202917 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189599 4509 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 04:47:16.202917 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189602 4509 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 04:47:16.202917 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189604 4509 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 04:47:16.202917 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189606 4509 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 04:47:16.202917 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189609 4509 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 04:47:16.202917 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189612 4509 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 04:47:16.203420 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189614 4509 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 04:47:16.203420 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.189641 4509 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 04 04:47:16.203420 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189784 4509 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 04:47:16.203420 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189788 4509 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 04:47:16.203420 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189792 4509 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 04:47:16.203420 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189795 4509 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 04:47:16.203420 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189797 4509 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 04:47:16.203420 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189799 4509 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 04:47:16.203420 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189802 4509 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 04:47:16.203420 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189805 4509 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 04:47:16.203420 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189807 4509 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 04:47:16.203420 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189810 4509 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 04:47:16.203420 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189813 4509 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 04:47:16.203420 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189815 4509 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 04:47:16.203420 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189817 4509 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 04:47:16.203420 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189820 4509 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 04:47:16.203802 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189822 4509 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 04:47:16.203802 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189825 4509 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 04:47:16.203802 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189827 4509 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 04:47:16.203802 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189829 4509 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 04:47:16.203802 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189832 4509 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 04:47:16.203802 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189834 4509 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 04:47:16.203802 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189836 4509 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 04:47:16.203802 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189840 4509 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 04:47:16.203802 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189842 4509 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 04:47:16.203802 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189845 4509 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 04:47:16.203802 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189847 4509 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 04:47:16.203802 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189849 4509 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 04:47:16.203802 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189852 4509 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 04:47:16.203802 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189854 4509 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 04:47:16.203802 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189856 4509 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 04:47:16.203802 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189859 4509 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 04:47:16.203802 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189861 4509 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 04:47:16.203802 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189863 4509 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 04:47:16.203802 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189867 4509 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 04:47:16.204236 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189871 4509 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 04:47:16.204236 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189873 4509 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 04:47:16.204236 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189876 4509 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 04:47:16.204236 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189878 4509 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 04:47:16.204236 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189884 4509 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 04:47:16.204236 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189887 4509 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 04:47:16.204236 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189889 4509 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 04:47:16.204236 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189892 4509 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 04:47:16.204236 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189894 4509 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 04:47:16.204236 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189897 4509 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 04:47:16.204236 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189899 4509 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 04:47:16.204236 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189901 4509 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 04:47:16.204236 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189904 4509 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 04:47:16.204236 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189906 4509 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 04:47:16.204236 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189909 4509 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 04:47:16.204236 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189911 4509 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 04:47:16.204236 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189914 4509 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 04:47:16.204236 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189916 4509 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 04:47:16.204236 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189919 4509 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 04:47:16.204236 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189921 4509 feature_gate.go:386] unrecognized feature gate: Example Nov 04 04:47:16.204770 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189923 4509 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 04:47:16.204770 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189927 4509 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 04:47:16.204770 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189930 4509 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 04:47:16.204770 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189932 4509 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 04:47:16.204770 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189934 4509 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 04:47:16.204770 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189937 4509 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 04:47:16.204770 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189939 4509 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 04:47:16.204770 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189942 4509 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 04:47:16.204770 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189944 4509 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 04:47:16.204770 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189947 4509 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 04:47:16.204770 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189949 4509 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 04:47:16.204770 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189951 4509 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 04:47:16.204770 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189954 4509 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 04:47:16.204770 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189956 4509 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 04:47:16.204770 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189958 4509 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 04:47:16.204770 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189961 4509 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 04:47:16.204770 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189965 4509 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 04:47:16.204770 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189967 4509 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 04:47:16.204770 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189970 4509 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 04:47:16.205223 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189972 4509 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 04:47:16.205223 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189975 4509 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 04:47:16.205223 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189977 4509 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 04:47:16.205223 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189980 4509 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 04:47:16.205223 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189983 4509 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 04:47:16.205223 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189985 4509 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 04:47:16.205223 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189988 4509 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 04:47:16.205223 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189990 4509 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 04:47:16.205223 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189993 4509 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 04:47:16.205223 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189995 4509 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 04:47:16.205223 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.189998 4509 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 04:47:16.205223 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.190000 4509 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 04:47:16.205223 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.190002 4509 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 04:47:16.205223 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.190005 4509 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 04:47:16.205223 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.190008 4509 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 04:47:16.205223 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.190055 4509 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 04:47:16.205223 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.190112 4509 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 04:47:16.205223 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.190117 4509 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 04:47:16.205223 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.190164 4509 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 04:47:16.205223 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.190169 4509 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 04:47:16.205802 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.190174 4509 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 04:47:16.205802 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.190181 4509 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 04:47:16.205802 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.190186 4509 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 04:47:16.205802 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.190191 4509 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 04:47:16.205802 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.190195 4509 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 04:47:16.205802 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.190200 4509 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 04:47:16.205802 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.190204 4509 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 04:47:16.205802 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.190210 4509 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 04:47:16.205802 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.190218 4509 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 04:47:16.205802 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.190223 4509 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 04:47:16.205802 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.190227 4509 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 04:47:16.205802 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.190232 4509 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 04:47:16.205802 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.190236 4509 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 04:47:16.205802 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.190240 4509 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 04:47:16.205802 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.190297 4509 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 04 04:47:16.206180 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.190428 4509 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 04:47:16.206180 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.190437 4509 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 04:47:16.206180 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.190442 4509 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 04:47:16.206180 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.190452 4509 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 04:47:16.206180 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.190458 4509 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 04:47:16.206180 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.190462 4509 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 04:47:16.206180 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.190467 4509 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 04:47:16.206180 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.190471 4509 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 04:47:16.206180 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.190475 4509 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 04:47:16.206180 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.190480 4509 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 04:47:16.206180 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.190484 4509 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 04:47:16.206180 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.190488 4509 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 04:47:16.206180 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.190492 4509 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 04:47:16.206180 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.190926 4509 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 04:47:16.206180 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.190932 4509 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 04:47:16.206180 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.190937 4509 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 04:47:16.206180 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.190946 4509 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 04:47:16.206180 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.190951 4509 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 04:47:16.206180 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.190955 4509 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 04:47:16.206180 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.190959 4509 feature_gate.go:386] unrecognized feature gate: Example Nov 04 04:47:16.206736 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.190963 4509 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 04:47:16.206736 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.190967 4509 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 04:47:16.206736 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.190971 4509 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 04:47:16.206736 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.190975 4509 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 04:47:16.206736 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.190980 4509 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 04:47:16.206736 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.190984 4509 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 04:47:16.206736 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.190988 4509 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 04:47:16.206736 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.190992 4509 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 04:47:16.206736 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.191001 4509 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 04:47:16.206736 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.191011 4509 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 04:47:16.206736 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.191015 4509 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 04:47:16.206736 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.191020 4509 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 04:47:16.206736 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.191024 4509 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 04:47:16.206736 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.191028 4509 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 04:47:16.206736 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.191032 4509 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 04:47:16.206736 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.191037 4509 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 04:47:16.206736 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.191041 4509 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 04:47:16.206736 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.191045 4509 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 04:47:16.206736 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.191049 4509 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 04:47:16.207619 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.191054 4509 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 04:47:16.207619 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.191058 4509 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 04:47:16.207619 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.191067 4509 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 04:47:16.207619 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.191071 4509 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 04:47:16.207619 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.191076 4509 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 04:47:16.207619 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.191083 4509 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 04:47:16.207619 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.191086 4509 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 04:47:16.207619 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.191092 4509 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 04:47:16.207619 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.191096 4509 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 04:47:16.207619 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.191100 4509 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 04:47:16.207619 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.191104 4509 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 04:47:16.207619 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.191108 4509 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 04:47:16.207619 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.191112 4509 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 04:47:16.207619 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.191117 4509 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 04:47:16.207619 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.191121 4509 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 04:47:16.207619 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.191132 4509 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 04:47:16.207619 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.191136 4509 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 04:47:16.207619 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.191140 4509 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 04:47:16.207619 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.191144 4509 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 04:47:16.207619 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.191149 4509 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 04:47:16.208500 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.191153 4509 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 04:47:16.208500 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.191159 4509 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 04:47:16.208500 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.191167 4509 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 04:47:16.208500 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.191171 4509 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 04:47:16.208500 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.191175 4509 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 04:47:16.208500 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.191179 4509 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 04:47:16.208500 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.191189 4509 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 04:47:16.208500 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.191193 4509 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 04:47:16.208500 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.191197 4509 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 04:47:16.208500 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.191201 4509 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 04:47:16.208500 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.191205 4509 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 04:47:16.208500 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.191209 4509 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 04:47:16.208500 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.191213 4509 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 04:47:16.208500 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.191217 4509 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 04:47:16.208500 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.191221 4509 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 04:47:16.208500 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.191225 4509 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 04:47:16.208500 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.191229 4509 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 04:47:16.208500 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.191233 4509 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 04:47:16.208500 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.191238 4509 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 04:47:16.209186 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.191246 4509 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 04:47:16.209186 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.191252 4509 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 04:47:16.209186 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.191256 4509 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 04:47:16.209186 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.191261 4509 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 04:47:16.209186 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.191265 4509 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 04:47:16.209186 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.191269 4509 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 04:47:16.209186 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.191273 4509 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 04:47:16.209186 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.191296 4509 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 04:47:16.209186 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.191456 4509 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 04:47:16.209186 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.191461 4509 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 04:47:16.209186 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.191464 4509 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 04:47:16.209186 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.191468 4509 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 04:47:16.209186 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.191471 4509 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 04:47:16.209186 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.191474 4509 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 04:47:16.209186 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.191476 4509 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 04:47:16.209186 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.191479 4509 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 04:47:16.209186 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.191482 4509 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 04:47:16.209186 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.191484 4509 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 04:47:16.209186 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.191487 4509 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 04:47:16.209186 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.191490 4509 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 04:47:16.209991 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.191492 4509 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 04:47:16.209991 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.191495 4509 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 04:47:16.209991 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.191498 4509 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 04:47:16.209991 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.191500 4509 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 04:47:16.209991 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.191503 4509 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 04:47:16.209991 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.191506 4509 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 04:47:16.209991 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.191508 4509 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 04:47:16.209991 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:16.191516 4509 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 04:47:16.209991 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.191614 4509 server.go:957] "Client rotation is on, will bootstrap in background" Nov 04 04:47:16.209991 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.192648 4509 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Nov 04 04:47:16.209991 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.192711 4509 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Nov 04 04:47:16.209991 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.192895 4509 server.go:1014] "Starting client certificate rotation" Nov 04 04:47:16.209991 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.193029 4509 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 04 04:47:16.209991 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.193090 4509 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-03 04:41:37 +0000 UTC" deadline="2028-06-16 10:42:22.926956588 +0000 UTC" Nov 04 04:47:16.209991 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.193126 4509 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14165h55m6.733832863s" Nov 04 04:47:16.209991 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.193376 4509 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 04 04:47:16.210691 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.194492 4509 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 04 04:47:16.210691 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.196125 4509 log.go:25] "Validated CRI v1 runtime API" Nov 04 04:47:16.210691 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.197831 4509 log.go:25] "Validated CRI v1 image API" Nov 04 04:47:16.210691 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.198549 4509 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 04 04:47:16.210691 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.200087 4509 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 8284cc23-11ba-4c56-b47a-6f747cf2d934:/dev/nvme0n1p3 c3d7e991-8daa-45be-bbbc-3a25fd29be14:/dev/nvme0n1p4] Nov 04 04:47:16.210917 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.200108 4509 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/07780832d01bfc20e7d9d4e0447f6b33fa59657b03493e9cf6835dbff988e3a7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/07780832d01bfc20e7d9d4e0447f6b33fa59657b03493e9cf6835dbff988e3a7/userdata/shm major:0 minor:74 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/08cbd96522b2f06bb323ee7dac9b9d159935e1ce665c5d9dd78e426e9a662fcf/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/08cbd96522b2f06bb323ee7dac9b9d159935e1ce665c5d9dd78e426e9a662fcf/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/10a3c53136d5e01c314b91221a5a31445eaf198fc677922193fc805f9c033083/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/10a3c53136d5e01c314b91221a5a31445eaf198fc677922193fc805f9c033083/userdata/shm major:0 minor:223 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1e385cdda0d1bccd6f07d5afcdaee46c8a729a2ec16ed217e691159218ab711f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1e385cdda0d1bccd6f07d5afcdaee46c8a729a2ec16ed217e691159218ab711f/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/319071750bc643809077f76db9d2220e29cc5c1a3924dd48f8a4608fb71cd798/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/319071750bc643809077f76db9d2220e29cc5c1a3924dd48f8a4608fb71cd798/userdata/shm major:0 minor:62 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/32ef11380c8ec7870a0852b0413a2c0c70f4ea455b4889ff21c914c5c95ace32/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/32ef11380c8ec7870a0852b0413a2c0c70f4ea455b4889ff21c914c5c95ace32/userdata/shm major:0 minor:203 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/39a9f52ed78a8bc4487051e839eee3354641e60a5732aede57702167ec597ace/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/39a9f52ed78a8bc4487051e839eee3354641e60a5732aede57702167ec597ace/userdata/shm major:0 minor:210 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/573341ae49ba363b5674428f3452af54a79baeec54fbc99e3391040e0213b64e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/573341ae49ba363b5674428f3452af54a79baeec54fbc99e3391040e0213b64e/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/61c7ccd8c37f4b1903256dd5a442b7df3c9fe02b842028cc54b884875b60942a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/61c7ccd8c37f4b1903256dd5a442b7df3c9fe02b842028cc54b884875b60942a/userdata/shm major:0 minor:60 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/699641a78e73f5bb420a0efb52bc044af718fdc4a940b1c5f2a5a01cd780941c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/699641a78e73f5bb420a0efb52bc044af718fdc4a940b1c5f2a5a01cd780941c/userdata/shm major:0 minor:77 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6d3be9a0db0ee308071b3422d53c6844098593216d463c6863ec775c3e47e798/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6d3be9a0db0ee308071b3422d53c6844098593216d463c6863ec775c3e47e798/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/861a0607ea89dfcd8263350e4f58f1cfa403d43419d2e64f3d1a7227d12e34fc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/861a0607ea89dfcd8263350e4f58f1cfa403d43419d2e64f3d1a7227d12e34fc/userdata/shm major:0 minor:214 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9025e93209d4a5f767ba5a97ab0f507331a3e676c4eb362351de7da833c8c582/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9025e93209d4a5f767ba5a97ab0f507331a3e676c4eb362351de7da833c8c582/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9a9d05d8927347bce1aaee6783a92cb2e0d8550b2ab0cabbc0015124f74b024e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9a9d05d8927347bce1aaee6783a92cb2e0d8550b2ab0cabbc0015124f74b024e/userdata/shm major:0 minor:61 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b3dc53f0a90e45b5e090ed31fa76115def5e68ba143668be0c4aeae7f2177c08/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b3dc53f0a90e45b5e090ed31fa76115def5e68ba143668be0c4aeae7f2177c08/userdata/shm major:0 minor:212 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/00e0471e-5fe8-4494-8f17-632aee4f6c30/volumes/kubernetes.io~projected/kube-api-access-gftxp:{mountpoint:/var/lib/kubelet/pods/00e0471e-5fe8-4494-8f17-632aee4f6c30/volumes/kubernetes.io~projected/kube-api-access-gftxp major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/198e9ab0-f615-4686-9acf-a6242f7e3584/volumes/kubernetes.io~projected/kube-api-access-ccfxs:{mountpoint:/var/lib/kubelet/pods/198e9ab0-f615-4686-9acf-a6242f7e3584/volumes/kubernetes.io~projected/kube-api-access-ccfxs major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/21f0e203-fdf7-42cf-9c0c-4103e6601020/volumes/kubernetes.io~projected/kube-api-access-x2qjt:{mountpoint:/var/lib/kubelet/pods/21f0e203-fdf7-42cf-9c0c-4103e6601020/volumes/kubernetes.io~projected/kube-api-access-x2qjt major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/21f0e203-fdf7-42cf-9c0c-4103e6601020/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/21f0e203-fdf7-42cf-9c0c-4103e6601020/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/22d31d0c-6edf-4351-8f41-a6dd4e02368b/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/22d31d0c-6edf-4351-8f41-a6dd4e02368b/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/22d31d0c-6edf-4351-8f41-a6dd4e02368b/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/22d31d0c-6edf-4351-8f41-a6dd4e02368b/volumes/kubernetes.io~empty-dir/tmp major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/22d31d0c-6edf-4351-8f41-a6dd4e02368b/volumes/kubernetes.io~projected/kube-api-access-zbtxf:{mountpoint:/var/lib/kubelet/pods/22d31d0c-6edf-4351-8f41-a6dd4e02368b/volumes/kubernetes.io~projected/kube-api-access-zbtxf major:0 minor:54 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/23ed4a39-18b5-4bc6-9c7b-8b60a7043ad2/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/23ed4a39-18b5-4bc6-9c7b-8b60a7043ad2/volumes/kubernetes.io~secret/agent-certs major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4dff9faf-d91f-4ed5-be82-555b9a6a924e/volumes/kubernetes.io~projected/kube-api-access-hx8dr:{mountpoint:/var/lib/kubelet/pods/4dff9faf-d91f-4ed5-be82-555b9a6a924e/volumes/kubernetes.io~projected/kube-api-access-hx8dr major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4dff9faf-d91f-4ed5-be82-555b9a6a924e/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/4dff9faf-d91f-4ed5-be82-555b9a6a924e/volumes/kubernetes.io~secret/metrics-certs major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/58be7092-9156-4787-8f49-a173fa0bd5a3/volumes/kubernetes.io~projected/kube-api-access-w4xvx:{mountpoint:/var/lib/kubelet/pods/58be7092-9156-4787-8f49-a173fa0bd5a3/volumes/kubernetes.io~projected/kube-api-access-w4xvx major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5f6922fa-9ba7-4d02-b550-606731852fde/volumes/kubernetes.io~projected/kube-api-access-47nf5:{mountpoint:/var/lib/kubelet/pods/5f6922fa-9ba7-4d02-b550-606731852fde/volumes/kubernetes.io~projected/kube-api-access-47nf5 major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8446aa71-5638-4b23-8f5b-12283ede21d4/volumes/kubernetes.io~projected/kube-api-access-25g6k:{mountpoint:/var/lib/kubelet/pods/8446aa71-5638-4b23-8f5b-12283ede21d4/volumes/kubernetes.io~projected/kube-api-access-25g6k major:0 minor:52 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/844cc12f-a7f6-4327-968e-5e9b2da00ad0/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/844cc12f-a7f6-4327-968e-5e9b2da00ad0/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/844cc12f-a7f6-4327-968e-5e9b2da00ad0/volumes/kubernetes.io~projected/kube-api-access-6nch9:{mountpoint:/var/lib/kubelet/pods/844cc12f-a7f6-4327-968e-5e9b2da00ad0/volumes/kubernetes.io~projected/kube-api-access-6nch9 major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/844cc12f-a7f6-4327-968e-5e9b2da00ad0/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/844cc12f-a7f6-4327-968e-5e9b2da00ad0/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9a06ec07-c72b-43af-aa27-7414aa9c9aba/volumes/kubernetes.io~projected/kube-api-access-mlz69:{mountpoint:/var/lib/kubelet/pods/9a06ec07-c72b-43af-aa27-7414aa9c9aba/volumes/kubernetes.io~projected/kube-api-access-mlz69 major:0 minor:53 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c79eff94-6543-4313-b59b-502ca4089e84/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/c79eff94-6543-4313-b59b-502ca4089e84/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cce8fa86-4f61-4dfc-b36c-2cefb7ecaa03/volumes/kubernetes.io~projected/kube-api-access-7cgfm:{mountpoint:/var/lib/kubelet/pods/cce8fa86-4f61-4dfc-b36c-2cefb7ecaa03/volumes/kubernetes.io~projected/kube-api-access-7cgfm major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d737ccc2-5686-4130-9d00-d2f21d5b417f/volumes/kubernetes.io~projected/kube-api-access-kz2hd:{mountpoint:/var/lib/kubelet/pods/d737ccc2-5686-4130-9d00-d2f21d5b417f/volumes/kubernetes.io~projected/kube-api-access-kz2hd major:0 minor:51 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dd0aaef0-d114-43b4-9ce4-135e818e4df3/volumes/kubernetes.io~projected/kube-api-access-7l4mv:{mountpoint:/var/lib/kubelet/pods/dd0aaef0-d114-43b4-9ce4-135e818e4df3/volumes/kubernetes.io~projected/kube-api-access-7l4mv major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dd0aaef0-d114-43b4-9ce4-135e818e4df3/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/dd0aaef0-d114-43b4-9ce4-135e818e4df3/volumes/kubernetes.io~secret/metrics-tls major:0 minor:222 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/4e253e2c145fa464026f33ab402030ae9f02221faebf77e820ddb5925f2881d5/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/9526eaee50077acb072cc7b7181f21b60b071d009785a1c8a6bdc353bc6ad6f1/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/2ec4bbea09f0d525555895c6beea987c33c062b059f3873b02df814045e54992/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/0d841481150d4f8ade6b4bfaadbe79c4d3bf10eeefb7a77c7c9d766b8ef8d6fd/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/7b71ef42aa4d9f58de97a643e570033014d85c8d8ebb05a1be4d1d0d51429a4a/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-118:{mountpoint:/var/lib/containers/storage/overlay/1ea2067b4fe4b511412a4c3d0f82a13d5873cdc20969e3dbb2cf1a5493b65247/merged major:0 minor:118 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/74a10154de56f91f2e1128e4775b435586daff94cd603ecf246ecbef48865018/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/cf15aa00cfe494c54080173d672865370ef32af135dd7cb153dbedaea99e164e/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/4f9060a555cc1005570af7e79342929dfbb198bc4719c767b8f575e55650a049/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/2a9bbf8ba8c123e5b06a799426e0261fd60c1d4bf73110ef95722dfdadf7f1b2/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/4b96ff6e47ea7021ae100ed0dee153520b71139d6668f7d9d3236b205dbbc637/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-147:{mountpoint:/var/lib/containers/storage/overlay/eadeb83535b0dd2b096b0c2aadc8773711f5766d279d11666a51a67f0a6e598d/merged major:0 minor:147 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/5987498ab606b129c934d9b2e6b485e5fa5dfa0dde3ca3b432ce86234c5c9d4a/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-154:{mountpoint:/var/lib/containers/storage/overlay/c28aed819497e7081025f09edddd67321865781a5b60e7a2cdf7e7cdd574a8d8/merged major:0 minor:154 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/22d07609be2d7890514ff3a46e8aaef25351dfc6bd311df631aad47b7fed911e/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/9b13a0e4388c92c8d45372263ba2d0fc97cea80ae492d1c6f896e82d0d8bcbab/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/756e2ff4c25e5092fc18410c15d83581e6baa2af44cf0703bdca3c3342d93fd9/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/f589d7af504d72f127764212830d7fee7d99c27c5538e514207848a958239481/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/066999a19425a5c620c48ffad64d5e774358a509e2d713c9ec4222c5109354aa/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/694ec42ed54f245c1b9975db2785747ee0fd5ff06a8e44d6c2682ad86fa81b14/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/e4f51f18056fc3bddba72d9d0573b426fb6ae40276ab8b979ae0a6a8c8bc7fbc/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-205:{mountpoint:/var/lib/containers/storage/overlay/eb5941a120048f7dd7086e5916f041cd596b3869d0edba332f0dc1dc63475fda/merged major:0 minor:205 fsType:overlay blockSize:0} overlay_0-216:{mountpoint:/var/lib/containers/storage/overlay/aee303dbbb0379a83b7440862bcf5cc615fc67bef97e65393a2f07de59882b31/merged major:0 minor:216 fsType:overlay blockSize:0} overlay_0-218:{mountpoint:/var/lib/containers/storage/overlay/96b8f6108a4b358242e1cb5281016e6f0a53ed76426f26d203d74ad7f0654d5c/merged major:0 minor:218 fsType:overlay blockSize:0} overlay_0-220:{mountpoint:/var/lib/containers/storage/overlay/50f59c3aec8ea1caa601010d9d210b9a8224e068a838de71dec0848f07c8d073/merged major:0 minor:220 fsType:overlay blockSize:0} overlay_0-225:{mountpoint:/var/lib/containers/storage/overlay/420366eaab93db5079e3e3583bda6129c05d3289dc7c810f7ba24fdc45ef9c40/merged major:0 minor:225 fsType:overlay blockSize:0} overlay_0-227:{mountpoint:/var/lib/containers/storage/overlay/db520d873b2977b2270a7e627a2f38924596850ac0b8cf73c9f404c463bf4094/merged major:0 minor:227 fsType:overlay blockSize:0} overlay_0-233:{mountpoint:/var/lib/containers/storage/overlay/75f22426a62f203fa3ae9b8cbb0a2d30d3e9e3184b1000526c854f39fd9bb408/merged major:0 minor:233 fsType:overlay blockSize:0} overlay_0-235:{mountpoint:/var/lib/containers/storage/overlay/5d017b9caf3e08757033b188c2afc30c3ad153172c64cd17635d5d0764d5321a/merged major:0 minor:235 fsType:overlay blockSize:0} overlay_0-241:{mountpoint:/var/lib/containers/storage/overlay/8f8abeacb9d4e13ba93d555a11d4321a0995720002871835567e7d032f2001e1/merged major:0 minor:241 fsType:overlay blockSize:0} overlay_0-246:{mountpoint:/var/lib/containers/storage/overlay/25ae5604adabbe8382dc75097f49bcb1314577e46754162a633b3d49a92909a3/merged major:0 minor:246 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/ec183b30b7afd43eebb7a6aeb6cad82100db815efb525025bbdc1994d1ad1d94/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/35c11d691dbadd2c9a3f8b4686950de836a5d6e72e0d22a1dbeb61fd938e0dec/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-66:{mountpoint:/var/lib/containers/storage/overlay/1b633201d9f4a439b713ad67c3c195141054662008ce245fb81eb83c896b02b3/merged major:0 minor:66 fsType:overlay blockSize:0} overlay_0-68:{mountpoint:/var/lib/containers/storage/overlay/f9df6875bbb5ef38c62922ff7390762202e6d1f2ec8b553371ceaad4ce7a8851/merged major:0 minor:68 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/1232007e727f7abd9c98d0bd9a1653fdec04868120f5f64f8220edcd9696085e/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/e3b0f952532e22ecc6622857a3ac78098de05fbd7c814be13982c61949728b1b/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/d09ef703f39a0cf8510eec3cdb5f8835d81d784c34ad01a1830ee3d5e4c7a4a0/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/cd149f25da427f2feb1cbc3a638b3ea5364fa8bc307232b57df4f26e47243aa8/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/a48e388f86d77e1ec4c13e9f2ea6c0c6eac85204383195f1f73046c621029a0c/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/084735d231b3928dd5a2a75656bac53d5452235a3b852615137c168da39f9c6c/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/0626477191d33940ddfd076e9724febfeac445dc3c91699b1beecc39d69bc727/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/3ec3ad9b2b3a1186ba997e7762a05b823912c6d20a3dea9ae8a834005317fd52/merged major:0 minor:96 fsType:overlay blockSize:0}] Nov 04 04:47:16.211139 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.206151 4509 manager.go:217] Machine: {Timestamp:2026-11-04 04:47:16.205695185 +0000 UTC m=+0.083418696 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3199868 MemoryCapacity:32812019712 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec28fd50b4bd208995dd578b7a38fd2c SystemUUID:ec28fd50-b4bd-2089-95dd-578b7a38fd2c BootID:667e3c15-2317-4bec-8bd3-7357e4534b20 Filesystems:[{Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/699641a78e73f5bb420a0efb52bc044af718fdc4a940b1c5f2a5a01cd780941c/userdata/shm DeviceMajor:0 DeviceMinor:77 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-246 DeviceMajor:0 DeviceMinor:246 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/4dff9faf-d91f-4ed5-be82-555b9a6a924e/volumes/kubernetes.io~projected/kube-api-access-hx8dr DeviceMajor:0 DeviceMinor:55 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1e385cdda0d1bccd6f07d5afcdaee46c8a729a2ec16ed217e691159218ab711f/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-205 DeviceMajor:0 DeviceMinor:205 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/39a9f52ed78a8bc4487051e839eee3354641e60a5732aede57702167ec597ace/userdata/shm DeviceMajor:0 DeviceMinor:210 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/198e9ab0-f615-4686-9acf-a6242f7e3584/volumes/kubernetes.io~projected/kube-api-access-ccfxs DeviceMajor:0 DeviceMinor:58 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/61c7ccd8c37f4b1903256dd5a442b7df3c9fe02b842028cc54b884875b60942a/userdata/shm DeviceMajor:0 DeviceMinor:60 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/21f0e203-fdf7-42cf-9c0c-4103e6601020/volumes/kubernetes.io~projected/kube-api-access-x2qjt DeviceMajor:0 DeviceMinor:201 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/32ef11380c8ec7870a0852b0413a2c0c70f4ea455b4889ff21c914c5c95ace32/userdata/shm DeviceMajor:0 DeviceMinor:203 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/c79eff94-6543-4313-b59b-502ca4089e84/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:208 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-147 DeviceMajor:0 DeviceMinor:147 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/dd0aaef0-d114-43b4-9ce4-135e818e4df3/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:222 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-218 DeviceMajor:0 DeviceMinor:218 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-220 DeviceMajor:0 DeviceMinor:220 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/10a3c53136d5e01c314b91221a5a31445eaf198fc677922193fc805f9c033083/userdata/shm DeviceMajor:0 DeviceMinor:223 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-225 DeviceMajor:0 DeviceMinor:225 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/844cc12f-a7f6-4327-968e-5e9b2da00ad0/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:48 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/22d31d0c-6edf-4351-8f41-a6dd4e02368b/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:50 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/844cc12f-a7f6-4327-968e-5e9b2da00ad0/volumes/kubernetes.io~projected/kube-api-access-6nch9 DeviceMajor:0 DeviceMinor:57 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/00e0471e-5fe8-4494-8f17-632aee4f6c30/volumes/kubernetes.io~projected/kube-api-access-gftxp DeviceMajor:0 DeviceMinor:59 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/319071750bc643809077f76db9d2220e29cc5c1a3924dd48f8a4608fb71cd798/userdata/shm DeviceMajor:0 DeviceMinor:62 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/844cc12f-a7f6-4327-968e-5e9b2da00ad0/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9a06ec07-c72b-43af-aa27-7414aa9c9aba/volumes/kubernetes.io~projected/kube-api-access-mlz69 DeviceMajor:0 DeviceMinor:53 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-66 DeviceMajor:0 DeviceMinor:66 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cce8fa86-4f61-4dfc-b36c-2cefb7ecaa03/volumes/kubernetes.io~projected/kube-api-access-7cgfm DeviceMajor:0 DeviceMinor:197 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/5f6922fa-9ba7-4d02-b550-606731852fde/volumes/kubernetes.io~projected/kube-api-access-47nf5 DeviceMajor:0 DeviceMinor:209 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406011904 Type:vfs Inodes:1048576 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6d3be9a0db0ee308071b3422d53c6844098593216d463c6863ec775c3e47e798/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-118 DeviceMajor:0 DeviceMinor:118 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-227 DeviceMajor:0 DeviceMinor:227 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9a9d05d8927347bce1aaee6783a92cb2e0d8550b2ab0cabbc0015124f74b024e/userdata/shm DeviceMajor:0 DeviceMinor:61 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/573341ae49ba363b5674428f3452af54a79baeec54fbc99e3391040e0213b64e/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/861a0607ea89dfcd8263350e4f58f1cfa403d43419d2e64f3d1a7227d12e34fc/userdata/shm DeviceMajor:0 DeviceMinor:214 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-216 DeviceMajor:0 DeviceMinor:216 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-233 DeviceMajor:0 DeviceMinor:233 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/8446aa71-5638-4b23-8f5b-12283ede21d4/volumes/kubernetes.io~projected/kube-api-access-25g6k DeviceMajor:0 DeviceMinor:52 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/22d31d0c-6edf-4351-8f41-a6dd4e02368b/volumes/kubernetes.io~projected/kube-api-access-zbtxf DeviceMajor:0 DeviceMinor:54 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9025e93209d4a5f767ba5a97ab0f507331a3e676c4eb362351de7da833c8c582/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b3dc53f0a90e45b5e090ed31fa76115def5e68ba143668be0c4aeae7f2177c08/userdata/shm DeviceMajor:0 DeviceMinor:212 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-241 DeviceMajor:0 DeviceMinor:241 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/58be7092-9156-4787-8f49-a173fa0bd5a3/volumes/kubernetes.io~projected/kube-api-access-w4xvx DeviceMajor:0 DeviceMinor:56 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/21f0e203-fdf7-42cf-9c0c-4103e6601020/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/22d31d0c-6edf-4351-8f41-a6dd4e02368b/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:47 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/08cbd96522b2f06bb323ee7dac9b9d159935e1ce665c5d9dd78e426e9a662fcf/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-235 DeviceMajor:0 DeviceMinor:235 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/23ed4a39-18b5-4bc6-9c7b-8b60a7043ad2/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:49 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4dff9faf-d91f-4ed5-be82-555b9a6a924e/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:207 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-68 DeviceMajor:0 DeviceMinor:68 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/07780832d01bfc20e7d9d4e0447f6b33fa59657b03493e9cf6835dbff988e3a7/userdata/shm DeviceMajor:0 DeviceMinor:74 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d737ccc2-5686-4130-9d00-d2f21d5b417f/volumes/kubernetes.io~projected/kube-api-access-kz2hd DeviceMajor:0 DeviceMinor:51 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-154 DeviceMajor:0 DeviceMinor:154 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/dd0aaef0-d114-43b4-9ce4-135e818e4df3/volumes/kubernetes.io~projected/kube-api-access-7l4mv DeviceMajor:0 DeviceMinor:202 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:10a3c53136d5e01 MacAddress:2e:93:0d:0a:20:b8 Speed:10000 Mtu:8901} {Name:32ef11380c8ec78 MacAddress:62:78:df:08:72:9e Speed:10000 Mtu:8901} {Name:39a9f52ed78a8bc MacAddress:8a:79:22:5e:6a:d6 Speed:10000 Mtu:8901} {Name:861a0607ea89dfc MacAddress:86:71:fb:43:e9:2d Speed:10000 Mtu:8901} {Name:b3dc53f0a90e45b MacAddress:b6:8a:59:09:9d:ea Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:5a:64:b6:d5:bb Speed:0 Mtu:9001} {Name:br-int MacAddress:6e:c4:fa:d4:b7:48 Speed:0 Mtu:8901} {Name:ens5 MacAddress:02:5a:64:b6:d5:bb Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:de:a0:41:5e:81:b3 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:16:8b:a8:fa:d3:2d Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812019712 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 04 04:47:16.211139 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.206434 4509 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 04 04:47:16.211139 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.206487 4509 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 04 04:47:16.211139 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.206876 4509 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 04 04:47:16.211139 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.206906 4509 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-180.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 04 04:47:16.211139 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.207108 4509 topology_manager.go:143] "Creating topology manager with none policy" Nov 04 04:47:16.211139 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.207121 4509 container_manager_linux.go:311] "Creating device plugin manager" Nov 04 04:47:16.211139 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.207147 4509 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 04 04:47:16.211139 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.207174 4509 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 04 04:47:16.211139 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.207313 4509 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 04 04:47:16.211139 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.207570 4509 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 04 04:47:16.211139 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.207732 4509 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 04 04:47:16.211139 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.207781 4509 kubelet.go:493] "Attempting to sync node with API server" Nov 04 04:47:16.211139 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.207802 4509 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 04 04:47:16.211139 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.207816 4509 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 04 04:47:16.211139 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.207832 4509 kubelet.go:405] "Adding apiserver pod source" Nov 04 04:47:16.211139 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.207844 4509 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 04 04:47:16.211139 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.209150 4509 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 04 04:47:16.211139 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.209166 4509 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 04 04:47:16.211139 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.210062 4509 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 04 04:47:16.213278 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.213261 4509 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 04 04:47:16.213278 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.213278 4509 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 04 04:47:16.213437 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.213426 4509 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Nov 04 04:47:16.213649 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.213637 4509 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 04 04:47:16.213685 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.213667 4509 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 04 04:47:16.213779 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.213769 4509 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 04 04:47:16.213810 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.213791 4509 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 04 04:47:16.213810 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.213802 4509 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 04 04:47:16.213861 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.213810 4509 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 04 04:47:16.213861 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.213816 4509 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 04 04:47:16.213861 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.213823 4509 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 04 04:47:16.213861 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.213828 4509 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 04 04:47:16.213861 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.213834 4509 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 04 04:47:16.213861 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.213841 4509 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 04 04:47:16.213861 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.213847 4509 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 04 04:47:16.213861 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.213857 4509 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 04 04:47:16.214065 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.213871 4509 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 04 04:47:16.214065 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.213901 4509 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 04 04:47:16.214065 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.213911 4509 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 04 04:47:16.217407 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.217393 4509 server.go:1263] "Started kubelet" Nov 04 04:47:16.217512 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.217482 4509 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 04 04:47:16.217642 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.217558 4509 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 04 04:47:16.217757 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.217678 4509 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 04 04:47:16.217991 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.217974 4509 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 04 04:47:16.217992 ip-10-0-133-180 systemd[1]: Started Kubernetes Kubelet. Nov 04 04:47:16.219458 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.219441 4509 server.go:317] "Adding debug handlers to kubelet server" Nov 04 04:47:16.220656 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.220634 4509 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 04 04:47:16.221060 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.221041 4509 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 04 04:47:16.221060 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.221058 4509 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 04 04:47:16.221895 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.221360 4509 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 04 04:47:16.221895 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.221543 4509 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-03 04:41:38 +0000 UTC" deadline="2028-06-05 17:00:34.112136068 +0000 UTC" Nov 04 04:47:16.221895 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.221594 4509 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13908h13m17.890545585s" Nov 04 04:47:16.222496 ip-10-0-133-180 kubenswrapper[4509]: E1104 04:47:16.222423 4509 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-180.us-west-1.compute.internal\" not found" Nov 04 04:47:16.222807 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.222737 4509 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 04 04:47:16.224624 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.224570 4509 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="198e9ab0-f615-4686-9acf-a6242f7e3584" volumeName="kubernetes.io/projected/198e9ab0-f615-4686-9acf-a6242f7e3584-kube-api-access-ccfxs" seLinuxMountContext="" Nov 04 04:47:16.224624 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.224615 4509 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="58be7092-9156-4787-8f49-a173fa0bd5a3" volumeName="kubernetes.io/projected/58be7092-9156-4787-8f49-a173fa0bd5a3-kube-api-access-w4xvx" seLinuxMountContext="" Nov 04 04:47:16.224742 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.224630 4509 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="844cc12f-a7f6-4327-968e-5e9b2da00ad0" volumeName="kubernetes.io/configmap/844cc12f-a7f6-4327-968e-5e9b2da00ad0-env-overrides" seLinuxMountContext="" Nov 04 04:47:16.224742 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.224644 4509 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="844cc12f-a7f6-4327-968e-5e9b2da00ad0" volumeName="kubernetes.io/secret/844cc12f-a7f6-4327-968e-5e9b2da00ad0-ovn-node-metrics-cert" seLinuxMountContext="" Nov 04 04:47:16.224742 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.224656 4509 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4dff9faf-d91f-4ed5-be82-555b9a6a924e" volumeName="kubernetes.io/projected/4dff9faf-d91f-4ed5-be82-555b9a6a924e-kube-api-access-hx8dr" seLinuxMountContext="" Nov 04 04:47:16.224742 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.224668 4509 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="198e9ab0-f615-4686-9acf-a6242f7e3584" volumeName="kubernetes.io/configmap/198e9ab0-f615-4686-9acf-a6242f7e3584-cni-sysctl-allowlist" seLinuxMountContext="" Nov 04 04:47:16.224742 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.224680 4509 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="22d31d0c-6edf-4351-8f41-a6dd4e02368b" volumeName="kubernetes.io/projected/22d31d0c-6edf-4351-8f41-a6dd4e02368b-kube-api-access-zbtxf" seLinuxMountContext="" Nov 04 04:47:16.224742 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.224691 4509 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c79eff94-6543-4313-b59b-502ca4089e84" volumeName="kubernetes.io/secret/c79eff94-6543-4313-b59b-502ca4089e84-original-pull-secret" seLinuxMountContext="" Nov 04 04:47:16.224742 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.224705 4509 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cce8fa86-4f61-4dfc-b36c-2cefb7ecaa03" volumeName="kubernetes.io/configmap/cce8fa86-4f61-4dfc-b36c-2cefb7ecaa03-kube-rbac-proxy-cm" seLinuxMountContext="" Nov 04 04:47:16.224742 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.224717 4509 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="21f0e203-fdf7-42cf-9c0c-4103e6601020" volumeName="kubernetes.io/secret/21f0e203-fdf7-42cf-9c0c-4103e6601020-cert" seLinuxMountContext="" Nov 04 04:47:16.224742 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.224729 4509 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="23ed4a39-18b5-4bc6-9c7b-8b60a7043ad2" volumeName="kubernetes.io/configmap/23ed4a39-18b5-4bc6-9c7b-8b60a7043ad2-konnectivity-ca" seLinuxMountContext="" Nov 04 04:47:16.224742 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.224741 4509 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4dff9faf-d91f-4ed5-be82-555b9a6a924e" volumeName="kubernetes.io/secret/4dff9faf-d91f-4ed5-be82-555b9a6a924e-metrics-certs" seLinuxMountContext="" Nov 04 04:47:16.224742 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.224751 4509 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="198e9ab0-f615-4686-9acf-a6242f7e3584" volumeName="kubernetes.io/configmap/198e9ab0-f615-4686-9acf-a6242f7e3584-whereabouts-flatfile-configmap" seLinuxMountContext="" Nov 04 04:47:16.225067 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.224762 4509 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="198e9ab0-f615-4686-9acf-a6242f7e3584" volumeName="kubernetes.io/configmap/198e9ab0-f615-4686-9acf-a6242f7e3584-cni-binary-copy" seLinuxMountContext="" Nov 04 04:47:16.225067 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.224775 4509 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="22d31d0c-6edf-4351-8f41-a6dd4e02368b" volumeName="kubernetes.io/empty-dir/22d31d0c-6edf-4351-8f41-a6dd4e02368b-etc-tuned" seLinuxMountContext="" Nov 04 04:47:16.225067 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.224785 4509 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="58be7092-9156-4787-8f49-a173fa0bd5a3" volumeName="kubernetes.io/configmap/58be7092-9156-4787-8f49-a173fa0bd5a3-cni-binary-copy" seLinuxMountContext="" Nov 04 04:47:16.225067 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.224795 4509 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="844cc12f-a7f6-4327-968e-5e9b2da00ad0" volumeName="kubernetes.io/configmap/844cc12f-a7f6-4327-968e-5e9b2da00ad0-ovnkube-config" seLinuxMountContext="" Nov 04 04:47:16.225067 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.224806 4509 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9a06ec07-c72b-43af-aa27-7414aa9c9aba" volumeName="kubernetes.io/configmap/9a06ec07-c72b-43af-aa27-7414aa9c9aba-serviceca" seLinuxMountContext="" Nov 04 04:47:16.225067 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.224817 4509 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cce8fa86-4f61-4dfc-b36c-2cefb7ecaa03" volumeName="kubernetes.io/empty-dir/cce8fa86-4f61-4dfc-b36c-2cefb7ecaa03-data-volume" seLinuxMountContext="" Nov 04 04:47:16.225067 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.224827 4509 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8446aa71-5638-4b23-8f5b-12283ede21d4" volumeName="kubernetes.io/empty-dir/8446aa71-5638-4b23-8f5b-12283ede21d4-tmp-dir" seLinuxMountContext="" Nov 04 04:47:16.225067 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.224837 4509 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d737ccc2-5686-4130-9d00-d2f21d5b417f" volumeName="kubernetes.io/projected/d737ccc2-5686-4130-9d00-d2f21d5b417f-kube-api-access-kz2hd" seLinuxMountContext="" Nov 04 04:47:16.225067 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.224859 4509 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dd0aaef0-d114-43b4-9ce4-135e818e4df3" volumeName="kubernetes.io/empty-dir/dd0aaef0-d114-43b4-9ce4-135e818e4df3-tmp-dir" seLinuxMountContext="" Nov 04 04:47:16.225067 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.224870 4509 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dd0aaef0-d114-43b4-9ce4-135e818e4df3" volumeName="kubernetes.io/projected/dd0aaef0-d114-43b4-9ce4-135e818e4df3-kube-api-access-7l4mv" seLinuxMountContext="" Nov 04 04:47:16.225067 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.224881 4509 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="21f0e203-fdf7-42cf-9c0c-4103e6601020" volumeName="kubernetes.io/projected/21f0e203-fdf7-42cf-9c0c-4103e6601020-kube-api-access-x2qjt" seLinuxMountContext="" Nov 04 04:47:16.225067 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.224892 4509 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="23ed4a39-18b5-4bc6-9c7b-8b60a7043ad2" volumeName="kubernetes.io/secret/23ed4a39-18b5-4bc6-9c7b-8b60a7043ad2-agent-certs" seLinuxMountContext="" Nov 04 04:47:16.225067 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.224902 4509 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="58be7092-9156-4787-8f49-a173fa0bd5a3" volumeName="kubernetes.io/configmap/58be7092-9156-4787-8f49-a173fa0bd5a3-multus-daemon-config" seLinuxMountContext="" Nov 04 04:47:16.225067 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.224913 4509 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8446aa71-5638-4b23-8f5b-12283ede21d4" volumeName="kubernetes.io/projected/8446aa71-5638-4b23-8f5b-12283ede21d4-kube-api-access-25g6k" seLinuxMountContext="" Nov 04 04:47:16.225067 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.224923 4509 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="844cc12f-a7f6-4327-968e-5e9b2da00ad0" volumeName="kubernetes.io/configmap/844cc12f-a7f6-4327-968e-5e9b2da00ad0-ovnkube-script-lib" seLinuxMountContext="" Nov 04 04:47:16.225067 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.224938 4509 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="844cc12f-a7f6-4327-968e-5e9b2da00ad0" volumeName="kubernetes.io/projected/844cc12f-a7f6-4327-968e-5e9b2da00ad0-kube-api-access-6nch9" seLinuxMountContext="" Nov 04 04:47:16.225067 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.224948 4509 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dd0aaef0-d114-43b4-9ce4-135e818e4df3" volumeName="kubernetes.io/configmap/dd0aaef0-d114-43b4-9ce4-135e818e4df3-config-volume" seLinuxMountContext="" Nov 04 04:47:16.225067 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.224958 4509 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dd0aaef0-d114-43b4-9ce4-135e818e4df3" volumeName="kubernetes.io/secret/dd0aaef0-d114-43b4-9ce4-135e818e4df3-metrics-tls" seLinuxMountContext="" Nov 04 04:47:16.225067 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.224969 4509 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="22d31d0c-6edf-4351-8f41-a6dd4e02368b" volumeName="kubernetes.io/empty-dir/22d31d0c-6edf-4351-8f41-a6dd4e02368b-tmp" seLinuxMountContext="" Nov 04 04:47:16.225067 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.224980 4509 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5f6922fa-9ba7-4d02-b550-606731852fde" volumeName="kubernetes.io/projected/5f6922fa-9ba7-4d02-b550-606731852fde-kube-api-access-47nf5" seLinuxMountContext="" Nov 04 04:47:16.225067 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.224991 4509 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9a06ec07-c72b-43af-aa27-7414aa9c9aba" volumeName="kubernetes.io/projected/9a06ec07-c72b-43af-aa27-7414aa9c9aba-kube-api-access-mlz69" seLinuxMountContext="" Nov 04 04:47:16.225067 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.225001 4509 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="00e0471e-5fe8-4494-8f17-632aee4f6c30" volumeName="kubernetes.io/projected/00e0471e-5fe8-4494-8f17-632aee4f6c30-kube-api-access-gftxp" seLinuxMountContext="" Nov 04 04:47:16.225067 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.225012 4509 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cce8fa86-4f61-4dfc-b36c-2cefb7ecaa03" volumeName="kubernetes.io/projected/cce8fa86-4f61-4dfc-b36c-2cefb7ecaa03-kube-api-access-7cgfm" seLinuxMountContext="" Nov 04 04:47:16.225067 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.225028 4509 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d737ccc2-5686-4130-9d00-d2f21d5b417f" volumeName="kubernetes.io/configmap/d737ccc2-5686-4130-9d00-d2f21d5b417f-iptables-alerter-script" seLinuxMountContext="" Nov 04 04:47:16.225067 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.225039 4509 reconstruct.go:97] "Volume reconstruction finished" Nov 04 04:47:16.225067 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.225046 4509 reconciler.go:29] "Reconciler: start to sync state" Nov 04 04:47:16.225947 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.225566 4509 factory.go:55] Registering systemd factory Nov 04 04:47:16.225947 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.225584 4509 factory.go:223] Registration of the systemd container factory successfully Nov 04 04:47:16.227618 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.227126 4509 factory.go:153] Registering CRI-O factory Nov 04 04:47:16.227618 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.227149 4509 factory.go:223] Registration of the crio container factory successfully Nov 04 04:47:16.227618 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.227250 4509 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 04 04:47:16.227618 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.227279 4509 factory.go:103] Registering Raw factory Nov 04 04:47:16.227618 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.227298 4509 manager.go:1199] Started watching for new ooms in manager Nov 04 04:47:16.227876 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.227682 4509 manager.go:319] Starting recovery of all containers Nov 04 04:47:16.252503 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.252467 4509 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 04 04:47:16.254146 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.254122 4509 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 04 04:47:16.254146 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.254152 4509 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 04 04:47:16.254360 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.254173 4509 kubelet.go:2521] "Starting kubelet main sync loop" Nov 04 04:47:16.254360 ip-10-0-133-180 kubenswrapper[4509]: E1104 04:47:16.254215 4509 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 04 04:47:16.264507 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.264458 4509 generic.go:356] "Generic (PLEG): container finished" podID="198e9ab0-f615-4686-9acf-a6242f7e3584" containerID="54f3cbda9d6ee2f42e6a7cd241272b78ca6c284da961c54d1674c33833a011aa" exitCode=0 Nov 04 04:47:16.264507 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.264505 4509 generic.go:356] "Generic (PLEG): container finished" podID="198e9ab0-f615-4686-9acf-a6242f7e3584" containerID="84838fa07477f4eb2ad0a3809e89ad1fceee70bee3f05d41d9e53ca23d6438e5" exitCode=0 Nov 04 04:47:16.264700 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.264517 4509 generic.go:356] "Generic (PLEG): container finished" podID="198e9ab0-f615-4686-9acf-a6242f7e3584" containerID="32038a768b343a887e01933064b0baa519cc33e054200b177d1088225b8efd75" exitCode=0 Nov 04 04:47:16.264700 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.264524 4509 generic.go:356] "Generic (PLEG): container finished" podID="198e9ab0-f615-4686-9acf-a6242f7e3584" containerID="323fc8e1cff69036edddfb491d7c584c0c28de977220a248a1c7aad5548e2ac7" exitCode=0 Nov 04 04:47:16.264700 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.264531 4509 generic.go:356] "Generic (PLEG): container finished" podID="198e9ab0-f615-4686-9acf-a6242f7e3584" containerID="590d8d4be63d5ccffc60b4f1f9153c9b75ffb08cea1c4afacabd879c04ba5bfa" exitCode=0 Nov 04 04:47:16.264700 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.264536 4509 generic.go:356] "Generic (PLEG): container finished" podID="198e9ab0-f615-4686-9acf-a6242f7e3584" containerID="68a3bfbe4c7af69a9d5b7f06c7eb1b4d323f201b91a0b6fc2d4b24cf335028cb" exitCode=0 Nov 04 04:47:16.270584 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.270553 4509 generic.go:356] "Generic (PLEG): container finished" podID="46addbc2e24e99ce85548c61cc2707a9" containerID="17a36ef60eab503b266d126674f81064202a9c2fe4867d3aba215a0eaced37eb" exitCode=0 Nov 04 04:47:16.296212 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.296186 4509 manager.go:324] Recovery completed Nov 04 04:47:16.309126 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.309104 4509 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 04:47:16.310460 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.310416 4509 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-180.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 04 04:47:16.310460 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.310449 4509 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-180.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 04 04:47:16.310460 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.310460 4509 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-180.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 04 04:47:16.312317 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.312299 4509 cpu_manager.go:226] "Starting" policy="none" Nov 04 04:47:16.312396 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.312317 4509 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 04 04:47:16.312396 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.312375 4509 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 04 04:47:16.312461 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.312305 4509 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 04:47:16.312552 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.312534 4509 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Nov 04 04:47:16.312581 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.312555 4509 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Nov 04 04:47:16.312581 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.312580 4509 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Nov 04 04:47:16.312650 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.312604 4509 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Nov 04 04:47:16.312650 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.312613 4509 policy_none.go:50] "Start" Nov 04 04:47:16.312650 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.312622 4509 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 04 04:47:16.312650 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.312633 4509 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 04 04:47:16.312816 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.312805 4509 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Nov 04 04:47:16.312849 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.312817 4509 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Nov 04 04:47:16.312849 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.312823 4509 policy_none.go:44] "Start" Nov 04 04:47:16.313415 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.313389 4509 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-180.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 04 04:47:16.313498 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.313420 4509 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-180.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 04 04:47:16.313498 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.313430 4509 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-180.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 04 04:47:16.317169 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.317140 4509 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 04 04:47:16.319694 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.319682 4509 manager.go:341] "Starting Device Plugin manager" Nov 04 04:47:16.319734 ip-10-0-133-180 kubenswrapper[4509]: E1104 04:47:16.319708 4509 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 04 04:47:16.319734 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.319717 4509 server.go:87] "Starting device plugin registration server" Nov 04 04:47:16.320055 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.320042 4509 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 04 04:47:16.320084 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.320059 4509 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 04 04:47:16.320172 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.320156 4509 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 04 04:47:16.320714 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.320671 4509 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 04 04:47:16.320817 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.320721 4509 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 04 04:47:16.320817 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.320728 4509 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 04 04:47:16.320954 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.320843 4509 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-04T04:47:16.320709067Z","UUID":"ef1b33be-de5f-4cd0-9fe2-5a3789d20fa8","Handler":null,"Name":"","Endpoint":""} Nov 04 04:47:16.322571 ip-10-0-133-180 kubenswrapper[4509]: E1104 04:47:16.322539 4509 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 04 04:47:16.322654 ip-10-0-133-180 kubenswrapper[4509]: E1104 04:47:16.322608 4509 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-180.us-west-1.compute.internal\" not found" Nov 04 04:47:16.324599 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.324577 4509 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 04 04:47:16.324599 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.324610 4509 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 04 04:47:16.354911 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.354377 4509 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-180.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-180.us-west-1.compute.internal"] Nov 04 04:47:16.355116 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.355033 4509 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 04:47:16.356070 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.356055 4509 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-180.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 04 04:47:16.356152 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.356085 4509 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-180.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 04 04:47:16.356152 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.356096 4509 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-180.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 04 04:47:16.356378 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.356366 4509 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 04:47:16.356493 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.356478 4509 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 04:47:16.357183 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.357157 4509 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-180.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 04 04:47:16.357183 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.357183 4509 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-180.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 04 04:47:16.357183 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.357192 4509 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-180.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 04 04:47:16.357343 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.357155 4509 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-180.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 04 04:47:16.357343 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.357262 4509 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-180.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 04 04:47:16.357343 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.357275 4509 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-180.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 04 04:47:16.357479 ip-10-0-133-180 kubenswrapper[4509]: E1104 04:47:16.357426 4509 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-180.us-west-1.compute.internal\" not found" node="ip-10-0-133-180.us-west-1.compute.internal" Nov 04 04:47:16.357479 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.357425 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-180.us-west-1.compute.internal" event={"ID":"46addbc2e24e99ce85548c61cc2707a9","Type":"ContainerStarted","Data":"8d3db7d02613eb50a6b709814f2b10edb68088541dbcf0f26579d907c4033b2f"} Nov 04 04:47:16.357541 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.357480 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-180.us-west-1.compute.internal" event={"ID":"46addbc2e24e99ce85548c61cc2707a9","Type":"ContainerDied","Data":"17a36ef60eab503b266d126674f81064202a9c2fe4867d3aba215a0eaced37eb"} Nov 04 04:47:16.357541 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.357495 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-180.us-west-1.compute.internal" event={"ID":"46addbc2e24e99ce85548c61cc2707a9","Type":"ContainerStarted","Data":"9a9d05d8927347bce1aaee6783a92cb2e0d8550b2ab0cabbc0015124f74b024e"} Nov 04 04:47:16.357541 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.357525 4509 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 04:47:16.357684 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.357545 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-180.us-west-1.compute.internal" event={"ID":"6aa9201e961e4ed5a49af4c50453ef92","Type":"ContainerStarted","Data":"12ad69f5d7bf82e2961b5d2ff6f0d403750c49e510a905db052fee4e2272789a"} Nov 04 04:47:16.357684 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.357557 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-180.us-west-1.compute.internal" event={"ID":"6aa9201e961e4ed5a49af4c50453ef92","Type":"ContainerStarted","Data":"61c7ccd8c37f4b1903256dd5a442b7df3c9fe02b842028cc54b884875b60942a"} Nov 04 04:47:16.358183 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.358169 4509 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-180.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 04 04:47:16.358253 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.358198 4509 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-180.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 04 04:47:16.358253 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.358213 4509 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-180.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 04 04:47:16.358406 ip-10-0-133-180 kubenswrapper[4509]: E1104 04:47:16.358393 4509 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-180.us-west-1.compute.internal\" not found" node="ip-10-0-133-180.us-west-1.compute.internal" Nov 04 04:47:16.420767 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.420733 4509 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 04:47:16.421875 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.421859 4509 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-180.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 04 04:47:16.421986 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.421891 4509 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-180.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 04 04:47:16.421986 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.421900 4509 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-180.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 04 04:47:16.421986 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.421918 4509 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-133-180.us-west-1.compute.internal" Nov 04 04:47:16.481546 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.481508 4509 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 04 04:47:16.481734 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.481635 4509 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 04 04:47:16.481734 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.481659 4509 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 04 04:47:16.515288 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.515248 4509 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 04 04:47:16.515288 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.515281 4509 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Nov 04 04:47:16.523542 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.523511 4509 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-180.us-west-1.compute.internal" Nov 04 04:47:16.567343 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.567234 4509 nodeinfomanager.go:471] existing CSINode "ip-10-0-133-180.us-west-1.compute.internal" is owned by different node (oldNodeID="b478449c-cace-443b-b695-c00bbdf8f920", newNodeID=""), cleaning up... Nov 04 04:47:16.567343 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.567304 4509 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-133-180.us-west-1.compute.internal" Nov 04 04:47:16.567534 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.567402 4509 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-133-180.us-west-1.compute.internal" Nov 04 04:47:16.567988 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.567973 4509 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-180.us-west-1.compute.internal" event="NodeNotReady" Nov 04 04:47:16.568051 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.568002 4509 setters.go:546] "Node became not ready" node="ip-10-0-133-180.us-west-1.compute.internal" condition={"type":"Ready","status":"False","lastHeartbeatTime":"2026-11-04T04:47:16Z","lastTransitionTime":"2026-11-04T04:47:16Z","reason":"KubeletNotReady","message":"CSINode is not yet initialized"} Nov 04 04:47:16.581801 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.581779 4509 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 04 04:47:16.581869 ip-10-0-133-180 kubenswrapper[4509]: E1104 04:47:16.581829 4509 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-133-180.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-180.us-west-1.compute.internal" Nov 04 04:47:16.581869 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.581846 4509 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-180.us-west-1.compute.internal" Nov 04 04:47:16.615447 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.615415 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/46addbc2e24e99ce85548c61cc2707a9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-180.us-west-1.compute.internal\" (UID: \"46addbc2e24e99ce85548c61cc2707a9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-180.us-west-1.compute.internal" Nov 04 04:47:16.615551 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.615454 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/46addbc2e24e99ce85548c61cc2707a9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-180.us-west-1.compute.internal\" (UID: \"46addbc2e24e99ce85548c61cc2707a9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-180.us-west-1.compute.internal" Nov 04 04:47:16.615551 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.615481 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6aa9201e961e4ed5a49af4c50453ef92-config\") pod \"kube-apiserver-proxy-ip-10-0-133-180.us-west-1.compute.internal\" (UID: \"6aa9201e961e4ed5a49af4c50453ef92\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-180.us-west-1.compute.internal" Nov 04 04:47:16.639934 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.639902 4509 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 04 04:47:16.640074 ip-10-0-133-180 kubenswrapper[4509]: E1104 04:47:16.639958 4509 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-133-180.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-180.us-west-1.compute.internal" Nov 04 04:47:16.716259 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.716228 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/46addbc2e24e99ce85548c61cc2707a9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-180.us-west-1.compute.internal\" (UID: \"46addbc2e24e99ce85548c61cc2707a9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-180.us-west-1.compute.internal" Nov 04 04:47:16.716411 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.716176 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/46addbc2e24e99ce85548c61cc2707a9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-180.us-west-1.compute.internal\" (UID: \"46addbc2e24e99ce85548c61cc2707a9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-180.us-west-1.compute.internal" Nov 04 04:47:16.716411 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.716297 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/46addbc2e24e99ce85548c61cc2707a9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-180.us-west-1.compute.internal\" (UID: \"46addbc2e24e99ce85548c61cc2707a9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-180.us-west-1.compute.internal" Nov 04 04:47:16.716411 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.716318 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6aa9201e961e4ed5a49af4c50453ef92-config\") pod \"kube-apiserver-proxy-ip-10-0-133-180.us-west-1.compute.internal\" (UID: \"6aa9201e961e4ed5a49af4c50453ef92\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-180.us-west-1.compute.internal" Nov 04 04:47:16.716520 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.716424 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6aa9201e961e4ed5a49af4c50453ef92-config\") pod \"kube-apiserver-proxy-ip-10-0-133-180.us-west-1.compute.internal\" (UID: \"6aa9201e961e4ed5a49af4c50453ef92\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-180.us-west-1.compute.internal" Nov 04 04:47:16.716520 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.716433 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/46addbc2e24e99ce85548c61cc2707a9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-180.us-west-1.compute.internal\" (UID: \"46addbc2e24e99ce85548c61cc2707a9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-180.us-west-1.compute.internal" Nov 04 04:47:16.834245 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.834174 4509 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-180.us-west-1.compute.internal" event="NodeReady" Nov 04 04:47:16.834388 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:16.834251 4509 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 04 04:47:17.209286 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.209261 4509 apiserver.go:52] "Watching apiserver" Nov 04 04:47:17.296099 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.296073 4509 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-180.us-west-1.compute.internal" Nov 04 04:47:17.296281 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.296152 4509 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-180.us-west-1.compute.internal" Nov 04 04:47:17.353557 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.353521 4509 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 04 04:47:17.353948 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.353929 4509 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-7m8fc","openshift-dns/node-resolver-gzctm","openshift-dns/dns-default-4xxrr","openshift-network-diagnostics/network-check-target-8hsp8","openshift-network-operator/iptables-alerter-wdk7z","openshift-image-registry/node-ca-4z89d","openshift-cluster-node-tuning-operator/tuned-c65v8","openshift-ingress-canary/ingress-canary-mt4dj","kube-system/konnectivity-agent-7tw97","openshift-multus/network-metrics-daemon-25flt","kube-system/kube-apiserver-proxy-ip-10-0-133-180.us-west-1.compute.internal","kube-system/global-pull-secret-syncer-gfjqz","openshift-insights/insights-runtime-extractor-dmdhl","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-180.us-west-1.compute.internal","openshift-multus/multus-additional-cni-plugins-cplsq","openshift-ovn-kubernetes/ovnkube-node-dvc6z","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f2zrp"] Nov 04 04:47:17.358119 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.358098 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-dmdhl" Nov 04 04:47:17.359870 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.359853 4509 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 04 04:47:17.359937 ip-10-0-133-180 kubenswrapper[4509]: E1104 04:47:17.359893 4509 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-133-180.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-180.us-west-1.compute.internal" Nov 04 04:47:17.360033 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.360019 4509 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 04 04:47:17.360069 ip-10-0-133-180 kubenswrapper[4509]: E1104 04:47:17.360057 4509 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-133-180.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-180.us-west-1.compute.internal" Nov 04 04:47:17.406807 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.406782 4509 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-jqdfn\"" Nov 04 04:47:17.406807 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.406785 4509 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-mrmbg\"" Nov 04 04:47:17.406949 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.406786 4509 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 04 04:47:17.408237 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.408217 4509 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 04 04:47:17.408498 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.408482 4509 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-kgq44\"" Nov 04 04:47:17.408632 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.408605 4509 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 04 04:47:17.408632 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.408620 4509 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-w4xfx\"" Nov 04 04:47:17.408799 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.408704 4509 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 04 04:47:17.408799 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.408740 4509 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-shz8v\"" Nov 04 04:47:17.408799 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.408741 4509 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-zsr9v\"" Nov 04 04:47:17.408799 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.408764 4509 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 04 04:47:17.408937 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.408845 4509 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 04 04:47:17.408996 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.408983 4509 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-tqbjc\"" Nov 04 04:47:17.409036 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.409028 4509 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 04 04:47:17.409077 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.409046 4509 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-mkkxt\"" Nov 04 04:47:17.409163 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.409111 4509 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 04 04:47:17.409163 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.409134 4509 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-g79lj\"" Nov 04 04:47:17.409241 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.409193 4509 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 04 04:47:17.409313 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.409298 4509 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 04 04:47:17.409313 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.409300 4509 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 04 04:47:17.409419 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.409316 4509 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 04 04:47:17.409419 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.409335 4509 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 04 04:47:17.409419 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.409412 4509 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 04 04:47:17.409538 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.409417 4509 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 04 04:47:17.409538 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.409431 4509 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 04 04:47:17.409538 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.409463 4509 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 04 04:47:17.409538 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.409493 4509 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 04 04:47:17.409538 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.409528 4509 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 04 04:47:17.409677 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.409564 4509 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 04 04:47:17.409677 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.409590 4509 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 04 04:47:17.409677 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.409606 4509 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 04 04:47:17.409677 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.409611 4509 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 04 04:47:17.409677 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.409625 4509 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 04 04:47:17.409677 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.409620 4509 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 04 04:47:17.409677 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.409676 4509 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 04 04:47:17.411549 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.411532 4509 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-n7w7t\"" Nov 04 04:47:17.411637 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.411560 4509 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 04 04:47:17.411688 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.411651 4509 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 04 04:47:17.411734 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.411713 4509 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-vmp7s\"" Nov 04 04:47:17.411734 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.411727 4509 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 04 04:47:17.411821 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.411742 4509 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-ssxrh\"" Nov 04 04:47:17.411821 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.411743 4509 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 04 04:47:17.411821 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.411759 4509 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 04 04:47:17.411821 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.411787 4509 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 04 04:47:17.411821 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.411795 4509 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 04 04:47:17.411821 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.411802 4509 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 04 04:47:17.411992 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.411830 4509 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 04 04:47:17.411992 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.411857 4509 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-jvrn2\"" Nov 04 04:47:17.423993 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.423975 4509 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 04 04:47:17.455053 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.455031 4509 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 04 04:47:17.455175 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.455074 4509 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 04 04:47:17.455175 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.455086 4509 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 04 04:47:17.455175 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.455094 4509 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 04 04:47:17.455175 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.455074 4509 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 04 04:47:17.455175 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.455162 4509 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-q5r74\"" Nov 04 04:47:17.455412 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.455196 4509 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-6sxpx\"" Nov 04 04:47:17.522289 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.522204 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/00e0471e-5fe8-4494-8f17-632aee4f6c30-sys-fs\") pod \"aws-ebs-csi-driver-node-f2zrp\" (UID: \"00e0471e-5fe8-4494-8f17-632aee4f6c30\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f2zrp" Nov 04 04:47:17.522289 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.522237 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/844cc12f-a7f6-4327-968e-5e9b2da00ad0-etc-openvswitch\") pod \"ovnkube-node-dvc6z\" (UID: \"844cc12f-a7f6-4327-968e-5e9b2da00ad0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvc6z" Nov 04 04:47:17.522289 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.522256 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/844cc12f-a7f6-4327-968e-5e9b2da00ad0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dvc6z\" (UID: \"844cc12f-a7f6-4327-968e-5e9b2da00ad0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvc6z" Nov 04 04:47:17.522289 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.522281 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nch9\" (UniqueName: \"kubernetes.io/projected/844cc12f-a7f6-4327-968e-5e9b2da00ad0-kube-api-access-6nch9\") pod \"ovnkube-node-dvc6z\" (UID: \"844cc12f-a7f6-4327-968e-5e9b2da00ad0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvc6z" Nov 04 04:47:17.522572 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.522355 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/58be7092-9156-4787-8f49-a173fa0bd5a3-cnibin\") pod \"multus-7m8fc\" (UID: \"58be7092-9156-4787-8f49-a173fa0bd5a3\") " pod="openshift-multus/multus-7m8fc" Nov 04 04:47:17.522572 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.522400 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/22d31d0c-6edf-4351-8f41-a6dd4e02368b-etc-systemd\") pod \"tuned-c65v8\" (UID: \"22d31d0c-6edf-4351-8f41-a6dd4e02368b\") " pod="openshift-cluster-node-tuning-operator/tuned-c65v8" Nov 04 04:47:17.522572 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.522429 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/22d31d0c-6edf-4351-8f41-a6dd4e02368b-run\") pod \"tuned-c65v8\" (UID: \"22d31d0c-6edf-4351-8f41-a6dd4e02368b\") " pod="openshift-cluster-node-tuning-operator/tuned-c65v8" Nov 04 04:47:17.522572 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.522443 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/22d31d0c-6edf-4351-8f41-a6dd4e02368b-lib-modules\") pod \"tuned-c65v8\" (UID: \"22d31d0c-6edf-4351-8f41-a6dd4e02368b\") " pod="openshift-cluster-node-tuning-operator/tuned-c65v8" Nov 04 04:47:17.522572 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.522476 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/22d31d0c-6edf-4351-8f41-a6dd4e02368b-tmp\") pod \"tuned-c65v8\" (UID: \"22d31d0c-6edf-4351-8f41-a6dd4e02368b\") " pod="openshift-cluster-node-tuning-operator/tuned-c65v8" Nov 04 04:47:17.522572 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.522512 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/844cc12f-a7f6-4327-968e-5e9b2da00ad0-run-systemd\") pod \"ovnkube-node-dvc6z\" (UID: \"844cc12f-a7f6-4327-968e-5e9b2da00ad0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvc6z" Nov 04 04:47:17.522572 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.522558 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/844cc12f-a7f6-4327-968e-5e9b2da00ad0-env-overrides\") pod \"ovnkube-node-dvc6z\" (UID: \"844cc12f-a7f6-4327-968e-5e9b2da00ad0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvc6z" Nov 04 04:47:17.522572 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.522560 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/22d31d0c-6edf-4351-8f41-a6dd4e02368b-tmp\") pod \"tuned-c65v8\" (UID: \"22d31d0c-6edf-4351-8f41-a6dd4e02368b\") " pod="openshift-cluster-node-tuning-operator/tuned-c65v8" Nov 04 04:47:17.522925 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.522595 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cce8fa86-4f61-4dfc-b36c-2cefb7ecaa03-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dmdhl\" (UID: \"cce8fa86-4f61-4dfc-b36c-2cefb7ecaa03\") " pod="openshift-insights/insights-runtime-extractor-dmdhl" Nov 04 04:47:17.522925 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.522628 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cgfm\" (UniqueName: \"kubernetes.io/projected/cce8fa86-4f61-4dfc-b36c-2cefb7ecaa03-kube-api-access-7cgfm\") pod \"insights-runtime-extractor-dmdhl\" (UID: \"cce8fa86-4f61-4dfc-b36c-2cefb7ecaa03\") " pod="openshift-insights/insights-runtime-extractor-dmdhl" Nov 04 04:47:17.522925 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.522654 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/00e0471e-5fe8-4494-8f17-632aee4f6c30-kubelet-dir\") pod \"aws-ebs-csi-driver-node-f2zrp\" (UID: \"00e0471e-5fe8-4494-8f17-632aee4f6c30\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f2zrp" Nov 04 04:47:17.522925 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.522681 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccfxs\" (UniqueName: \"kubernetes.io/projected/198e9ab0-f615-4686-9acf-a6242f7e3584-kube-api-access-ccfxs\") pod \"multus-additional-cni-plugins-cplsq\" (UID: \"198e9ab0-f615-4686-9acf-a6242f7e3584\") " pod="openshift-multus/multus-additional-cni-plugins-cplsq" Nov 04 04:47:17.522925 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.522770 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/844cc12f-a7f6-4327-968e-5e9b2da00ad0-env-overrides\") pod \"ovnkube-node-dvc6z\" (UID: \"844cc12f-a7f6-4327-968e-5e9b2da00ad0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvc6z" Nov 04 04:47:17.522925 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.522820 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9a06ec07-c72b-43af-aa27-7414aa9c9aba-host\") pod \"node-ca-4z89d\" (UID: \"9a06ec07-c72b-43af-aa27-7414aa9c9aba\") " pod="openshift-image-registry/node-ca-4z89d" Nov 04 04:47:17.522925 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.522852 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/58be7092-9156-4787-8f49-a173fa0bd5a3-cni-binary-copy\") pod \"multus-7m8fc\" (UID: \"58be7092-9156-4787-8f49-a173fa0bd5a3\") " pod="openshift-multus/multus-7m8fc" Nov 04 04:47:17.522925 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.522850 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cce8fa86-4f61-4dfc-b36c-2cefb7ecaa03-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dmdhl\" (UID: \"cce8fa86-4f61-4dfc-b36c-2cefb7ecaa03\") " pod="openshift-insights/insights-runtime-extractor-dmdhl" Nov 04 04:47:17.522925 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.522869 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4xvx\" (UniqueName: \"kubernetes.io/projected/58be7092-9156-4787-8f49-a173fa0bd5a3-kube-api-access-w4xvx\") pod \"multus-7m8fc\" (UID: \"58be7092-9156-4787-8f49-a173fa0bd5a3\") " pod="openshift-multus/multus-7m8fc" Nov 04 04:47:17.522925 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.522902 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/22d31d0c-6edf-4351-8f41-a6dd4e02368b-etc-kubernetes\") pod \"tuned-c65v8\" (UID: \"22d31d0c-6edf-4351-8f41-a6dd4e02368b\") " pod="openshift-cluster-node-tuning-operator/tuned-c65v8" Nov 04 04:47:17.522925 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.522926 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/844cc12f-a7f6-4327-968e-5e9b2da00ad0-var-lib-openvswitch\") pod \"ovnkube-node-dvc6z\" (UID: \"844cc12f-a7f6-4327-968e-5e9b2da00ad0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvc6z" Nov 04 04:47:17.523359 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.522979 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/21f0e203-fdf7-42cf-9c0c-4103e6601020-cert\") pod \"ingress-canary-mt4dj\" (UID: \"21f0e203-fdf7-42cf-9c0c-4103e6601020\") " pod="openshift-ingress-canary/ingress-canary-mt4dj" Nov 04 04:47:17.523359 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.523015 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/58be7092-9156-4787-8f49-a173fa0bd5a3-system-cni-dir\") pod \"multus-7m8fc\" (UID: \"58be7092-9156-4787-8f49-a173fa0bd5a3\") " pod="openshift-multus/multus-7m8fc" Nov 04 04:47:17.523359 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.523046 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/58be7092-9156-4787-8f49-a173fa0bd5a3-host-var-lib-kubelet\") pod \"multus-7m8fc\" (UID: \"58be7092-9156-4787-8f49-a173fa0bd5a3\") " pod="openshift-multus/multus-7m8fc" Nov 04 04:47:17.523359 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.523086 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/198e9ab0-f615-4686-9acf-a6242f7e3584-cni-binary-copy\") pod \"multus-additional-cni-plugins-cplsq\" (UID: \"198e9ab0-f615-4686-9acf-a6242f7e3584\") " pod="openshift-multus/multus-additional-cni-plugins-cplsq" Nov 04 04:47:17.523359 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.523123 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gftxp\" (UniqueName: \"kubernetes.io/projected/00e0471e-5fe8-4494-8f17-632aee4f6c30-kube-api-access-gftxp\") pod \"aws-ebs-csi-driver-node-f2zrp\" (UID: \"00e0471e-5fe8-4494-8f17-632aee4f6c30\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f2zrp" Nov 04 04:47:17.523359 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.523131 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/58be7092-9156-4787-8f49-a173fa0bd5a3-cni-binary-copy\") pod \"multus-7m8fc\" (UID: \"58be7092-9156-4787-8f49-a173fa0bd5a3\") " pod="openshift-multus/multus-7m8fc" Nov 04 04:47:17.523359 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.523143 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47nf5\" (UniqueName: \"kubernetes.io/projected/5f6922fa-9ba7-4d02-b550-606731852fde-kube-api-access-47nf5\") pod \"network-check-target-8hsp8\" (UID: \"5f6922fa-9ba7-4d02-b550-606731852fde\") " pod="openshift-network-diagnostics/network-check-target-8hsp8" Nov 04 04:47:17.523359 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.523169 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/00e0471e-5fe8-4494-8f17-632aee4f6c30-etc-selinux\") pod \"aws-ebs-csi-driver-node-f2zrp\" (UID: \"00e0471e-5fe8-4494-8f17-632aee4f6c30\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f2zrp" Nov 04 04:47:17.523359 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.523193 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/21f0e203-fdf7-42cf-9c0c-4103e6601020-cert\") pod \"ingress-canary-mt4dj\" (UID: \"21f0e203-fdf7-42cf-9c0c-4103e6601020\") " pod="openshift-ingress-canary/ingress-canary-mt4dj" Nov 04 04:47:17.523359 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.523239 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/22d31d0c-6edf-4351-8f41-a6dd4e02368b-etc-sysctl-conf\") pod \"tuned-c65v8\" (UID: \"22d31d0c-6edf-4351-8f41-a6dd4e02368b\") " pod="openshift-cluster-node-tuning-operator/tuned-c65v8" Nov 04 04:47:17.523359 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.523265 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/844cc12f-a7f6-4327-968e-5e9b2da00ad0-host-run-netns\") pod \"ovnkube-node-dvc6z\" (UID: \"844cc12f-a7f6-4327-968e-5e9b2da00ad0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvc6z" Nov 04 04:47:17.523359 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.523299 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/cce8fa86-4f61-4dfc-b36c-2cefb7ecaa03-data-volume\") pod \"insights-runtime-extractor-dmdhl\" (UID: \"cce8fa86-4f61-4dfc-b36c-2cefb7ecaa03\") " pod="openshift-insights/insights-runtime-extractor-dmdhl" Nov 04 04:47:17.523359 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.523352 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9a06ec07-c72b-43af-aa27-7414aa9c9aba-serviceca\") pod \"node-ca-4z89d\" (UID: \"9a06ec07-c72b-43af-aa27-7414aa9c9aba\") " pod="openshift-image-registry/node-ca-4z89d" Nov 04 04:47:17.523906 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.523388 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/cce8fa86-4f61-4dfc-b36c-2cefb7ecaa03-data-volume\") pod \"insights-runtime-extractor-dmdhl\" (UID: \"cce8fa86-4f61-4dfc-b36c-2cefb7ecaa03\") " pod="openshift-insights/insights-runtime-extractor-dmdhl" Nov 04 04:47:17.523906 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.523421 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/844cc12f-a7f6-4327-968e-5e9b2da00ad0-log-socket\") pod \"ovnkube-node-dvc6z\" (UID: \"844cc12f-a7f6-4327-968e-5e9b2da00ad0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvc6z" Nov 04 04:47:17.523906 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.523448 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/58be7092-9156-4787-8f49-a173fa0bd5a3-multus-cni-dir\") pod \"multus-7m8fc\" (UID: \"58be7092-9156-4787-8f49-a173fa0bd5a3\") " pod="openshift-multus/multus-7m8fc" Nov 04 04:47:17.523906 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.523468 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/58be7092-9156-4787-8f49-a173fa0bd5a3-host-var-lib-cni-bin\") pod \"multus-7m8fc\" (UID: \"58be7092-9156-4787-8f49-a173fa0bd5a3\") " pod="openshift-multus/multus-7m8fc" Nov 04 04:47:17.523906 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.523471 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/198e9ab0-f615-4686-9acf-a6242f7e3584-cni-binary-copy\") pod \"multus-additional-cni-plugins-cplsq\" (UID: \"198e9ab0-f615-4686-9acf-a6242f7e3584\") " pod="openshift-multus/multus-additional-cni-plugins-cplsq" Nov 04 04:47:17.523906 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.523505 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/58be7092-9156-4787-8f49-a173fa0bd5a3-hostroot\") pod \"multus-7m8fc\" (UID: \"58be7092-9156-4787-8f49-a173fa0bd5a3\") " pod="openshift-multus/multus-7m8fc" Nov 04 04:47:17.523906 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.523534 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/22d31d0c-6edf-4351-8f41-a6dd4e02368b-etc-tuned\") pod \"tuned-c65v8\" (UID: \"22d31d0c-6edf-4351-8f41-a6dd4e02368b\") " pod="openshift-cluster-node-tuning-operator/tuned-c65v8" Nov 04 04:47:17.523906 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.523561 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/198e9ab0-f615-4686-9acf-a6242f7e3584-system-cni-dir\") pod \"multus-additional-cni-plugins-cplsq\" (UID: \"198e9ab0-f615-4686-9acf-a6242f7e3584\") " pod="openshift-multus/multus-additional-cni-plugins-cplsq" Nov 04 04:47:17.523906 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.523587 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/cce8fa86-4f61-4dfc-b36c-2cefb7ecaa03-crio-socket\") pod \"insights-runtime-extractor-dmdhl\" (UID: \"cce8fa86-4f61-4dfc-b36c-2cefb7ecaa03\") " pod="openshift-insights/insights-runtime-extractor-dmdhl" Nov 04 04:47:17.523906 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.523603 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/22d31d0c-6edf-4351-8f41-a6dd4e02368b-etc-tuned\") pod \"tuned-c65v8\" (UID: \"22d31d0c-6edf-4351-8f41-a6dd4e02368b\") " pod="openshift-cluster-node-tuning-operator/tuned-c65v8" Nov 04 04:47:17.523906 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.523613 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d737ccc2-5686-4130-9d00-d2f21d5b417f-iptables-alerter-script\") pod \"iptables-alerter-wdk7z\" (UID: \"d737ccc2-5686-4130-9d00-d2f21d5b417f\") " pod="openshift-network-operator/iptables-alerter-wdk7z" Nov 04 04:47:17.523906 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.523650 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kz2hd\" (UniqueName: \"kubernetes.io/projected/d737ccc2-5686-4130-9d00-d2f21d5b417f-kube-api-access-kz2hd\") pod \"iptables-alerter-wdk7z\" (UID: \"d737ccc2-5686-4130-9d00-d2f21d5b417f\") " pod="openshift-network-operator/iptables-alerter-wdk7z" Nov 04 04:47:17.523906 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.523717 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/22d31d0c-6edf-4351-8f41-a6dd4e02368b-etc-sysconfig\") pod \"tuned-c65v8\" (UID: \"22d31d0c-6edf-4351-8f41-a6dd4e02368b\") " pod="openshift-cluster-node-tuning-operator/tuned-c65v8" Nov 04 04:47:17.523906 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.523737 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/844cc12f-a7f6-4327-968e-5e9b2da00ad0-node-log\") pod \"ovnkube-node-dvc6z\" (UID: \"844cc12f-a7f6-4327-968e-5e9b2da00ad0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvc6z" Nov 04 04:47:17.523906 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.523756 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/844cc12f-a7f6-4327-968e-5e9b2da00ad0-host-run-ovn-kubernetes\") pod \"ovnkube-node-dvc6z\" (UID: \"844cc12f-a7f6-4327-968e-5e9b2da00ad0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvc6z" Nov 04 04:47:17.523906 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.523777 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/198e9ab0-f615-4686-9acf-a6242f7e3584-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cplsq\" (UID: \"198e9ab0-f615-4686-9acf-a6242f7e3584\") " pod="openshift-multus/multus-additional-cni-plugins-cplsq" Nov 04 04:47:17.523906 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.523815 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/58be7092-9156-4787-8f49-a173fa0bd5a3-host-var-lib-cni-multus\") pod \"multus-7m8fc\" (UID: \"58be7092-9156-4787-8f49-a173fa0bd5a3\") " pod="openshift-multus/multus-7m8fc" Nov 04 04:47:17.523906 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.523828 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d737ccc2-5686-4130-9d00-d2f21d5b417f-iptables-alerter-script\") pod \"iptables-alerter-wdk7z\" (UID: \"d737ccc2-5686-4130-9d00-d2f21d5b417f\") " pod="openshift-network-operator/iptables-alerter-wdk7z" Nov 04 04:47:17.523906 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.523858 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/00e0471e-5fe8-4494-8f17-632aee4f6c30-device-dir\") pod \"aws-ebs-csi-driver-node-f2zrp\" (UID: \"00e0471e-5fe8-4494-8f17-632aee4f6c30\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f2zrp" Nov 04 04:47:17.523906 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.523896 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c79eff94-6543-4313-b59b-502ca4089e84-dbus\") pod \"global-pull-secret-syncer-gfjqz\" (UID: \"c79eff94-6543-4313-b59b-502ca4089e84\") " pod="kube-system/global-pull-secret-syncer-gfjqz" Nov 04 04:47:17.523906 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.523907 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9a06ec07-c72b-43af-aa27-7414aa9c9aba-serviceca\") pod \"node-ca-4z89d\" (UID: \"9a06ec07-c72b-43af-aa27-7414aa9c9aba\") " pod="openshift-image-registry/node-ca-4z89d" Nov 04 04:47:17.524910 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.523949 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/22d31d0c-6edf-4351-8f41-a6dd4e02368b-etc-modprobe-d\") pod \"tuned-c65v8\" (UID: \"22d31d0c-6edf-4351-8f41-a6dd4e02368b\") " pod="openshift-cluster-node-tuning-operator/tuned-c65v8" Nov 04 04:47:17.524910 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.523960 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/198e9ab0-f615-4686-9acf-a6242f7e3584-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cplsq\" (UID: \"198e9ab0-f615-4686-9acf-a6242f7e3584\") " pod="openshift-multus/multus-additional-cni-plugins-cplsq" Nov 04 04:47:17.524910 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.523987 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbtxf\" (UniqueName: \"kubernetes.io/projected/22d31d0c-6edf-4351-8f41-a6dd4e02368b-kube-api-access-zbtxf\") pod \"tuned-c65v8\" (UID: \"22d31d0c-6edf-4351-8f41-a6dd4e02368b\") " pod="openshift-cluster-node-tuning-operator/tuned-c65v8" Nov 04 04:47:17.524910 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.524017 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/844cc12f-a7f6-4327-968e-5e9b2da00ad0-host-kubelet\") pod \"ovnkube-node-dvc6z\" (UID: \"844cc12f-a7f6-4327-968e-5e9b2da00ad0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvc6z" Nov 04 04:47:17.524910 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.524128 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/844cc12f-a7f6-4327-968e-5e9b2da00ad0-ovn-node-metrics-cert\") pod \"ovnkube-node-dvc6z\" (UID: \"844cc12f-a7f6-4327-968e-5e9b2da00ad0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvc6z" Nov 04 04:47:17.524910 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.524169 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/198e9ab0-f615-4686-9acf-a6242f7e3584-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cplsq\" (UID: \"198e9ab0-f615-4686-9acf-a6242f7e3584\") " pod="openshift-multus/multus-additional-cni-plugins-cplsq" Nov 04 04:47:17.524910 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.524210 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c79eff94-6543-4313-b59b-502ca4089e84-kubelet-config\") pod \"global-pull-secret-syncer-gfjqz\" (UID: \"c79eff94-6543-4313-b59b-502ca4089e84\") " pod="kube-system/global-pull-secret-syncer-gfjqz" Nov 04 04:47:17.524910 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.524263 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cce8fa86-4f61-4dfc-b36c-2cefb7ecaa03-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dmdhl\" (UID: \"cce8fa86-4f61-4dfc-b36c-2cefb7ecaa03\") " pod="openshift-insights/insights-runtime-extractor-dmdhl" Nov 04 04:47:17.524910 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.524311 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/dd0aaef0-d114-43b4-9ce4-135e818e4df3-config-volume\") pod \"dns-default-4xxrr\" (UID: \"dd0aaef0-d114-43b4-9ce4-135e818e4df3\") " pod="openshift-dns/dns-default-4xxrr" Nov 04 04:47:17.524910 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.524397 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7l4mv\" (UniqueName: \"kubernetes.io/projected/dd0aaef0-d114-43b4-9ce4-135e818e4df3-kube-api-access-7l4mv\") pod \"dns-default-4xxrr\" (UID: \"dd0aaef0-d114-43b4-9ce4-135e818e4df3\") " pod="openshift-dns/dns-default-4xxrr" Nov 04 04:47:17.524910 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.524428 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4dff9faf-d91f-4ed5-be82-555b9a6a924e-metrics-certs\") pod \"network-metrics-daemon-25flt\" (UID: \"4dff9faf-d91f-4ed5-be82-555b9a6a924e\") " pod="openshift-multus/network-metrics-daemon-25flt" Nov 04 04:47:17.524910 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.524455 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d737ccc2-5686-4130-9d00-d2f21d5b417f-host-slash\") pod \"iptables-alerter-wdk7z\" (UID: \"d737ccc2-5686-4130-9d00-d2f21d5b417f\") " pod="openshift-network-operator/iptables-alerter-wdk7z" Nov 04 04:47:17.524910 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.524548 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/dd0aaef0-d114-43b4-9ce4-135e818e4df3-config-volume\") pod \"dns-default-4xxrr\" (UID: \"dd0aaef0-d114-43b4-9ce4-135e818e4df3\") " pod="openshift-dns/dns-default-4xxrr" Nov 04 04:47:17.524910 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.524550 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/844cc12f-a7f6-4327-968e-5e9b2da00ad0-ovn-node-metrics-cert\") pod \"ovnkube-node-dvc6z\" (UID: \"844cc12f-a7f6-4327-968e-5e9b2da00ad0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvc6z" Nov 04 04:47:17.524910 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.524561 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlz69\" (UniqueName: \"kubernetes.io/projected/9a06ec07-c72b-43af-aa27-7414aa9c9aba-kube-api-access-mlz69\") pod \"node-ca-4z89d\" (UID: \"9a06ec07-c72b-43af-aa27-7414aa9c9aba\") " pod="openshift-image-registry/node-ca-4z89d" Nov 04 04:47:17.524910 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.524592 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/00e0471e-5fe8-4494-8f17-632aee4f6c30-socket-dir\") pod \"aws-ebs-csi-driver-node-f2zrp\" (UID: \"00e0471e-5fe8-4494-8f17-632aee4f6c30\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f2zrp" Nov 04 04:47:17.524910 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.524608 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8446aa71-5638-4b23-8f5b-12283ede21d4-tmp-dir\") pod \"node-resolver-gzctm\" (UID: \"8446aa71-5638-4b23-8f5b-12283ede21d4\") " pod="openshift-dns/node-resolver-gzctm" Nov 04 04:47:17.524910 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.524629 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/844cc12f-a7f6-4327-968e-5e9b2da00ad0-host-cni-bin\") pod \"ovnkube-node-dvc6z\" (UID: \"844cc12f-a7f6-4327-968e-5e9b2da00ad0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvc6z" Nov 04 04:47:17.524910 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.524661 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4dff9faf-d91f-4ed5-be82-555b9a6a924e-metrics-certs\") pod \"network-metrics-daemon-25flt\" (UID: \"4dff9faf-d91f-4ed5-be82-555b9a6a924e\") " pod="openshift-multus/network-metrics-daemon-25flt" Nov 04 04:47:17.524910 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.524745 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2qjt\" (UniqueName: \"kubernetes.io/projected/21f0e203-fdf7-42cf-9c0c-4103e6601020-kube-api-access-x2qjt\") pod \"ingress-canary-mt4dj\" (UID: \"21f0e203-fdf7-42cf-9c0c-4103e6601020\") " pod="openshift-ingress-canary/ingress-canary-mt4dj" Nov 04 04:47:17.524910 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.524763 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8446aa71-5638-4b23-8f5b-12283ede21d4-tmp-dir\") pod \"node-resolver-gzctm\" (UID: \"8446aa71-5638-4b23-8f5b-12283ede21d4\") " pod="openshift-dns/node-resolver-gzctm" Nov 04 04:47:17.524910 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.524775 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hx8dr\" (UniqueName: \"kubernetes.io/projected/4dff9faf-d91f-4ed5-be82-555b9a6a924e-kube-api-access-hx8dr\") pod \"network-metrics-daemon-25flt\" (UID: \"4dff9faf-d91f-4ed5-be82-555b9a6a924e\") " pod="openshift-multus/network-metrics-daemon-25flt" Nov 04 04:47:17.524910 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.524803 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/58be7092-9156-4787-8f49-a173fa0bd5a3-multus-socket-dir-parent\") pod \"multus-7m8fc\" (UID: \"58be7092-9156-4787-8f49-a173fa0bd5a3\") " pod="openshift-multus/multus-7m8fc" Nov 04 04:47:17.524910 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.524831 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/58be7092-9156-4787-8f49-a173fa0bd5a3-host-run-k8s-cni-cncf-io\") pod \"multus-7m8fc\" (UID: \"58be7092-9156-4787-8f49-a173fa0bd5a3\") " pod="openshift-multus/multus-7m8fc" Nov 04 04:47:17.524910 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.524855 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8446aa71-5638-4b23-8f5b-12283ede21d4-hosts-file\") pod \"node-resolver-gzctm\" (UID: \"8446aa71-5638-4b23-8f5b-12283ede21d4\") " pod="openshift-dns/node-resolver-gzctm" Nov 04 04:47:17.524910 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.524897 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/844cc12f-a7f6-4327-968e-5e9b2da00ad0-ovnkube-script-lib\") pod \"ovnkube-node-dvc6z\" (UID: \"844cc12f-a7f6-4327-968e-5e9b2da00ad0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvc6z" Nov 04 04:47:17.524910 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.524923 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/58be7092-9156-4787-8f49-a173fa0bd5a3-multus-conf-dir\") pod \"multus-7m8fc\" (UID: \"58be7092-9156-4787-8f49-a173fa0bd5a3\") " pod="openshift-multus/multus-7m8fc" Nov 04 04:47:17.525828 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.524961 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/22d31d0c-6edf-4351-8f41-a6dd4e02368b-sys\") pod \"tuned-c65v8\" (UID: \"22d31d0c-6edf-4351-8f41-a6dd4e02368b\") " pod="openshift-cluster-node-tuning-operator/tuned-c65v8" Nov 04 04:47:17.525828 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.525000 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/23ed4a39-18b5-4bc6-9c7b-8b60a7043ad2-agent-certs\") pod \"konnectivity-agent-7tw97\" (UID: \"23ed4a39-18b5-4bc6-9c7b-8b60a7043ad2\") " pod="kube-system/konnectivity-agent-7tw97" Nov 04 04:47:17.525828 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.525041 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/198e9ab0-f615-4686-9acf-a6242f7e3584-cnibin\") pod \"multus-additional-cni-plugins-cplsq\" (UID: \"198e9ab0-f615-4686-9acf-a6242f7e3584\") " pod="openshift-multus/multus-additional-cni-plugins-cplsq" Nov 04 04:47:17.525828 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.525110 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/844cc12f-a7f6-4327-968e-5e9b2da00ad0-ovnkube-script-lib\") pod \"ovnkube-node-dvc6z\" (UID: \"844cc12f-a7f6-4327-968e-5e9b2da00ad0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvc6z" Nov 04 04:47:17.525828 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.525124 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/58be7092-9156-4787-8f49-a173fa0bd5a3-multus-daemon-config\") pod \"multus-7m8fc\" (UID: \"58be7092-9156-4787-8f49-a173fa0bd5a3\") " pod="openshift-multus/multus-7m8fc" Nov 04 04:47:17.525828 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.525217 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/58be7092-9156-4787-8f49-a173fa0bd5a3-host-run-multus-certs\") pod \"multus-7m8fc\" (UID: \"58be7092-9156-4787-8f49-a173fa0bd5a3\") " pod="openshift-multus/multus-7m8fc" Nov 04 04:47:17.525828 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.525232 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/23ed4a39-18b5-4bc6-9c7b-8b60a7043ad2-agent-certs\") pod \"konnectivity-agent-7tw97\" (UID: \"23ed4a39-18b5-4bc6-9c7b-8b60a7043ad2\") " pod="kube-system/konnectivity-agent-7tw97" Nov 04 04:47:17.525828 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.525235 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/23ed4a39-18b5-4bc6-9c7b-8b60a7043ad2-konnectivity-ca\") pod \"konnectivity-agent-7tw97\" (UID: \"23ed4a39-18b5-4bc6-9c7b-8b60a7043ad2\") " pod="kube-system/konnectivity-agent-7tw97" Nov 04 04:47:17.525828 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.525259 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/58be7092-9156-4787-8f49-a173fa0bd5a3-multus-daemon-config\") pod \"multus-7m8fc\" (UID: \"58be7092-9156-4787-8f49-a173fa0bd5a3\") " pod="openshift-multus/multus-7m8fc" Nov 04 04:47:17.525828 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.525265 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c79eff94-6543-4313-b59b-502ca4089e84-original-pull-secret\") pod \"global-pull-secret-syncer-gfjqz\" (UID: \"c79eff94-6543-4313-b59b-502ca4089e84\") " pod="kube-system/global-pull-secret-syncer-gfjqz" Nov 04 04:47:17.525828 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.525291 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/22d31d0c-6edf-4351-8f41-a6dd4e02368b-var-lib-kubelet\") pod \"tuned-c65v8\" (UID: \"22d31d0c-6edf-4351-8f41-a6dd4e02368b\") " pod="openshift-cluster-node-tuning-operator/tuned-c65v8" Nov 04 04:47:17.525828 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.525306 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/22d31d0c-6edf-4351-8f41-a6dd4e02368b-host\") pod \"tuned-c65v8\" (UID: \"22d31d0c-6edf-4351-8f41-a6dd4e02368b\") " pod="openshift-cluster-node-tuning-operator/tuned-c65v8" Nov 04 04:47:17.525828 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.525341 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/844cc12f-a7f6-4327-968e-5e9b2da00ad0-run-openvswitch\") pod \"ovnkube-node-dvc6z\" (UID: \"844cc12f-a7f6-4327-968e-5e9b2da00ad0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvc6z" Nov 04 04:47:17.525828 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.525367 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/844cc12f-a7f6-4327-968e-5e9b2da00ad0-ovnkube-config\") pod \"ovnkube-node-dvc6z\" (UID: \"844cc12f-a7f6-4327-968e-5e9b2da00ad0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvc6z" Nov 04 04:47:17.525828 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.525399 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/00e0471e-5fe8-4494-8f17-632aee4f6c30-registration-dir\") pod \"aws-ebs-csi-driver-node-f2zrp\" (UID: \"00e0471e-5fe8-4494-8f17-632aee4f6c30\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f2zrp" Nov 04 04:47:17.525828 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.525402 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/23ed4a39-18b5-4bc6-9c7b-8b60a7043ad2-konnectivity-ca\") pod \"konnectivity-agent-7tw97\" (UID: \"23ed4a39-18b5-4bc6-9c7b-8b60a7043ad2\") " pod="kube-system/konnectivity-agent-7tw97" Nov 04 04:47:17.525828 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.525453 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c79eff94-6543-4313-b59b-502ca4089e84-original-pull-secret\") pod \"global-pull-secret-syncer-gfjqz\" (UID: \"c79eff94-6543-4313-b59b-502ca4089e84\") " pod="kube-system/global-pull-secret-syncer-gfjqz" Nov 04 04:47:17.525828 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.525523 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/844cc12f-a7f6-4327-968e-5e9b2da00ad0-systemd-units\") pod \"ovnkube-node-dvc6z\" (UID: \"844cc12f-a7f6-4327-968e-5e9b2da00ad0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvc6z" Nov 04 04:47:17.525828 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.525558 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/844cc12f-a7f6-4327-968e-5e9b2da00ad0-host-slash\") pod \"ovnkube-node-dvc6z\" (UID: \"844cc12f-a7f6-4327-968e-5e9b2da00ad0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvc6z" Nov 04 04:47:17.525828 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.525562 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/844cc12f-a7f6-4327-968e-5e9b2da00ad0-ovnkube-config\") pod \"ovnkube-node-dvc6z\" (UID: \"844cc12f-a7f6-4327-968e-5e9b2da00ad0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvc6z" Nov 04 04:47:17.525828 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.525596 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/844cc12f-a7f6-4327-968e-5e9b2da00ad0-run-ovn\") pod \"ovnkube-node-dvc6z\" (UID: \"844cc12f-a7f6-4327-968e-5e9b2da00ad0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvc6z" Nov 04 04:47:17.525828 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.525621 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/198e9ab0-f615-4686-9acf-a6242f7e3584-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cplsq\" (UID: \"198e9ab0-f615-4686-9acf-a6242f7e3584\") " pod="openshift-multus/multus-additional-cni-plugins-cplsq" Nov 04 04:47:17.525828 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.525636 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dd0aaef0-d114-43b4-9ce4-135e818e4df3-metrics-tls\") pod \"dns-default-4xxrr\" (UID: \"dd0aaef0-d114-43b4-9ce4-135e818e4df3\") " pod="openshift-dns/dns-default-4xxrr" Nov 04 04:47:17.525828 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.525666 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/58be7092-9156-4787-8f49-a173fa0bd5a3-host-run-netns\") pod \"multus-7m8fc\" (UID: \"58be7092-9156-4787-8f49-a173fa0bd5a3\") " pod="openshift-multus/multus-7m8fc" Nov 04 04:47:17.525828 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.525768 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25g6k\" (UniqueName: \"kubernetes.io/projected/8446aa71-5638-4b23-8f5b-12283ede21d4-kube-api-access-25g6k\") pod \"node-resolver-gzctm\" (UID: \"8446aa71-5638-4b23-8f5b-12283ede21d4\") " pod="openshift-dns/node-resolver-gzctm" Nov 04 04:47:17.525828 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.525792 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dd0aaef0-d114-43b4-9ce4-135e818e4df3-metrics-tls\") pod \"dns-default-4xxrr\" (UID: \"dd0aaef0-d114-43b4-9ce4-135e818e4df3\") " pod="openshift-dns/dns-default-4xxrr" Nov 04 04:47:17.525828 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.525799 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/22d31d0c-6edf-4351-8f41-a6dd4e02368b-etc-sysctl-d\") pod \"tuned-c65v8\" (UID: \"22d31d0c-6edf-4351-8f41-a6dd4e02368b\") " pod="openshift-cluster-node-tuning-operator/tuned-c65v8" Nov 04 04:47:17.525828 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.525819 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/844cc12f-a7f6-4327-968e-5e9b2da00ad0-host-cni-netd\") pod \"ovnkube-node-dvc6z\" (UID: \"844cc12f-a7f6-4327-968e-5e9b2da00ad0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvc6z" Nov 04 04:47:17.525828 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.525834 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/198e9ab0-f615-4686-9acf-a6242f7e3584-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cplsq\" (UID: \"198e9ab0-f615-4686-9acf-a6242f7e3584\") " pod="openshift-multus/multus-additional-cni-plugins-cplsq" Nov 04 04:47:17.525828 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.525839 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd0aaef0-d114-43b4-9ce4-135e818e4df3-tmp-dir\") pod \"dns-default-4xxrr\" (UID: \"dd0aaef0-d114-43b4-9ce4-135e818e4df3\") " pod="openshift-dns/dns-default-4xxrr" Nov 04 04:47:17.526659 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.525907 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd0aaef0-d114-43b4-9ce4-135e818e4df3-tmp-dir\") pod \"dns-default-4xxrr\" (UID: \"dd0aaef0-d114-43b4-9ce4-135e818e4df3\") " pod="openshift-dns/dns-default-4xxrr" Nov 04 04:47:17.526659 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.525938 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/58be7092-9156-4787-8f49-a173fa0bd5a3-etc-kubernetes\") pod \"multus-7m8fc\" (UID: \"58be7092-9156-4787-8f49-a173fa0bd5a3\") " pod="openshift-multus/multus-7m8fc" Nov 04 04:47:17.577535 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.577507 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cgfm\" (UniqueName: \"kubernetes.io/projected/cce8fa86-4f61-4dfc-b36c-2cefb7ecaa03-kube-api-access-7cgfm\") pod \"insights-runtime-extractor-dmdhl\" (UID: \"cce8fa86-4f61-4dfc-b36c-2cefb7ecaa03\") " pod="openshift-insights/insights-runtime-extractor-dmdhl" Nov 04 04:47:17.577773 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.577747 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nch9\" (UniqueName: \"kubernetes.io/projected/844cc12f-a7f6-4327-968e-5e9b2da00ad0-kube-api-access-6nch9\") pod \"ovnkube-node-dvc6z\" (UID: \"844cc12f-a7f6-4327-968e-5e9b2da00ad0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvc6z" Nov 04 04:47:17.578078 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.578057 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4xvx\" (UniqueName: \"kubernetes.io/projected/58be7092-9156-4787-8f49-a173fa0bd5a3-kube-api-access-w4xvx\") pod \"multus-7m8fc\" (UID: \"58be7092-9156-4787-8f49-a173fa0bd5a3\") " pod="openshift-multus/multus-7m8fc" Nov 04 04:47:17.578282 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.578261 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccfxs\" (UniqueName: \"kubernetes.io/projected/198e9ab0-f615-4686-9acf-a6242f7e3584-kube-api-access-ccfxs\") pod \"multus-additional-cni-plugins-cplsq\" (UID: \"198e9ab0-f615-4686-9acf-a6242f7e3584\") " pod="openshift-multus/multus-additional-cni-plugins-cplsq" Nov 04 04:47:17.578681 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.578654 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kz2hd\" (UniqueName: \"kubernetes.io/projected/d737ccc2-5686-4130-9d00-d2f21d5b417f-kube-api-access-kz2hd\") pod \"iptables-alerter-wdk7z\" (UID: \"d737ccc2-5686-4130-9d00-d2f21d5b417f\") " pod="openshift-network-operator/iptables-alerter-wdk7z" Nov 04 04:47:17.578681 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.578676 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gftxp\" (UniqueName: \"kubernetes.io/projected/00e0471e-5fe8-4494-8f17-632aee4f6c30-kube-api-access-gftxp\") pod \"aws-ebs-csi-driver-node-f2zrp\" (UID: \"00e0471e-5fe8-4494-8f17-632aee4f6c30\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f2zrp" Nov 04 04:47:17.578800 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.578717 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7l4mv\" (UniqueName: \"kubernetes.io/projected/dd0aaef0-d114-43b4-9ce4-135e818e4df3-kube-api-access-7l4mv\") pod \"dns-default-4xxrr\" (UID: \"dd0aaef0-d114-43b4-9ce4-135e818e4df3\") " pod="openshift-dns/dns-default-4xxrr" Nov 04 04:47:17.578800 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.578710 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbtxf\" (UniqueName: \"kubernetes.io/projected/22d31d0c-6edf-4351-8f41-a6dd4e02368b-kube-api-access-zbtxf\") pod \"tuned-c65v8\" (UID: \"22d31d0c-6edf-4351-8f41-a6dd4e02368b\") " pod="openshift-cluster-node-tuning-operator/tuned-c65v8" Nov 04 04:47:17.578989 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.578820 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47nf5\" (UniqueName: \"kubernetes.io/projected/5f6922fa-9ba7-4d02-b550-606731852fde-kube-api-access-47nf5\") pod \"network-check-target-8hsp8\" (UID: \"5f6922fa-9ba7-4d02-b550-606731852fde\") " pod="openshift-network-diagnostics/network-check-target-8hsp8" Nov 04 04:47:17.579156 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.579137 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlz69\" (UniqueName: \"kubernetes.io/projected/9a06ec07-c72b-43af-aa27-7414aa9c9aba-kube-api-access-mlz69\") pod \"node-ca-4z89d\" (UID: \"9a06ec07-c72b-43af-aa27-7414aa9c9aba\") " pod="openshift-image-registry/node-ca-4z89d" Nov 04 04:47:17.579433 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.579416 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hx8dr\" (UniqueName: \"kubernetes.io/projected/4dff9faf-d91f-4ed5-be82-555b9a6a924e-kube-api-access-hx8dr\") pod \"network-metrics-daemon-25flt\" (UID: \"4dff9faf-d91f-4ed5-be82-555b9a6a924e\") " pod="openshift-multus/network-metrics-daemon-25flt" Nov 04 04:47:17.579573 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.579557 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2qjt\" (UniqueName: \"kubernetes.io/projected/21f0e203-fdf7-42cf-9c0c-4103e6601020-kube-api-access-x2qjt\") pod \"ingress-canary-mt4dj\" (UID: \"21f0e203-fdf7-42cf-9c0c-4103e6601020\") " pod="openshift-ingress-canary/ingress-canary-mt4dj" Nov 04 04:47:17.581658 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.581640 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-25g6k\" (UniqueName: \"kubernetes.io/projected/8446aa71-5638-4b23-8f5b-12283ede21d4-kube-api-access-25g6k\") pod \"node-resolver-gzctm\" (UID: \"8446aa71-5638-4b23-8f5b-12283ede21d4\") " pod="openshift-dns/node-resolver-gzctm" Nov 04 04:47:17.626590 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.626567 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/844cc12f-a7f6-4327-968e-5e9b2da00ad0-etc-openvswitch\") pod \"ovnkube-node-dvc6z\" (UID: \"844cc12f-a7f6-4327-968e-5e9b2da00ad0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvc6z" Nov 04 04:47:17.626674 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.626594 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/844cc12f-a7f6-4327-968e-5e9b2da00ad0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dvc6z\" (UID: \"844cc12f-a7f6-4327-968e-5e9b2da00ad0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvc6z" Nov 04 04:47:17.626674 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.626612 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/58be7092-9156-4787-8f49-a173fa0bd5a3-cnibin\") pod \"multus-7m8fc\" (UID: \"58be7092-9156-4787-8f49-a173fa0bd5a3\") " pod="openshift-multus/multus-7m8fc" Nov 04 04:47:17.626674 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.626644 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/22d31d0c-6edf-4351-8f41-a6dd4e02368b-etc-systemd\") pod \"tuned-c65v8\" (UID: \"22d31d0c-6edf-4351-8f41-a6dd4e02368b\") " pod="openshift-cluster-node-tuning-operator/tuned-c65v8" Nov 04 04:47:17.626674 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.626664 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/22d31d0c-6edf-4351-8f41-a6dd4e02368b-run\") pod \"tuned-c65v8\" (UID: \"22d31d0c-6edf-4351-8f41-a6dd4e02368b\") " pod="openshift-cluster-node-tuning-operator/tuned-c65v8" Nov 04 04:47:17.626867 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.626681 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/844cc12f-a7f6-4327-968e-5e9b2da00ad0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dvc6z\" (UID: \"844cc12f-a7f6-4327-968e-5e9b2da00ad0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvc6z" Nov 04 04:47:17.626867 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.626685 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/22d31d0c-6edf-4351-8f41-a6dd4e02368b-lib-modules\") pod \"tuned-c65v8\" (UID: \"22d31d0c-6edf-4351-8f41-a6dd4e02368b\") " pod="openshift-cluster-node-tuning-operator/tuned-c65v8" Nov 04 04:47:17.626867 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.626721 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/58be7092-9156-4787-8f49-a173fa0bd5a3-cnibin\") pod \"multus-7m8fc\" (UID: \"58be7092-9156-4787-8f49-a173fa0bd5a3\") " pod="openshift-multus/multus-7m8fc" Nov 04 04:47:17.626867 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.626732 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/844cc12f-a7f6-4327-968e-5e9b2da00ad0-run-systemd\") pod \"ovnkube-node-dvc6z\" (UID: \"844cc12f-a7f6-4327-968e-5e9b2da00ad0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvc6z" Nov 04 04:47:17.626867 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.626731 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/22d31d0c-6edf-4351-8f41-a6dd4e02368b-etc-systemd\") pod \"tuned-c65v8\" (UID: \"22d31d0c-6edf-4351-8f41-a6dd4e02368b\") " pod="openshift-cluster-node-tuning-operator/tuned-c65v8" Nov 04 04:47:17.626867 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.626681 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/844cc12f-a7f6-4327-968e-5e9b2da00ad0-etc-openvswitch\") pod \"ovnkube-node-dvc6z\" (UID: \"844cc12f-a7f6-4327-968e-5e9b2da00ad0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvc6z" Nov 04 04:47:17.626867 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.626742 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/22d31d0c-6edf-4351-8f41-a6dd4e02368b-run\") pod \"tuned-c65v8\" (UID: \"22d31d0c-6edf-4351-8f41-a6dd4e02368b\") " pod="openshift-cluster-node-tuning-operator/tuned-c65v8" Nov 04 04:47:17.626867 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.626770 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/00e0471e-5fe8-4494-8f17-632aee4f6c30-kubelet-dir\") pod \"aws-ebs-csi-driver-node-f2zrp\" (UID: \"00e0471e-5fe8-4494-8f17-632aee4f6c30\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f2zrp" Nov 04 04:47:17.626867 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.626774 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/844cc12f-a7f6-4327-968e-5e9b2da00ad0-run-systemd\") pod \"ovnkube-node-dvc6z\" (UID: \"844cc12f-a7f6-4327-968e-5e9b2da00ad0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvc6z" Nov 04 04:47:17.626867 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.626783 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/22d31d0c-6edf-4351-8f41-a6dd4e02368b-lib-modules\") pod \"tuned-c65v8\" (UID: \"22d31d0c-6edf-4351-8f41-a6dd4e02368b\") " pod="openshift-cluster-node-tuning-operator/tuned-c65v8" Nov 04 04:47:17.626867 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.626798 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9a06ec07-c72b-43af-aa27-7414aa9c9aba-host\") pod \"node-ca-4z89d\" (UID: \"9a06ec07-c72b-43af-aa27-7414aa9c9aba\") " pod="openshift-image-registry/node-ca-4z89d" Nov 04 04:47:17.626867 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.626835 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/00e0471e-5fe8-4494-8f17-632aee4f6c30-kubelet-dir\") pod \"aws-ebs-csi-driver-node-f2zrp\" (UID: \"00e0471e-5fe8-4494-8f17-632aee4f6c30\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f2zrp" Nov 04 04:47:17.626867 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.626865 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/22d31d0c-6edf-4351-8f41-a6dd4e02368b-etc-kubernetes\") pod \"tuned-c65v8\" (UID: \"22d31d0c-6edf-4351-8f41-a6dd4e02368b\") " pod="openshift-cluster-node-tuning-operator/tuned-c65v8" Nov 04 04:47:17.627438 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.626887 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/844cc12f-a7f6-4327-968e-5e9b2da00ad0-var-lib-openvswitch\") pod \"ovnkube-node-dvc6z\" (UID: \"844cc12f-a7f6-4327-968e-5e9b2da00ad0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvc6z" Nov 04 04:47:17.627438 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.626904 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/58be7092-9156-4787-8f49-a173fa0bd5a3-system-cni-dir\") pod \"multus-7m8fc\" (UID: \"58be7092-9156-4787-8f49-a173fa0bd5a3\") " pod="openshift-multus/multus-7m8fc" Nov 04 04:47:17.627438 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.626918 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/58be7092-9156-4787-8f49-a173fa0bd5a3-host-var-lib-kubelet\") pod \"multus-7m8fc\" (UID: \"58be7092-9156-4787-8f49-a173fa0bd5a3\") " pod="openshift-multus/multus-7m8fc" Nov 04 04:47:17.627438 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.626862 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9a06ec07-c72b-43af-aa27-7414aa9c9aba-host\") pod \"node-ca-4z89d\" (UID: \"9a06ec07-c72b-43af-aa27-7414aa9c9aba\") " pod="openshift-image-registry/node-ca-4z89d" Nov 04 04:47:17.627438 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.626938 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/00e0471e-5fe8-4494-8f17-632aee4f6c30-etc-selinux\") pod \"aws-ebs-csi-driver-node-f2zrp\" (UID: \"00e0471e-5fe8-4494-8f17-632aee4f6c30\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f2zrp" Nov 04 04:47:17.627438 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.626959 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/22d31d0c-6edf-4351-8f41-a6dd4e02368b-etc-kubernetes\") pod \"tuned-c65v8\" (UID: \"22d31d0c-6edf-4351-8f41-a6dd4e02368b\") " pod="openshift-cluster-node-tuning-operator/tuned-c65v8" Nov 04 04:47:17.627438 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.626960 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/22d31d0c-6edf-4351-8f41-a6dd4e02368b-etc-sysctl-conf\") pod \"tuned-c65v8\" (UID: \"22d31d0c-6edf-4351-8f41-a6dd4e02368b\") " pod="openshift-cluster-node-tuning-operator/tuned-c65v8" Nov 04 04:47:17.627438 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.626974 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/58be7092-9156-4787-8f49-a173fa0bd5a3-host-var-lib-kubelet\") pod \"multus-7m8fc\" (UID: \"58be7092-9156-4787-8f49-a173fa0bd5a3\") " pod="openshift-multus/multus-7m8fc" Nov 04 04:47:17.627438 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.626981 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/844cc12f-a7f6-4327-968e-5e9b2da00ad0-host-run-netns\") pod \"ovnkube-node-dvc6z\" (UID: \"844cc12f-a7f6-4327-968e-5e9b2da00ad0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvc6z" Nov 04 04:47:17.627438 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.626986 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/58be7092-9156-4787-8f49-a173fa0bd5a3-system-cni-dir\") pod \"multus-7m8fc\" (UID: \"58be7092-9156-4787-8f49-a173fa0bd5a3\") " pod="openshift-multus/multus-7m8fc" Nov 04 04:47:17.627438 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.627000 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/844cc12f-a7f6-4327-968e-5e9b2da00ad0-host-run-netns\") pod \"ovnkube-node-dvc6z\" (UID: \"844cc12f-a7f6-4327-968e-5e9b2da00ad0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvc6z" Nov 04 04:47:17.627438 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.627013 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/844cc12f-a7f6-4327-968e-5e9b2da00ad0-log-socket\") pod \"ovnkube-node-dvc6z\" (UID: \"844cc12f-a7f6-4327-968e-5e9b2da00ad0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvc6z" Nov 04 04:47:17.627438 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.627024 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/844cc12f-a7f6-4327-968e-5e9b2da00ad0-var-lib-openvswitch\") pod \"ovnkube-node-dvc6z\" (UID: \"844cc12f-a7f6-4327-968e-5e9b2da00ad0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvc6z" Nov 04 04:47:17.627438 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.627036 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/00e0471e-5fe8-4494-8f17-632aee4f6c30-etc-selinux\") pod \"aws-ebs-csi-driver-node-f2zrp\" (UID: \"00e0471e-5fe8-4494-8f17-632aee4f6c30\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f2zrp" Nov 04 04:47:17.627438 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.627040 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/58be7092-9156-4787-8f49-a173fa0bd5a3-multus-cni-dir\") pod \"multus-7m8fc\" (UID: \"58be7092-9156-4787-8f49-a173fa0bd5a3\") " pod="openshift-multus/multus-7m8fc" Nov 04 04:47:17.627438 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.627048 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/844cc12f-a7f6-4327-968e-5e9b2da00ad0-log-socket\") pod \"ovnkube-node-dvc6z\" (UID: \"844cc12f-a7f6-4327-968e-5e9b2da00ad0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvc6z" Nov 04 04:47:17.627438 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.627064 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/58be7092-9156-4787-8f49-a173fa0bd5a3-host-var-lib-cni-bin\") pod \"multus-7m8fc\" (UID: \"58be7092-9156-4787-8f49-a173fa0bd5a3\") " pod="openshift-multus/multus-7m8fc" Nov 04 04:47:17.627438 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.627085 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/58be7092-9156-4787-8f49-a173fa0bd5a3-multus-cni-dir\") pod \"multus-7m8fc\" (UID: \"58be7092-9156-4787-8f49-a173fa0bd5a3\") " pod="openshift-multus/multus-7m8fc" Nov 04 04:47:17.627438 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.627089 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/58be7092-9156-4787-8f49-a173fa0bd5a3-hostroot\") pod \"multus-7m8fc\" (UID: \"58be7092-9156-4787-8f49-a173fa0bd5a3\") " pod="openshift-multus/multus-7m8fc" Nov 04 04:47:17.627438 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.627102 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/58be7092-9156-4787-8f49-a173fa0bd5a3-host-var-lib-cni-bin\") pod \"multus-7m8fc\" (UID: \"58be7092-9156-4787-8f49-a173fa0bd5a3\") " pod="openshift-multus/multus-7m8fc" Nov 04 04:47:17.627438 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.627109 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/58be7092-9156-4787-8f49-a173fa0bd5a3-hostroot\") pod \"multus-7m8fc\" (UID: \"58be7092-9156-4787-8f49-a173fa0bd5a3\") " pod="openshift-multus/multus-7m8fc" Nov 04 04:47:17.627438 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.627068 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/22d31d0c-6edf-4351-8f41-a6dd4e02368b-etc-sysctl-conf\") pod \"tuned-c65v8\" (UID: \"22d31d0c-6edf-4351-8f41-a6dd4e02368b\") " pod="openshift-cluster-node-tuning-operator/tuned-c65v8" Nov 04 04:47:17.627438 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.627113 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/198e9ab0-f615-4686-9acf-a6242f7e3584-system-cni-dir\") pod \"multus-additional-cni-plugins-cplsq\" (UID: \"198e9ab0-f615-4686-9acf-a6242f7e3584\") " pod="openshift-multus/multus-additional-cni-plugins-cplsq" Nov 04 04:47:17.627438 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.627135 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/cce8fa86-4f61-4dfc-b36c-2cefb7ecaa03-crio-socket\") pod \"insights-runtime-extractor-dmdhl\" (UID: \"cce8fa86-4f61-4dfc-b36c-2cefb7ecaa03\") " pod="openshift-insights/insights-runtime-extractor-dmdhl" Nov 04 04:47:17.627438 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.627148 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/198e9ab0-f615-4686-9acf-a6242f7e3584-system-cni-dir\") pod \"multus-additional-cni-plugins-cplsq\" (UID: \"198e9ab0-f615-4686-9acf-a6242f7e3584\") " pod="openshift-multus/multus-additional-cni-plugins-cplsq" Nov 04 04:47:17.627438 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.627151 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/22d31d0c-6edf-4351-8f41-a6dd4e02368b-etc-sysconfig\") pod \"tuned-c65v8\" (UID: \"22d31d0c-6edf-4351-8f41-a6dd4e02368b\") " pod="openshift-cluster-node-tuning-operator/tuned-c65v8" Nov 04 04:47:17.627438 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.627176 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/844cc12f-a7f6-4327-968e-5e9b2da00ad0-node-log\") pod \"ovnkube-node-dvc6z\" (UID: \"844cc12f-a7f6-4327-968e-5e9b2da00ad0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvc6z" Nov 04 04:47:17.627438 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.627185 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/22d31d0c-6edf-4351-8f41-a6dd4e02368b-etc-sysconfig\") pod \"tuned-c65v8\" (UID: \"22d31d0c-6edf-4351-8f41-a6dd4e02368b\") " pod="openshift-cluster-node-tuning-operator/tuned-c65v8" Nov 04 04:47:17.627438 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.627198 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/844cc12f-a7f6-4327-968e-5e9b2da00ad0-node-log\") pod \"ovnkube-node-dvc6z\" (UID: \"844cc12f-a7f6-4327-968e-5e9b2da00ad0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvc6z" Nov 04 04:47:17.627438 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.627202 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/844cc12f-a7f6-4327-968e-5e9b2da00ad0-host-run-ovn-kubernetes\") pod \"ovnkube-node-dvc6z\" (UID: \"844cc12f-a7f6-4327-968e-5e9b2da00ad0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvc6z" Nov 04 04:47:17.627438 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.627217 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/cce8fa86-4f61-4dfc-b36c-2cefb7ecaa03-crio-socket\") pod \"insights-runtime-extractor-dmdhl\" (UID: \"cce8fa86-4f61-4dfc-b36c-2cefb7ecaa03\") " pod="openshift-insights/insights-runtime-extractor-dmdhl" Nov 04 04:47:17.627438 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.627219 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/844cc12f-a7f6-4327-968e-5e9b2da00ad0-host-run-ovn-kubernetes\") pod \"ovnkube-node-dvc6z\" (UID: \"844cc12f-a7f6-4327-968e-5e9b2da00ad0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvc6z" Nov 04 04:47:17.627438 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.627227 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/58be7092-9156-4787-8f49-a173fa0bd5a3-host-var-lib-cni-multus\") pod \"multus-7m8fc\" (UID: \"58be7092-9156-4787-8f49-a173fa0bd5a3\") " pod="openshift-multus/multus-7m8fc" Nov 04 04:47:17.627438 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.627254 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/00e0471e-5fe8-4494-8f17-632aee4f6c30-device-dir\") pod \"aws-ebs-csi-driver-node-f2zrp\" (UID: \"00e0471e-5fe8-4494-8f17-632aee4f6c30\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f2zrp" Nov 04 04:47:17.627438 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.627285 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c79eff94-6543-4313-b59b-502ca4089e84-dbus\") pod \"global-pull-secret-syncer-gfjqz\" (UID: \"c79eff94-6543-4313-b59b-502ca4089e84\") " pod="kube-system/global-pull-secret-syncer-gfjqz" Nov 04 04:47:17.627438 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.627306 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/58be7092-9156-4787-8f49-a173fa0bd5a3-host-var-lib-cni-multus\") pod \"multus-7m8fc\" (UID: \"58be7092-9156-4787-8f49-a173fa0bd5a3\") " pod="openshift-multus/multus-7m8fc" Nov 04 04:47:17.627438 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.627314 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/22d31d0c-6edf-4351-8f41-a6dd4e02368b-etc-modprobe-d\") pod \"tuned-c65v8\" (UID: \"22d31d0c-6edf-4351-8f41-a6dd4e02368b\") " pod="openshift-cluster-node-tuning-operator/tuned-c65v8" Nov 04 04:47:17.627438 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.627363 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/844cc12f-a7f6-4327-968e-5e9b2da00ad0-host-kubelet\") pod \"ovnkube-node-dvc6z\" (UID: \"844cc12f-a7f6-4327-968e-5e9b2da00ad0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvc6z" Nov 04 04:47:17.627438 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.627376 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c79eff94-6543-4313-b59b-502ca4089e84-dbus\") pod \"global-pull-secret-syncer-gfjqz\" (UID: \"c79eff94-6543-4313-b59b-502ca4089e84\") " pod="kube-system/global-pull-secret-syncer-gfjqz" Nov 04 04:47:17.627438 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.627390 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/198e9ab0-f615-4686-9acf-a6242f7e3584-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cplsq\" (UID: \"198e9ab0-f615-4686-9acf-a6242f7e3584\") " pod="openshift-multus/multus-additional-cni-plugins-cplsq" Nov 04 04:47:17.627438 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.627409 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/844cc12f-a7f6-4327-968e-5e9b2da00ad0-host-kubelet\") pod \"ovnkube-node-dvc6z\" (UID: \"844cc12f-a7f6-4327-968e-5e9b2da00ad0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvc6z" Nov 04 04:47:17.627438 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.627414 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c79eff94-6543-4313-b59b-502ca4089e84-kubelet-config\") pod \"global-pull-secret-syncer-gfjqz\" (UID: \"c79eff94-6543-4313-b59b-502ca4089e84\") " pod="kube-system/global-pull-secret-syncer-gfjqz" Nov 04 04:47:17.627438 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.627424 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/198e9ab0-f615-4686-9acf-a6242f7e3584-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cplsq\" (UID: \"198e9ab0-f615-4686-9acf-a6242f7e3584\") " pod="openshift-multus/multus-additional-cni-plugins-cplsq" Nov 04 04:47:17.627438 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.627439 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cce8fa86-4f61-4dfc-b36c-2cefb7ecaa03-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dmdhl\" (UID: \"cce8fa86-4f61-4dfc-b36c-2cefb7ecaa03\") " pod="openshift-insights/insights-runtime-extractor-dmdhl" Nov 04 04:47:17.627438 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.627445 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/22d31d0c-6edf-4351-8f41-a6dd4e02368b-etc-modprobe-d\") pod \"tuned-c65v8\" (UID: \"22d31d0c-6edf-4351-8f41-a6dd4e02368b\") " pod="openshift-cluster-node-tuning-operator/tuned-c65v8" Nov 04 04:47:17.627438 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.627363 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/00e0471e-5fe8-4494-8f17-632aee4f6c30-device-dir\") pod \"aws-ebs-csi-driver-node-f2zrp\" (UID: \"00e0471e-5fe8-4494-8f17-632aee4f6c30\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f2zrp" Nov 04 04:47:17.627438 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.627454 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c79eff94-6543-4313-b59b-502ca4089e84-kubelet-config\") pod \"global-pull-secret-syncer-gfjqz\" (UID: \"c79eff94-6543-4313-b59b-502ca4089e84\") " pod="kube-system/global-pull-secret-syncer-gfjqz" Nov 04 04:47:17.627438 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.627463 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d737ccc2-5686-4130-9d00-d2f21d5b417f-host-slash\") pod \"iptables-alerter-wdk7z\" (UID: \"d737ccc2-5686-4130-9d00-d2f21d5b417f\") " pod="openshift-network-operator/iptables-alerter-wdk7z" Nov 04 04:47:17.628910 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.627486 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/00e0471e-5fe8-4494-8f17-632aee4f6c30-socket-dir\") pod \"aws-ebs-csi-driver-node-f2zrp\" (UID: \"00e0471e-5fe8-4494-8f17-632aee4f6c30\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f2zrp" Nov 04 04:47:17.628910 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.627490 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d737ccc2-5686-4130-9d00-d2f21d5b417f-host-slash\") pod \"iptables-alerter-wdk7z\" (UID: \"d737ccc2-5686-4130-9d00-d2f21d5b417f\") " pod="openshift-network-operator/iptables-alerter-wdk7z" Nov 04 04:47:17.628910 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.627508 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/844cc12f-a7f6-4327-968e-5e9b2da00ad0-host-cni-bin\") pod \"ovnkube-node-dvc6z\" (UID: \"844cc12f-a7f6-4327-968e-5e9b2da00ad0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvc6z" Nov 04 04:47:17.628910 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.627535 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/58be7092-9156-4787-8f49-a173fa0bd5a3-multus-socket-dir-parent\") pod \"multus-7m8fc\" (UID: \"58be7092-9156-4787-8f49-a173fa0bd5a3\") " pod="openshift-multus/multus-7m8fc" Nov 04 04:47:17.628910 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.627557 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/00e0471e-5fe8-4494-8f17-632aee4f6c30-socket-dir\") pod \"aws-ebs-csi-driver-node-f2zrp\" (UID: \"00e0471e-5fe8-4494-8f17-632aee4f6c30\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f2zrp" Nov 04 04:47:17.628910 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.627561 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/58be7092-9156-4787-8f49-a173fa0bd5a3-host-run-k8s-cni-cncf-io\") pod \"multus-7m8fc\" (UID: \"58be7092-9156-4787-8f49-a173fa0bd5a3\") " pod="openshift-multus/multus-7m8fc" Nov 04 04:47:17.628910 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.627573 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/844cc12f-a7f6-4327-968e-5e9b2da00ad0-host-cni-bin\") pod \"ovnkube-node-dvc6z\" (UID: \"844cc12f-a7f6-4327-968e-5e9b2da00ad0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvc6z" Nov 04 04:47:17.628910 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.627585 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8446aa71-5638-4b23-8f5b-12283ede21d4-hosts-file\") pod \"node-resolver-gzctm\" (UID: \"8446aa71-5638-4b23-8f5b-12283ede21d4\") " pod="openshift-dns/node-resolver-gzctm" Nov 04 04:47:17.628910 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.627601 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/58be7092-9156-4787-8f49-a173fa0bd5a3-multus-socket-dir-parent\") pod \"multus-7m8fc\" (UID: \"58be7092-9156-4787-8f49-a173fa0bd5a3\") " pod="openshift-multus/multus-7m8fc" Nov 04 04:47:17.628910 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.627610 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/58be7092-9156-4787-8f49-a173fa0bd5a3-multus-conf-dir\") pod \"multus-7m8fc\" (UID: \"58be7092-9156-4787-8f49-a173fa0bd5a3\") " pod="openshift-multus/multus-7m8fc" Nov 04 04:47:17.628910 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.627616 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/58be7092-9156-4787-8f49-a173fa0bd5a3-host-run-k8s-cni-cncf-io\") pod \"multus-7m8fc\" (UID: \"58be7092-9156-4787-8f49-a173fa0bd5a3\") " pod="openshift-multus/multus-7m8fc" Nov 04 04:47:17.628910 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.627631 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/22d31d0c-6edf-4351-8f41-a6dd4e02368b-sys\") pod \"tuned-c65v8\" (UID: \"22d31d0c-6edf-4351-8f41-a6dd4e02368b\") " pod="openshift-cluster-node-tuning-operator/tuned-c65v8" Nov 04 04:47:17.628910 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.627657 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/58be7092-9156-4787-8f49-a173fa0bd5a3-multus-conf-dir\") pod \"multus-7m8fc\" (UID: \"58be7092-9156-4787-8f49-a173fa0bd5a3\") " pod="openshift-multus/multus-7m8fc" Nov 04 04:47:17.628910 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.627657 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/198e9ab0-f615-4686-9acf-a6242f7e3584-cnibin\") pod \"multus-additional-cni-plugins-cplsq\" (UID: \"198e9ab0-f615-4686-9acf-a6242f7e3584\") " pod="openshift-multus/multus-additional-cni-plugins-cplsq" Nov 04 04:47:17.628910 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.627643 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8446aa71-5638-4b23-8f5b-12283ede21d4-hosts-file\") pod \"node-resolver-gzctm\" (UID: \"8446aa71-5638-4b23-8f5b-12283ede21d4\") " pod="openshift-dns/node-resolver-gzctm" Nov 04 04:47:17.628910 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.627686 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/58be7092-9156-4787-8f49-a173fa0bd5a3-host-run-multus-certs\") pod \"multus-7m8fc\" (UID: \"58be7092-9156-4787-8f49-a173fa0bd5a3\") " pod="openshift-multus/multus-7m8fc" Nov 04 04:47:17.628910 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.627704 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/22d31d0c-6edf-4351-8f41-a6dd4e02368b-var-lib-kubelet\") pod \"tuned-c65v8\" (UID: \"22d31d0c-6edf-4351-8f41-a6dd4e02368b\") " pod="openshift-cluster-node-tuning-operator/tuned-c65v8" Nov 04 04:47:17.628910 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.627708 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/198e9ab0-f615-4686-9acf-a6242f7e3584-cnibin\") pod \"multus-additional-cni-plugins-cplsq\" (UID: \"198e9ab0-f615-4686-9acf-a6242f7e3584\") " pod="openshift-multus/multus-additional-cni-plugins-cplsq" Nov 04 04:47:17.628910 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.627722 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/22d31d0c-6edf-4351-8f41-a6dd4e02368b-host\") pod \"tuned-c65v8\" (UID: \"22d31d0c-6edf-4351-8f41-a6dd4e02368b\") " pod="openshift-cluster-node-tuning-operator/tuned-c65v8" Nov 04 04:47:17.628910 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.627735 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/58be7092-9156-4787-8f49-a173fa0bd5a3-host-run-multus-certs\") pod \"multus-7m8fc\" (UID: \"58be7092-9156-4787-8f49-a173fa0bd5a3\") " pod="openshift-multus/multus-7m8fc" Nov 04 04:47:17.628910 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.627741 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/844cc12f-a7f6-4327-968e-5e9b2da00ad0-run-openvswitch\") pod \"ovnkube-node-dvc6z\" (UID: \"844cc12f-a7f6-4327-968e-5e9b2da00ad0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvc6z" Nov 04 04:47:17.628910 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.627758 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/844cc12f-a7f6-4327-968e-5e9b2da00ad0-run-openvswitch\") pod \"ovnkube-node-dvc6z\" (UID: \"844cc12f-a7f6-4327-968e-5e9b2da00ad0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvc6z" Nov 04 04:47:17.628910 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.627685 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/22d31d0c-6edf-4351-8f41-a6dd4e02368b-sys\") pod \"tuned-c65v8\" (UID: \"22d31d0c-6edf-4351-8f41-a6dd4e02368b\") " pod="openshift-cluster-node-tuning-operator/tuned-c65v8" Nov 04 04:47:17.628910 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.627760 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/22d31d0c-6edf-4351-8f41-a6dd4e02368b-var-lib-kubelet\") pod \"tuned-c65v8\" (UID: \"22d31d0c-6edf-4351-8f41-a6dd4e02368b\") " pod="openshift-cluster-node-tuning-operator/tuned-c65v8" Nov 04 04:47:17.628910 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.627778 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/00e0471e-5fe8-4494-8f17-632aee4f6c30-registration-dir\") pod \"aws-ebs-csi-driver-node-f2zrp\" (UID: \"00e0471e-5fe8-4494-8f17-632aee4f6c30\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f2zrp" Nov 04 04:47:17.628910 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.627783 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/22d31d0c-6edf-4351-8f41-a6dd4e02368b-host\") pod \"tuned-c65v8\" (UID: \"22d31d0c-6edf-4351-8f41-a6dd4e02368b\") " pod="openshift-cluster-node-tuning-operator/tuned-c65v8" Nov 04 04:47:17.628910 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.627753 4509 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 04 04:47:17.628910 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.627809 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/844cc12f-a7f6-4327-968e-5e9b2da00ad0-systemd-units\") pod \"ovnkube-node-dvc6z\" (UID: \"844cc12f-a7f6-4327-968e-5e9b2da00ad0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvc6z" Nov 04 04:47:17.628910 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.627813 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/00e0471e-5fe8-4494-8f17-632aee4f6c30-registration-dir\") pod \"aws-ebs-csi-driver-node-f2zrp\" (UID: \"00e0471e-5fe8-4494-8f17-632aee4f6c30\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f2zrp" Nov 04 04:47:17.628910 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.627824 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/844cc12f-a7f6-4327-968e-5e9b2da00ad0-host-slash\") pod \"ovnkube-node-dvc6z\" (UID: \"844cc12f-a7f6-4327-968e-5e9b2da00ad0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvc6z" Nov 04 04:47:17.628910 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.627839 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/844cc12f-a7f6-4327-968e-5e9b2da00ad0-run-ovn\") pod \"ovnkube-node-dvc6z\" (UID: \"844cc12f-a7f6-4327-968e-5e9b2da00ad0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvc6z" Nov 04 04:47:17.628910 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.627840 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/844cc12f-a7f6-4327-968e-5e9b2da00ad0-systemd-units\") pod \"ovnkube-node-dvc6z\" (UID: \"844cc12f-a7f6-4327-968e-5e9b2da00ad0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvc6z" Nov 04 04:47:17.628910 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.627853 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/58be7092-9156-4787-8f49-a173fa0bd5a3-host-run-netns\") pod \"multus-7m8fc\" (UID: \"58be7092-9156-4787-8f49-a173fa0bd5a3\") " pod="openshift-multus/multus-7m8fc" Nov 04 04:47:17.628910 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.627862 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/844cc12f-a7f6-4327-968e-5e9b2da00ad0-host-slash\") pod \"ovnkube-node-dvc6z\" (UID: \"844cc12f-a7f6-4327-968e-5e9b2da00ad0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvc6z" Nov 04 04:47:17.628910 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.627868 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/22d31d0c-6edf-4351-8f41-a6dd4e02368b-etc-sysctl-d\") pod \"tuned-c65v8\" (UID: \"22d31d0c-6edf-4351-8f41-a6dd4e02368b\") " pod="openshift-cluster-node-tuning-operator/tuned-c65v8" Nov 04 04:47:17.628910 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.627908 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/58be7092-9156-4787-8f49-a173fa0bd5a3-host-run-netns\") pod \"multus-7m8fc\" (UID: \"58be7092-9156-4787-8f49-a173fa0bd5a3\") " pod="openshift-multus/multus-7m8fc" Nov 04 04:47:17.628910 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.627916 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/844cc12f-a7f6-4327-968e-5e9b2da00ad0-run-ovn\") pod \"ovnkube-node-dvc6z\" (UID: \"844cc12f-a7f6-4327-968e-5e9b2da00ad0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvc6z" Nov 04 04:47:17.628910 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.627939 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/844cc12f-a7f6-4327-968e-5e9b2da00ad0-host-cni-netd\") pod \"ovnkube-node-dvc6z\" (UID: \"844cc12f-a7f6-4327-968e-5e9b2da00ad0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvc6z" Nov 04 04:47:17.628910 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.627949 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/22d31d0c-6edf-4351-8f41-a6dd4e02368b-etc-sysctl-d\") pod \"tuned-c65v8\" (UID: \"22d31d0c-6edf-4351-8f41-a6dd4e02368b\") " pod="openshift-cluster-node-tuning-operator/tuned-c65v8" Nov 04 04:47:17.628910 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.627958 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/58be7092-9156-4787-8f49-a173fa0bd5a3-etc-kubernetes\") pod \"multus-7m8fc\" (UID: \"58be7092-9156-4787-8f49-a173fa0bd5a3\") " pod="openshift-multus/multus-7m8fc" Nov 04 04:47:17.628910 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.627974 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/00e0471e-5fe8-4494-8f17-632aee4f6c30-sys-fs\") pod \"aws-ebs-csi-driver-node-f2zrp\" (UID: \"00e0471e-5fe8-4494-8f17-632aee4f6c30\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f2zrp" Nov 04 04:47:17.628910 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.628001 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/844cc12f-a7f6-4327-968e-5e9b2da00ad0-host-cni-netd\") pod \"ovnkube-node-dvc6z\" (UID: \"844cc12f-a7f6-4327-968e-5e9b2da00ad0\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvc6z" Nov 04 04:47:17.628910 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.628018 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/00e0471e-5fe8-4494-8f17-632aee4f6c30-sys-fs\") pod \"aws-ebs-csi-driver-node-f2zrp\" (UID: \"00e0471e-5fe8-4494-8f17-632aee4f6c30\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f2zrp" Nov 04 04:47:17.628910 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.628028 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/58be7092-9156-4787-8f49-a173fa0bd5a3-etc-kubernetes\") pod \"multus-7m8fc\" (UID: \"58be7092-9156-4787-8f49-a173fa0bd5a3\") " pod="openshift-multus/multus-7m8fc" Nov 04 04:47:17.630085 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.630027 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cce8fa86-4f61-4dfc-b36c-2cefb7ecaa03-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dmdhl\" (UID: \"cce8fa86-4f61-4dfc-b36c-2cefb7ecaa03\") " pod="openshift-insights/insights-runtime-extractor-dmdhl" Nov 04 04:47:17.659227 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:17.659203 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-dmdhl" Nov 04 04:47:18.068017 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:18.067987 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-dmdhl"] Nov 04 04:47:18.071664 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:18.071633 4509 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcce8fa86_4f61_4dfc_b36c_2cefb7ecaa03.slice/crio-fb2da1e5f83e058ea2ec89cabf98b33381d0cc5f3d8d3d551e8e8c125fbbc872 WatchSource:0}: Error finding container fb2da1e5f83e058ea2ec89cabf98b33381d0cc5f3d8d3d551e8e8c125fbbc872: Status 404 returned error can't find the container with id fb2da1e5f83e058ea2ec89cabf98b33381d0cc5f3d8d3d551e8e8c125fbbc872 Nov 04 04:47:18.176480 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:18.176451 4509 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 04 04:47:18.299994 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:18.299956 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dmdhl" event={"ID":"cce8fa86-4f61-4dfc-b36c-2cefb7ecaa03","Type":"ContainerStarted","Data":"74a3cdcb0b72c29fc3ff2a153f3d5d5634066bf378a9f4d0a26ac478668382c5"} Nov 04 04:47:18.300515 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:18.300018 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dmdhl" event={"ID":"cce8fa86-4f61-4dfc-b36c-2cefb7ecaa03","Type":"ContainerStarted","Data":"fb2da1e5f83e058ea2ec89cabf98b33381d0cc5f3d8d3d551e8e8c125fbbc872"} Nov 04 04:47:19.039804 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:19.039777 4509 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-4xxrr" Nov 04 04:47:19.757658 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:19.757629 4509 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dvc6z" Nov 04 04:47:20.308722 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:20.308685 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dmdhl" event={"ID":"cce8fa86-4f61-4dfc-b36c-2cefb7ecaa03","Type":"ContainerStarted","Data":"6439cf8a6c575527d9eadb1b0bebff7a3305a712a5a0ff7512b318868d44187b"} Nov 04 04:47:20.456674 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:20.456631 4509 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-8hsp8" Nov 04 04:47:21.249054 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:21.249011 4509 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tw97_23ed4a39-18b5-4bc6-9c7b-8b60a7043ad2/konnectivity-agent/0.log" limit=1024 Nov 04 04:47:21.583575 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:21.583550 4509 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tw97_23ed4a39-18b5-4bc6-9c7b-8b60a7043ad2/konnectivity-agent/0.log" limit=1024 Nov 04 04:47:21.766111 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:21.766077 4509 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tw97_23ed4a39-18b5-4bc6-9c7b-8b60a7043ad2/konnectivity-agent/0.log" limit=1024 Nov 04 04:47:22.011369 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:22.011340 4509 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tw97_23ed4a39-18b5-4bc6-9c7b-8b60a7043ad2/konnectivity-agent/0.log" limit=1024 Nov 04 04:47:22.184137 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:22.184105 4509 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tw97_23ed4a39-18b5-4bc6-9c7b-8b60a7043ad2/konnectivity-agent/0.log" limit=1024 Nov 04 04:47:22.317385 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:22.317302 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dmdhl" event={"ID":"cce8fa86-4f61-4dfc-b36c-2cefb7ecaa03","Type":"ContainerStarted","Data":"4105343270dcc05a70fc2e62322a13eee1908b438f31388c7f79ec43c88a0057"} Nov 04 04:47:24.620151 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:24.620118 4509 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dvc6z" Nov 04 04:47:24.620605 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:24.620407 4509 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 04 04:47:25.042510 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:25.042480 4509 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-4xxrr" Nov 04 04:47:26.994845 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:26.994812 4509 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dvc6z" Nov 04 04:47:30.342557 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:30.342524 4509 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-9k5vv"] Nov 04 04:47:30.344086 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:30.344061 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-9k5vv" Nov 04 04:47:30.392781 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:30.392747 4509 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-kllmc"] Nov 04 04:47:30.394457 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:30.394441 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-kllmc" Nov 04 04:47:30.401838 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:30.401813 4509 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Nov 04 04:47:30.408603 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:30.408579 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pvzz\" (UniqueName: \"kubernetes.io/projected/50a99580-f574-461f-95e2-f6bc901f77ce-kube-api-access-9pvzz\") pod \"downloads-6dd789c8b7-kllmc\" (UID: \"50a99580-f574-461f-95e2-f6bc901f77ce\") " pod="openshift-console/downloads-6dd789c8b7-kllmc" Nov 04 04:47:30.408714 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:30.408611 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/50a99580-f574-461f-95e2-f6bc901f77ce-tmp\") pod \"downloads-6dd789c8b7-kllmc\" (UID: \"50a99580-f574-461f-95e2-f6bc901f77ce\") " pod="openshift-console/downloads-6dd789c8b7-kllmc" Nov 04 04:47:30.408714 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:30.408647 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/b4d26ce7-b503-4c3d-8e1e-02ff73858fbe-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-9k5vv\" (UID: \"b4d26ce7-b503-4c3d-8e1e-02ff73858fbe\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-9k5vv" Nov 04 04:47:30.442181 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:30.442158 4509 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-k22xh\"" Nov 04 04:47:30.452482 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:30.452460 4509 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Nov 04 04:47:30.452603 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:30.452512 4509 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Nov 04 04:47:30.453439 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:30.453421 4509 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5dd6d64c77-mpvgf"] Nov 04 04:47:30.455471 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:30.455453 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5dd6d64c77-mpvgf" Nov 04 04:47:30.465362 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:30.465316 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-kllmc"] Nov 04 04:47:30.502289 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:30.502269 4509 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-s8cc5\"" Nov 04 04:47:30.509480 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:30.509442 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/44b44e26-531b-4e12-bec6-c6df914813e5-ca-trust-extracted\") pod \"image-registry-5dd6d64c77-mpvgf\" (UID: \"44b44e26-531b-4e12-bec6-c6df914813e5\") " pod="openshift-image-registry/image-registry-5dd6d64c77-mpvgf" Nov 04 04:47:30.509545 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:30.509500 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/b4d26ce7-b503-4c3d-8e1e-02ff73858fbe-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-9k5vv\" (UID: \"b4d26ce7-b503-4c3d-8e1e-02ff73858fbe\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-9k5vv" Nov 04 04:47:30.509545 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:30.509524 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/44b44e26-531b-4e12-bec6-c6df914813e5-registry-tls\") pod \"image-registry-5dd6d64c77-mpvgf\" (UID: \"44b44e26-531b-4e12-bec6-c6df914813e5\") " pod="openshift-image-registry/image-registry-5dd6d64c77-mpvgf" Nov 04 04:47:30.509637 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:30.509558 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/44b44e26-531b-4e12-bec6-c6df914813e5-registry-certificates\") pod \"image-registry-5dd6d64c77-mpvgf\" (UID: \"44b44e26-531b-4e12-bec6-c6df914813e5\") " pod="openshift-image-registry/image-registry-5dd6d64c77-mpvgf" Nov 04 04:47:30.509637 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:30.509582 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/44b44e26-531b-4e12-bec6-c6df914813e5-bound-sa-token\") pod \"image-registry-5dd6d64c77-mpvgf\" (UID: \"44b44e26-531b-4e12-bec6-c6df914813e5\") " pod="openshift-image-registry/image-registry-5dd6d64c77-mpvgf" Nov 04 04:47:30.509708 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:30.509686 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/44b44e26-531b-4e12-bec6-c6df914813e5-image-registry-private-configuration\") pod \"image-registry-5dd6d64c77-mpvgf\" (UID: \"44b44e26-531b-4e12-bec6-c6df914813e5\") " pod="openshift-image-registry/image-registry-5dd6d64c77-mpvgf" Nov 04 04:47:30.509743 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:30.509713 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nswml\" (UniqueName: \"kubernetes.io/projected/44b44e26-531b-4e12-bec6-c6df914813e5-kube-api-access-nswml\") pod \"image-registry-5dd6d64c77-mpvgf\" (UID: \"44b44e26-531b-4e12-bec6-c6df914813e5\") " pod="openshift-image-registry/image-registry-5dd6d64c77-mpvgf" Nov 04 04:47:30.509743 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:30.509736 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pvzz\" (UniqueName: \"kubernetes.io/projected/50a99580-f574-461f-95e2-f6bc901f77ce-kube-api-access-9pvzz\") pod \"downloads-6dd789c8b7-kllmc\" (UID: \"50a99580-f574-461f-95e2-f6bc901f77ce\") " pod="openshift-console/downloads-6dd789c8b7-kllmc" Nov 04 04:47:30.509818 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:30.509758 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/44b44e26-531b-4e12-bec6-c6df914813e5-installation-pull-secrets\") pod \"image-registry-5dd6d64c77-mpvgf\" (UID: \"44b44e26-531b-4e12-bec6-c6df914813e5\") " pod="openshift-image-registry/image-registry-5dd6d64c77-mpvgf" Nov 04 04:47:30.509850 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:30.509822 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/50a99580-f574-461f-95e2-f6bc901f77ce-tmp\") pod \"downloads-6dd789c8b7-kllmc\" (UID: \"50a99580-f574-461f-95e2-f6bc901f77ce\") " pod="openshift-console/downloads-6dd789c8b7-kllmc" Nov 04 04:47:30.509931 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:30.509914 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/44b44e26-531b-4e12-bec6-c6df914813e5-trusted-ca\") pod \"image-registry-5dd6d64c77-mpvgf\" (UID: \"44b44e26-531b-4e12-bec6-c6df914813e5\") " pod="openshift-image-registry/image-registry-5dd6d64c77-mpvgf" Nov 04 04:47:30.510170 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:30.510155 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/50a99580-f574-461f-95e2-f6bc901f77ce-tmp\") pod \"downloads-6dd789c8b7-kllmc\" (UID: \"50a99580-f574-461f-95e2-f6bc901f77ce\") " pod="openshift-console/downloads-6dd789c8b7-kllmc" Nov 04 04:47:30.511955 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:30.511940 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/b4d26ce7-b503-4c3d-8e1e-02ff73858fbe-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-9k5vv\" (UID: \"b4d26ce7-b503-4c3d-8e1e-02ff73858fbe\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-9k5vv" Nov 04 04:47:30.558735 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:30.558709 4509 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Nov 04 04:47:30.564755 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:30.564735 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5dd6d64c77-mpvgf"] Nov 04 04:47:30.565076 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:30.565061 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-9k5vv"] Nov 04 04:47:30.569653 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:30.569638 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pvzz\" (UniqueName: \"kubernetes.io/projected/50a99580-f574-461f-95e2-f6bc901f77ce-kube-api-access-9pvzz\") pod \"downloads-6dd789c8b7-kllmc\" (UID: \"50a99580-f574-461f-95e2-f6bc901f77ce\") " pod="openshift-console/downloads-6dd789c8b7-kllmc" Nov 04 04:47:30.602608 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:30.602523 4509 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Nov 04 04:47:30.602608 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:30.602533 4509 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-cvvjk\"" Nov 04 04:47:30.602608 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:30.602599 4509 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Nov 04 04:47:30.602795 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:30.602600 4509 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Nov 04 04:47:30.610862 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:30.610841 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/44b44e26-531b-4e12-bec6-c6df914813e5-bound-sa-token\") pod \"image-registry-5dd6d64c77-mpvgf\" (UID: \"44b44e26-531b-4e12-bec6-c6df914813e5\") " pod="openshift-image-registry/image-registry-5dd6d64c77-mpvgf" Nov 04 04:47:30.610909 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:30.610871 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/44b44e26-531b-4e12-bec6-c6df914813e5-image-registry-private-configuration\") pod \"image-registry-5dd6d64c77-mpvgf\" (UID: \"44b44e26-531b-4e12-bec6-c6df914813e5\") " pod="openshift-image-registry/image-registry-5dd6d64c77-mpvgf" Nov 04 04:47:30.610909 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:30.610887 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nswml\" (UniqueName: \"kubernetes.io/projected/44b44e26-531b-4e12-bec6-c6df914813e5-kube-api-access-nswml\") pod \"image-registry-5dd6d64c77-mpvgf\" (UID: \"44b44e26-531b-4e12-bec6-c6df914813e5\") " pod="openshift-image-registry/image-registry-5dd6d64c77-mpvgf" Nov 04 04:47:30.610989 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:30.610908 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/44b44e26-531b-4e12-bec6-c6df914813e5-installation-pull-secrets\") pod \"image-registry-5dd6d64c77-mpvgf\" (UID: \"44b44e26-531b-4e12-bec6-c6df914813e5\") " pod="openshift-image-registry/image-registry-5dd6d64c77-mpvgf" Nov 04 04:47:30.610989 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:30.610932 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/44b44e26-531b-4e12-bec6-c6df914813e5-trusted-ca\") pod \"image-registry-5dd6d64c77-mpvgf\" (UID: \"44b44e26-531b-4e12-bec6-c6df914813e5\") " pod="openshift-image-registry/image-registry-5dd6d64c77-mpvgf" Nov 04 04:47:30.610989 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:30.610959 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/44b44e26-531b-4e12-bec6-c6df914813e5-ca-trust-extracted\") pod \"image-registry-5dd6d64c77-mpvgf\" (UID: \"44b44e26-531b-4e12-bec6-c6df914813e5\") " pod="openshift-image-registry/image-registry-5dd6d64c77-mpvgf" Nov 04 04:47:30.611143 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:30.610995 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/44b44e26-531b-4e12-bec6-c6df914813e5-registry-tls\") pod \"image-registry-5dd6d64c77-mpvgf\" (UID: \"44b44e26-531b-4e12-bec6-c6df914813e5\") " pod="openshift-image-registry/image-registry-5dd6d64c77-mpvgf" Nov 04 04:47:30.611143 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:30.611011 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/44b44e26-531b-4e12-bec6-c6df914813e5-registry-certificates\") pod \"image-registry-5dd6d64c77-mpvgf\" (UID: \"44b44e26-531b-4e12-bec6-c6df914813e5\") " pod="openshift-image-registry/image-registry-5dd6d64c77-mpvgf" Nov 04 04:47:30.611519 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:30.611497 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/44b44e26-531b-4e12-bec6-c6df914813e5-ca-trust-extracted\") pod \"image-registry-5dd6d64c77-mpvgf\" (UID: \"44b44e26-531b-4e12-bec6-c6df914813e5\") " pod="openshift-image-registry/image-registry-5dd6d64c77-mpvgf" Nov 04 04:47:30.611915 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:30.611893 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/44b44e26-531b-4e12-bec6-c6df914813e5-registry-certificates\") pod \"image-registry-5dd6d64c77-mpvgf\" (UID: \"44b44e26-531b-4e12-bec6-c6df914813e5\") " pod="openshift-image-registry/image-registry-5dd6d64c77-mpvgf" Nov 04 04:47:30.612092 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:30.612076 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/44b44e26-531b-4e12-bec6-c6df914813e5-trusted-ca\") pod \"image-registry-5dd6d64c77-mpvgf\" (UID: \"44b44e26-531b-4e12-bec6-c6df914813e5\") " pod="openshift-image-registry/image-registry-5dd6d64c77-mpvgf" Nov 04 04:47:30.613521 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:30.613499 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/44b44e26-531b-4e12-bec6-c6df914813e5-registry-tls\") pod \"image-registry-5dd6d64c77-mpvgf\" (UID: \"44b44e26-531b-4e12-bec6-c6df914813e5\") " pod="openshift-image-registry/image-registry-5dd6d64c77-mpvgf" Nov 04 04:47:30.613628 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:30.613505 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/44b44e26-531b-4e12-bec6-c6df914813e5-image-registry-private-configuration\") pod \"image-registry-5dd6d64c77-mpvgf\" (UID: \"44b44e26-531b-4e12-bec6-c6df914813e5\") " pod="openshift-image-registry/image-registry-5dd6d64c77-mpvgf" Nov 04 04:47:30.613682 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:30.613631 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/44b44e26-531b-4e12-bec6-c6df914813e5-installation-pull-secrets\") pod \"image-registry-5dd6d64c77-mpvgf\" (UID: \"44b44e26-531b-4e12-bec6-c6df914813e5\") " pod="openshift-image-registry/image-registry-5dd6d64c77-mpvgf" Nov 04 04:47:30.653471 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:30.653436 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-9k5vv" Nov 04 04:47:30.667342 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:30.667303 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/44b44e26-531b-4e12-bec6-c6df914813e5-bound-sa-token\") pod \"image-registry-5dd6d64c77-mpvgf\" (UID: \"44b44e26-531b-4e12-bec6-c6df914813e5\") " pod="openshift-image-registry/image-registry-5dd6d64c77-mpvgf" Nov 04 04:47:30.667606 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:30.667589 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nswml\" (UniqueName: \"kubernetes.io/projected/44b44e26-531b-4e12-bec6-c6df914813e5-kube-api-access-nswml\") pod \"image-registry-5dd6d64c77-mpvgf\" (UID: \"44b44e26-531b-4e12-bec6-c6df914813e5\") " pod="openshift-image-registry/image-registry-5dd6d64c77-mpvgf" Nov 04 04:47:30.704038 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:30.704010 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-kllmc" Nov 04 04:47:30.766359 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:30.765645 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5dd6d64c77-mpvgf" Nov 04 04:47:31.016927 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:31.016824 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-9k5vv"] Nov 04 04:47:31.020635 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:31.020612 4509 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4d26ce7_b503_4c3d_8e1e_02ff73858fbe.slice/crio-3c611b223a1fabb47d3044e7b00f9cf7acf7d15f6fbfa9de49d795514d818b82 WatchSource:0}: Error finding container 3c611b223a1fabb47d3044e7b00f9cf7acf7d15f6fbfa9de49d795514d818b82: Status 404 returned error can't find the container with id 3c611b223a1fabb47d3044e7b00f9cf7acf7d15f6fbfa9de49d795514d818b82 Nov 04 04:47:31.082635 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:31.079450 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-kllmc"] Nov 04 04:47:31.133607 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:31.133571 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5dd6d64c77-mpvgf"] Nov 04 04:47:31.136608 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:31.136584 4509 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44b44e26_531b_4e12_bec6_c6df914813e5.slice/crio-bd6919f905697314e5985d887a59d36d3579dba5f63b99c848b883cc12d7280b WatchSource:0}: Error finding container bd6919f905697314e5985d887a59d36d3579dba5f63b99c848b883cc12d7280b: Status 404 returned error can't find the container with id bd6919f905697314e5985d887a59d36d3579dba5f63b99c848b883cc12d7280b Nov 04 04:47:31.346312 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:31.346231 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5dd6d64c77-mpvgf" event={"ID":"44b44e26-531b-4e12-bec6-c6df914813e5","Type":"ContainerStarted","Data":"8723ac4f0ff40e08105f6d9caa83594e706b87a8ffc32fad6f326809f32dd840"} Nov 04 04:47:31.346312 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:31.346279 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5dd6d64c77-mpvgf" event={"ID":"44b44e26-531b-4e12-bec6-c6df914813e5","Type":"ContainerStarted","Data":"bd6919f905697314e5985d887a59d36d3579dba5f63b99c848b883cc12d7280b"} Nov 04 04:47:31.346812 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:31.346337 4509 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5dd6d64c77-mpvgf" Nov 04 04:47:31.347611 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:31.347588 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-kllmc" event={"ID":"50a99580-f574-461f-95e2-f6bc901f77ce","Type":"ContainerStarted","Data":"fb21664b720c123b3d9718e2517ce58347259d3f64a765cf798d5c7274fa75ce"} Nov 04 04:47:31.348849 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:31.348828 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-9k5vv" event={"ID":"b4d26ce7-b503-4c3d-8e1e-02ff73858fbe","Type":"ContainerStarted","Data":"3c611b223a1fabb47d3044e7b00f9cf7acf7d15f6fbfa9de49d795514d818b82"} Nov 04 04:47:31.461955 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:31.461889 4509 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5dd6d64c77-mpvgf" podStartSLOduration=1.461869621 podStartE2EDuration="1.461869621s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 04:47:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:47:31.461703951 +0000 UTC m=+15.339427494" watchObservedRunningTime="2026-11-04 04:47:31.461869621 +0000 UTC m=+15.339593152" Nov 04 04:47:31.967506 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:31.967469 4509 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-7567f6b9c5-nztb2"] Nov 04 04:47:31.969958 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:31.969942 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7567f6b9c5-nztb2" Nov 04 04:47:32.020424 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:32.020385 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/50622b62-1450-4af2-80fc-d2f3d8f82d27-console-oauth-config\") pod \"console-7567f6b9c5-nztb2\" (UID: \"50622b62-1450-4af2-80fc-d2f3d8f82d27\") " pod="openshift-console/console-7567f6b9c5-nztb2" Nov 04 04:47:32.020586 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:32.020435 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/50622b62-1450-4af2-80fc-d2f3d8f82d27-console-config\") pod \"console-7567f6b9c5-nztb2\" (UID: \"50622b62-1450-4af2-80fc-d2f3d8f82d27\") " pod="openshift-console/console-7567f6b9c5-nztb2" Nov 04 04:47:32.020586 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:32.020530 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/50622b62-1450-4af2-80fc-d2f3d8f82d27-tmp\") pod \"console-7567f6b9c5-nztb2\" (UID: \"50622b62-1450-4af2-80fc-d2f3d8f82d27\") " pod="openshift-console/console-7567f6b9c5-nztb2" Nov 04 04:47:32.020586 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:32.020576 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/50622b62-1450-4af2-80fc-d2f3d8f82d27-console-serving-cert\") pod \"console-7567f6b9c5-nztb2\" (UID: \"50622b62-1450-4af2-80fc-d2f3d8f82d27\") " pod="openshift-console/console-7567f6b9c5-nztb2" Nov 04 04:47:32.020747 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:32.020603 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/50622b62-1450-4af2-80fc-d2f3d8f82d27-service-ca\") pod \"console-7567f6b9c5-nztb2\" (UID: \"50622b62-1450-4af2-80fc-d2f3d8f82d27\") " pod="openshift-console/console-7567f6b9c5-nztb2" Nov 04 04:47:32.020747 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:32.020680 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/50622b62-1450-4af2-80fc-d2f3d8f82d27-oauth-serving-cert\") pod \"console-7567f6b9c5-nztb2\" (UID: \"50622b62-1450-4af2-80fc-d2f3d8f82d27\") " pod="openshift-console/console-7567f6b9c5-nztb2" Nov 04 04:47:32.020813 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:32.020779 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlnb5\" (UniqueName: \"kubernetes.io/projected/50622b62-1450-4af2-80fc-d2f3d8f82d27-kube-api-access-nlnb5\") pod \"console-7567f6b9c5-nztb2\" (UID: \"50622b62-1450-4af2-80fc-d2f3d8f82d27\") " pod="openshift-console/console-7567f6b9c5-nztb2" Nov 04 04:47:32.023480 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:32.023455 4509 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Nov 04 04:47:32.023480 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:32.023463 4509 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Nov 04 04:47:32.023480 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:32.023491 4509 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Nov 04 04:47:32.023680 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:32.023509 4509 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Nov 04 04:47:32.023680 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:32.023519 4509 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Nov 04 04:47:32.023680 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:32.023575 4509 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-drn8g\"" Nov 04 04:47:32.028719 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:32.028697 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7567f6b9c5-nztb2"] Nov 04 04:47:32.121166 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:32.121128 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/50622b62-1450-4af2-80fc-d2f3d8f82d27-console-oauth-config\") pod \"console-7567f6b9c5-nztb2\" (UID: \"50622b62-1450-4af2-80fc-d2f3d8f82d27\") " pod="openshift-console/console-7567f6b9c5-nztb2" Nov 04 04:47:32.121364 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:32.121173 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/50622b62-1450-4af2-80fc-d2f3d8f82d27-console-config\") pod \"console-7567f6b9c5-nztb2\" (UID: \"50622b62-1450-4af2-80fc-d2f3d8f82d27\") " pod="openshift-console/console-7567f6b9c5-nztb2" Nov 04 04:47:32.121364 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:32.121346 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/50622b62-1450-4af2-80fc-d2f3d8f82d27-tmp\") pod \"console-7567f6b9c5-nztb2\" (UID: \"50622b62-1450-4af2-80fc-d2f3d8f82d27\") " pod="openshift-console/console-7567f6b9c5-nztb2" Nov 04 04:47:32.121504 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:32.121406 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/50622b62-1450-4af2-80fc-d2f3d8f82d27-console-serving-cert\") pod \"console-7567f6b9c5-nztb2\" (UID: \"50622b62-1450-4af2-80fc-d2f3d8f82d27\") " pod="openshift-console/console-7567f6b9c5-nztb2" Nov 04 04:47:32.121504 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:32.121440 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/50622b62-1450-4af2-80fc-d2f3d8f82d27-service-ca\") pod \"console-7567f6b9c5-nztb2\" (UID: \"50622b62-1450-4af2-80fc-d2f3d8f82d27\") " pod="openshift-console/console-7567f6b9c5-nztb2" Nov 04 04:47:32.121504 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:32.121480 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/50622b62-1450-4af2-80fc-d2f3d8f82d27-oauth-serving-cert\") pod \"console-7567f6b9c5-nztb2\" (UID: \"50622b62-1450-4af2-80fc-d2f3d8f82d27\") " pod="openshift-console/console-7567f6b9c5-nztb2" Nov 04 04:47:32.121637 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:32.121515 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlnb5\" (UniqueName: \"kubernetes.io/projected/50622b62-1450-4af2-80fc-d2f3d8f82d27-kube-api-access-nlnb5\") pod \"console-7567f6b9c5-nztb2\" (UID: \"50622b62-1450-4af2-80fc-d2f3d8f82d27\") " pod="openshift-console/console-7567f6b9c5-nztb2" Nov 04 04:47:32.122044 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:32.121918 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/50622b62-1450-4af2-80fc-d2f3d8f82d27-console-config\") pod \"console-7567f6b9c5-nztb2\" (UID: \"50622b62-1450-4af2-80fc-d2f3d8f82d27\") " pod="openshift-console/console-7567f6b9c5-nztb2" Nov 04 04:47:32.122437 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:32.122414 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/50622b62-1450-4af2-80fc-d2f3d8f82d27-service-ca\") pod \"console-7567f6b9c5-nztb2\" (UID: \"50622b62-1450-4af2-80fc-d2f3d8f82d27\") " pod="openshift-console/console-7567f6b9c5-nztb2" Nov 04 04:47:32.122886 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:32.122868 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/50622b62-1450-4af2-80fc-d2f3d8f82d27-oauth-serving-cert\") pod \"console-7567f6b9c5-nztb2\" (UID: \"50622b62-1450-4af2-80fc-d2f3d8f82d27\") " pod="openshift-console/console-7567f6b9c5-nztb2" Nov 04 04:47:32.123150 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:32.123119 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/50622b62-1450-4af2-80fc-d2f3d8f82d27-tmp\") pod \"console-7567f6b9c5-nztb2\" (UID: \"50622b62-1450-4af2-80fc-d2f3d8f82d27\") " pod="openshift-console/console-7567f6b9c5-nztb2" Nov 04 04:47:32.124496 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:32.124475 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/50622b62-1450-4af2-80fc-d2f3d8f82d27-console-serving-cert\") pod \"console-7567f6b9c5-nztb2\" (UID: \"50622b62-1450-4af2-80fc-d2f3d8f82d27\") " pod="openshift-console/console-7567f6b9c5-nztb2" Nov 04 04:47:32.125061 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:32.125040 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/50622b62-1450-4af2-80fc-d2f3d8f82d27-console-oauth-config\") pod \"console-7567f6b9c5-nztb2\" (UID: \"50622b62-1450-4af2-80fc-d2f3d8f82d27\") " pod="openshift-console/console-7567f6b9c5-nztb2" Nov 04 04:47:32.179233 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:32.179202 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlnb5\" (UniqueName: \"kubernetes.io/projected/50622b62-1450-4af2-80fc-d2f3d8f82d27-kube-api-access-nlnb5\") pod \"console-7567f6b9c5-nztb2\" (UID: \"50622b62-1450-4af2-80fc-d2f3d8f82d27\") " pod="openshift-console/console-7567f6b9c5-nztb2" Nov 04 04:47:32.279460 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:32.279377 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7567f6b9c5-nztb2" Nov 04 04:47:32.655186 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:32.655160 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7567f6b9c5-nztb2"] Nov 04 04:47:32.690680 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:32.690653 4509 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50622b62_1450_4af2_80fc_d2f3d8f82d27.slice/crio-a06bc4abd29f912f1c40df9ee4858acca7788369b2e169d2df97c576ed1bd9c9 WatchSource:0}: Error finding container a06bc4abd29f912f1c40df9ee4858acca7788369b2e169d2df97c576ed1bd9c9: Status 404 returned error can't find the container with id a06bc4abd29f912f1c40df9ee4858acca7788369b2e169d2df97c576ed1bd9c9 Nov 04 04:47:33.358475 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:33.358436 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7567f6b9c5-nztb2" event={"ID":"50622b62-1450-4af2-80fc-d2f3d8f82d27","Type":"ContainerStarted","Data":"a06bc4abd29f912f1c40df9ee4858acca7788369b2e169d2df97c576ed1bd9c9"} Nov 04 04:47:33.360173 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:33.360140 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-9k5vv" event={"ID":"b4d26ce7-b503-4c3d-8e1e-02ff73858fbe","Type":"ContainerStarted","Data":"6eea0abc3e594946e974ff6b1a75a5d9b8b93e892dfecf1b5abab01ca548b420"} Nov 04 04:47:33.360490 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:33.360471 4509 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-9k5vv" Nov 04 04:47:33.365742 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:33.365724 4509 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-9k5vv" Nov 04 04:47:33.473404 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:33.473316 4509 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-9k5vv" podStartSLOduration=1.916377078 podStartE2EDuration="3.473299793s" totalImagesPullingTime="1.556922715s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 04:47:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:47:33.473041031 +0000 UTC m=+17.350764557" watchObservedRunningTime="2026-11-04 04:47:33.473299793 +0000 UTC m=+17.351023317" Nov 04 04:47:36.371857 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:36.371778 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7567f6b9c5-nztb2" event={"ID":"50622b62-1450-4af2-80fc-d2f3d8f82d27","Type":"ContainerStarted","Data":"ae73a568e8b235ae34d54d5babea6cb6214bdf6a364b968e218ff745296ef536"} Nov 04 04:47:40.774788 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:40.774723 4509 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-7567f6b9c5-nztb2" podStartSLOduration=6.473599489 podStartE2EDuration="9.774708733s" totalImagesPullingTime="3.301109244s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 04:47:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:47:36.487501042 +0000 UTC m=+20.365224563" watchObservedRunningTime="2026-11-04 04:47:40.774708733 +0000 UTC m=+24.652432254" Nov 04 04:47:40.776629 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:40.775489 4509 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-cxhln"] Nov 04 04:47:40.777813 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:40.777792 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-cxhln" Nov 04 04:47:40.797239 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:40.797217 4509 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-7tw97" Nov 04 04:47:40.884562 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:40.884479 4509 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Nov 04 04:47:40.884562 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:40.884479 4509 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Nov 04 04:47:40.884562 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:40.884541 4509 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-74fzv\"" Nov 04 04:47:40.884562 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:40.884541 4509 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 04 04:47:40.884905 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:40.884545 4509 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 04 04:47:40.884905 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:40.884559 4509 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 04 04:47:40.885560 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:40.885535 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-cxhln"] Nov 04 04:47:40.887093 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:40.887074 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0c773e84-74ce-494f-8322-8d6204654e7f-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-cxhln\" (UID: \"0c773e84-74ce-494f-8322-8d6204654e7f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-cxhln" Nov 04 04:47:40.887223 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:40.887113 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0c773e84-74ce-494f-8322-8d6204654e7f-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-cxhln\" (UID: \"0c773e84-74ce-494f-8322-8d6204654e7f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-cxhln" Nov 04 04:47:40.887223 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:40.887142 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5pd5\" (UniqueName: \"kubernetes.io/projected/0c773e84-74ce-494f-8322-8d6204654e7f-kube-api-access-c5pd5\") pod \"openshift-state-metrics-7bd7489957-cxhln\" (UID: \"0c773e84-74ce-494f-8322-8d6204654e7f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-cxhln" Nov 04 04:47:40.887341 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:40.887287 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0c773e84-74ce-494f-8322-8d6204654e7f-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-cxhln\" (UID: \"0c773e84-74ce-494f-8322-8d6204654e7f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-cxhln" Nov 04 04:47:40.935558 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:40.935526 4509 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-l64n5"] Nov 04 04:47:40.937926 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:40.937905 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-l64n5" Nov 04 04:47:40.988795 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:40.988758 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ca21ea05-78e7-4261-98a4-7025888ef362-sys\") pod \"node-exporter-l64n5\" (UID: \"ca21ea05-78e7-4261-98a4-7025888ef362\") " pod="openshift-monitoring/node-exporter-l64n5" Nov 04 04:47:40.988957 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:40.988812 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ca21ea05-78e7-4261-98a4-7025888ef362-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-l64n5\" (UID: \"ca21ea05-78e7-4261-98a4-7025888ef362\") " pod="openshift-monitoring/node-exporter-l64n5" Nov 04 04:47:40.988957 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:40.988892 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0c773e84-74ce-494f-8322-8d6204654e7f-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-cxhln\" (UID: \"0c773e84-74ce-494f-8322-8d6204654e7f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-cxhln" Nov 04 04:47:40.988957 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:40.988923 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ca21ea05-78e7-4261-98a4-7025888ef362-node-exporter-wtmp\") pod \"node-exporter-l64n5\" (UID: \"ca21ea05-78e7-4261-98a4-7025888ef362\") " pod="openshift-monitoring/node-exporter-l64n5" Nov 04 04:47:40.989093 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:40.988954 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0c773e84-74ce-494f-8322-8d6204654e7f-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-cxhln\" (UID: \"0c773e84-74ce-494f-8322-8d6204654e7f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-cxhln" Nov 04 04:47:40.989093 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:40.988984 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ca21ea05-78e7-4261-98a4-7025888ef362-node-exporter-accelerators-collector-config\") pod \"node-exporter-l64n5\" (UID: \"ca21ea05-78e7-4261-98a4-7025888ef362\") " pod="openshift-monitoring/node-exporter-l64n5" Nov 04 04:47:40.989189 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:40.989153 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ca21ea05-78e7-4261-98a4-7025888ef362-metrics-client-ca\") pod \"node-exporter-l64n5\" (UID: \"ca21ea05-78e7-4261-98a4-7025888ef362\") " pod="openshift-monitoring/node-exporter-l64n5" Nov 04 04:47:40.989239 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:40.989222 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5pd5\" (UniqueName: \"kubernetes.io/projected/0c773e84-74ce-494f-8322-8d6204654e7f-kube-api-access-c5pd5\") pod \"openshift-state-metrics-7bd7489957-cxhln\" (UID: \"0c773e84-74ce-494f-8322-8d6204654e7f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-cxhln" Nov 04 04:47:40.989292 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:40.989254 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ca21ea05-78e7-4261-98a4-7025888ef362-root\") pod \"node-exporter-l64n5\" (UID: \"ca21ea05-78e7-4261-98a4-7025888ef362\") " pod="openshift-monitoring/node-exporter-l64n5" Nov 04 04:47:40.989356 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:40.989292 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0c773e84-74ce-494f-8322-8d6204654e7f-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-cxhln\" (UID: \"0c773e84-74ce-494f-8322-8d6204654e7f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-cxhln" Nov 04 04:47:40.989416 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:40.989351 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ca21ea05-78e7-4261-98a4-7025888ef362-node-exporter-tls\") pod \"node-exporter-l64n5\" (UID: \"ca21ea05-78e7-4261-98a4-7025888ef362\") " pod="openshift-monitoring/node-exporter-l64n5" Nov 04 04:47:40.989466 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:40.989412 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frbl5\" (UniqueName: \"kubernetes.io/projected/ca21ea05-78e7-4261-98a4-7025888ef362-kube-api-access-frbl5\") pod \"node-exporter-l64n5\" (UID: \"ca21ea05-78e7-4261-98a4-7025888ef362\") " pod="openshift-monitoring/node-exporter-l64n5" Nov 04 04:47:40.989591 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:40.989572 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ca21ea05-78e7-4261-98a4-7025888ef362-node-exporter-textfile\") pod \"node-exporter-l64n5\" (UID: \"ca21ea05-78e7-4261-98a4-7025888ef362\") " pod="openshift-monitoring/node-exporter-l64n5" Nov 04 04:47:40.989682 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:40.989660 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0c773e84-74ce-494f-8322-8d6204654e7f-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-cxhln\" (UID: \"0c773e84-74ce-494f-8322-8d6204654e7f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-cxhln" Nov 04 04:47:40.992031 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:40.992005 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0c773e84-74ce-494f-8322-8d6204654e7f-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-cxhln\" (UID: \"0c773e84-74ce-494f-8322-8d6204654e7f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-cxhln" Nov 04 04:47:40.992306 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:40.992280 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0c773e84-74ce-494f-8322-8d6204654e7f-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-cxhln\" (UID: \"0c773e84-74ce-494f-8322-8d6204654e7f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-cxhln" Nov 04 04:47:41.038170 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:41.038097 4509 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-dxmws\"" Nov 04 04:47:41.038170 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:41.038097 4509 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 04 04:47:41.038369 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:41.038190 4509 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 04 04:47:41.045522 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:41.045481 4509 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 04 04:47:41.050671 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:41.050642 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5pd5\" (UniqueName: \"kubernetes.io/projected/0c773e84-74ce-494f-8322-8d6204654e7f-kube-api-access-c5pd5\") pod \"openshift-state-metrics-7bd7489957-cxhln\" (UID: \"0c773e84-74ce-494f-8322-8d6204654e7f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-cxhln" Nov 04 04:47:41.090335 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:41.090297 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ca21ea05-78e7-4261-98a4-7025888ef362-node-exporter-wtmp\") pod \"node-exporter-l64n5\" (UID: \"ca21ea05-78e7-4261-98a4-7025888ef362\") " pod="openshift-monitoring/node-exporter-l64n5" Nov 04 04:47:41.090508 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:41.090353 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ca21ea05-78e7-4261-98a4-7025888ef362-node-exporter-accelerators-collector-config\") pod \"node-exporter-l64n5\" (UID: \"ca21ea05-78e7-4261-98a4-7025888ef362\") " pod="openshift-monitoring/node-exporter-l64n5" Nov 04 04:47:41.090508 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:41.090378 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ca21ea05-78e7-4261-98a4-7025888ef362-metrics-client-ca\") pod \"node-exporter-l64n5\" (UID: \"ca21ea05-78e7-4261-98a4-7025888ef362\") " pod="openshift-monitoring/node-exporter-l64n5" Nov 04 04:47:41.090508 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:41.090398 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ca21ea05-78e7-4261-98a4-7025888ef362-root\") pod \"node-exporter-l64n5\" (UID: \"ca21ea05-78e7-4261-98a4-7025888ef362\") " pod="openshift-monitoring/node-exporter-l64n5" Nov 04 04:47:41.090508 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:41.090476 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ca21ea05-78e7-4261-98a4-7025888ef362-node-exporter-wtmp\") pod \"node-exporter-l64n5\" (UID: \"ca21ea05-78e7-4261-98a4-7025888ef362\") " pod="openshift-monitoring/node-exporter-l64n5" Nov 04 04:47:41.090722 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:41.090534 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ca21ea05-78e7-4261-98a4-7025888ef362-node-exporter-tls\") pod \"node-exporter-l64n5\" (UID: \"ca21ea05-78e7-4261-98a4-7025888ef362\") " pod="openshift-monitoring/node-exporter-l64n5" Nov 04 04:47:41.090722 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:41.090560 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ca21ea05-78e7-4261-98a4-7025888ef362-root\") pod \"node-exporter-l64n5\" (UID: \"ca21ea05-78e7-4261-98a4-7025888ef362\") " pod="openshift-monitoring/node-exporter-l64n5" Nov 04 04:47:41.090722 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:41.090636 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frbl5\" (UniqueName: \"kubernetes.io/projected/ca21ea05-78e7-4261-98a4-7025888ef362-kube-api-access-frbl5\") pod \"node-exporter-l64n5\" (UID: \"ca21ea05-78e7-4261-98a4-7025888ef362\") " pod="openshift-monitoring/node-exporter-l64n5" Nov 04 04:47:41.090722 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:41.090669 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ca21ea05-78e7-4261-98a4-7025888ef362-node-exporter-textfile\") pod \"node-exporter-l64n5\" (UID: \"ca21ea05-78e7-4261-98a4-7025888ef362\") " pod="openshift-monitoring/node-exporter-l64n5" Nov 04 04:47:41.090722 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:41.090713 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ca21ea05-78e7-4261-98a4-7025888ef362-sys\") pod \"node-exporter-l64n5\" (UID: \"ca21ea05-78e7-4261-98a4-7025888ef362\") " pod="openshift-monitoring/node-exporter-l64n5" Nov 04 04:47:41.090950 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:41.090746 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ca21ea05-78e7-4261-98a4-7025888ef362-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-l64n5\" (UID: \"ca21ea05-78e7-4261-98a4-7025888ef362\") " pod="openshift-monitoring/node-exporter-l64n5" Nov 04 04:47:41.091003 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:41.090979 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ca21ea05-78e7-4261-98a4-7025888ef362-sys\") pod \"node-exporter-l64n5\" (UID: \"ca21ea05-78e7-4261-98a4-7025888ef362\") " pod="openshift-monitoring/node-exporter-l64n5" Nov 04 04:47:41.091080 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:41.091065 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-cxhln" Nov 04 04:47:41.091575 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:41.091551 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ca21ea05-78e7-4261-98a4-7025888ef362-node-exporter-accelerators-collector-config\") pod \"node-exporter-l64n5\" (UID: \"ca21ea05-78e7-4261-98a4-7025888ef362\") " pod="openshift-monitoring/node-exporter-l64n5" Nov 04 04:47:41.091760 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:41.091736 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ca21ea05-78e7-4261-98a4-7025888ef362-node-exporter-textfile\") pod \"node-exporter-l64n5\" (UID: \"ca21ea05-78e7-4261-98a4-7025888ef362\") " pod="openshift-monitoring/node-exporter-l64n5" Nov 04 04:47:41.092009 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:41.091986 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ca21ea05-78e7-4261-98a4-7025888ef362-metrics-client-ca\") pod \"node-exporter-l64n5\" (UID: \"ca21ea05-78e7-4261-98a4-7025888ef362\") " pod="openshift-monitoring/node-exporter-l64n5" Nov 04 04:47:41.093756 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:41.093732 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ca21ea05-78e7-4261-98a4-7025888ef362-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-l64n5\" (UID: \"ca21ea05-78e7-4261-98a4-7025888ef362\") " pod="openshift-monitoring/node-exporter-l64n5" Nov 04 04:47:41.093853 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:41.093735 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ca21ea05-78e7-4261-98a4-7025888ef362-node-exporter-tls\") pod \"node-exporter-l64n5\" (UID: \"ca21ea05-78e7-4261-98a4-7025888ef362\") " pod="openshift-monitoring/node-exporter-l64n5" Nov 04 04:47:41.156537 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:41.154409 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frbl5\" (UniqueName: \"kubernetes.io/projected/ca21ea05-78e7-4261-98a4-7025888ef362-kube-api-access-frbl5\") pod \"node-exporter-l64n5\" (UID: \"ca21ea05-78e7-4261-98a4-7025888ef362\") " pod="openshift-monitoring/node-exporter-l64n5" Nov 04 04:47:41.251838 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:41.251807 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-l64n5" Nov 04 04:47:41.263185 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:41.263155 4509 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca21ea05_78e7_4261_98a4_7025888ef362.slice/crio-8bb6679fda600ffee85ef4a616c2ce0f16242d0d1a311cb68f3f0a066f3daabc WatchSource:0}: Error finding container 8bb6679fda600ffee85ef4a616c2ce0f16242d0d1a311cb68f3f0a066f3daabc: Status 404 returned error can't find the container with id 8bb6679fda600ffee85ef4a616c2ce0f16242d0d1a311cb68f3f0a066f3daabc Nov 04 04:47:41.392453 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:41.392339 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-l64n5" event={"ID":"ca21ea05-78e7-4261-98a4-7025888ef362","Type":"ContainerStarted","Data":"8bb6679fda600ffee85ef4a616c2ce0f16242d0d1a311cb68f3f0a066f3daabc"} Nov 04 04:47:41.517060 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:41.517023 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-cxhln"] Nov 04 04:47:41.520202 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:41.520169 4509 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c773e84_74ce_494f_8322_8d6204654e7f.slice/crio-2a7b858949f01bbff973b5c5379e463895e2554bccc907d2a77b796c75cee4b9 WatchSource:0}: Error finding container 2a7b858949f01bbff973b5c5379e463895e2554bccc907d2a77b796c75cee4b9: Status 404 returned error can't find the container with id 2a7b858949f01bbff973b5c5379e463895e2554bccc907d2a77b796c75cee4b9 Nov 04 04:47:42.280081 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:42.280046 4509 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7567f6b9c5-nztb2" Nov 04 04:47:42.280591 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:42.280118 4509 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7567f6b9c5-nztb2" Nov 04 04:47:42.286502 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:42.286463 4509 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7567f6b9c5-nztb2" Nov 04 04:47:42.398717 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:42.398678 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-cxhln" event={"ID":"0c773e84-74ce-494f-8322-8d6204654e7f","Type":"ContainerStarted","Data":"4e44d7a3a273561da387ad1feb8c619a85de23e57654e630cd72d1b55324d8a0"} Nov 04 04:47:42.398894 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:42.398756 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-cxhln" event={"ID":"0c773e84-74ce-494f-8322-8d6204654e7f","Type":"ContainerStarted","Data":"71f2a1155f2e23526a24fcc80201d45dd455d08427c56fbe1a4c8efbc17bbf8d"} Nov 04 04:47:42.398894 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:42.398787 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-cxhln" event={"ID":"0c773e84-74ce-494f-8322-8d6204654e7f","Type":"ContainerStarted","Data":"2a7b858949f01bbff973b5c5379e463895e2554bccc907d2a77b796c75cee4b9"} Nov 04 04:47:42.403108 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:42.403083 4509 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7567f6b9c5-nztb2" Nov 04 04:47:43.405053 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:43.404961 4509 generic.go:356] "Generic (PLEG): container finished" podID="ca21ea05-78e7-4261-98a4-7025888ef362" containerID="1d5b8048b019092daa4ec9da0eedb752ace9fd135de7e0938e1443774f4f32de" exitCode=0 Nov 04 04:47:43.405526 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:43.405057 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-l64n5" event={"ID":"ca21ea05-78e7-4261-98a4-7025888ef362","Type":"ContainerDied","Data":"1d5b8048b019092daa4ec9da0eedb752ace9fd135de7e0938e1443774f4f32de"} Nov 04 04:47:44.411092 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:44.411058 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-l64n5" event={"ID":"ca21ea05-78e7-4261-98a4-7025888ef362","Type":"ContainerStarted","Data":"3b79061457eea808addc81b9d7a782d9e5641db148306e37b5c5a5c255e65ccf"} Nov 04 04:47:44.411617 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:44.411117 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-l64n5" event={"ID":"ca21ea05-78e7-4261-98a4-7025888ef362","Type":"ContainerStarted","Data":"37f15ae130a1d99251bd65a53d0c16b9e96383a10add7cdd2a3b7ac9b3fdadc3"} Nov 04 04:47:44.413676 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:44.413642 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-cxhln" event={"ID":"0c773e84-74ce-494f-8322-8d6204654e7f","Type":"ContainerStarted","Data":"b681a1cde3d1c97120d7f44dc5e63339e136bac0896bba0fe62e897f8a6339e9"} Nov 04 04:47:44.455166 ip-10-0-133-180 kubenswrapper[4509]: E1104 04:47:44.455128 4509 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmdhl" containerName="extractor" Nov 04 04:47:44.528189 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:44.528142 4509 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-l64n5" podStartSLOduration=2.653558449 podStartE2EDuration="4.528129159s" totalImagesPullingTime="1.778174706s" totalInitContainerRuntime="96.396004ms" isStatefulPod=true podCreationTimestamp="2026-11-04 04:47:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:47:44.527296109 +0000 UTC m=+28.405019630" watchObservedRunningTime="2026-11-04 04:47:44.528129159 +0000 UTC m=+28.405852681" Nov 04 04:47:44.647860 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:44.645255 4509 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-cxhln" podStartSLOduration=2.845376522 podStartE2EDuration="4.64523839s" totalImagesPullingTime="1.799861868s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 04:47:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:47:44.642920501 +0000 UTC m=+28.520644023" watchObservedRunningTime="2026-11-04 04:47:44.64523839 +0000 UTC m=+28.522961914" Nov 04 04:47:45.570121 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:45.570090 4509 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-957r8"] Nov 04 04:47:45.595634 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:45.595595 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-957r8" Nov 04 04:47:45.629048 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:45.629014 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-957r8"] Nov 04 04:47:45.678113 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:45.678082 4509 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-n7p2b\"" Nov 04 04:47:45.692365 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:45.692338 4509 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Nov 04 04:47:45.726024 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:45.725982 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/841efaa4-fa65-4d0e-887f-7390c27db3f5-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-957r8\" (UID: \"841efaa4-fa65-4d0e-887f-7390c27db3f5\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-957r8" Nov 04 04:47:45.726181 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:45.726059 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwpww\" (UniqueName: \"kubernetes.io/projected/841efaa4-fa65-4d0e-887f-7390c27db3f5-kube-api-access-nwpww\") pod \"monitoring-plugin-77f4f76594-957r8\" (UID: \"841efaa4-fa65-4d0e-887f-7390c27db3f5\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-957r8" Nov 04 04:47:45.827690 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:45.827608 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwpww\" (UniqueName: \"kubernetes.io/projected/841efaa4-fa65-4d0e-887f-7390c27db3f5-kube-api-access-nwpww\") pod \"monitoring-plugin-77f4f76594-957r8\" (UID: \"841efaa4-fa65-4d0e-887f-7390c27db3f5\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-957r8" Nov 04 04:47:45.827690 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:45.827685 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/841efaa4-fa65-4d0e-887f-7390c27db3f5-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-957r8\" (UID: \"841efaa4-fa65-4d0e-887f-7390c27db3f5\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-957r8" Nov 04 04:47:45.830892 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:45.830866 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/841efaa4-fa65-4d0e-887f-7390c27db3f5-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-957r8\" (UID: \"841efaa4-fa65-4d0e-887f-7390c27db3f5\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-957r8" Nov 04 04:47:45.885356 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:45.885302 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwpww\" (UniqueName: \"kubernetes.io/projected/841efaa4-fa65-4d0e-887f-7390c27db3f5-kube-api-access-nwpww\") pod \"monitoring-plugin-77f4f76594-957r8\" (UID: \"841efaa4-fa65-4d0e-887f-7390c27db3f5\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-957r8" Nov 04 04:47:45.920336 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:45.920293 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-957r8" Nov 04 04:47:46.308526 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:46.308492 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-957r8"] Nov 04 04:47:46.422572 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:46.422536 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-957r8" event={"ID":"841efaa4-fa65-4d0e-887f-7390c27db3f5","Type":"ContainerStarted","Data":"f8a4b139f2efa4d895e8b1a9eebf9e5e47426b9ecd6cbf416152bf8b32c1f578"} Nov 04 04:47:46.668198 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:46.668126 4509 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-c656644f9-tk9wq"] Nov 04 04:47:46.679856 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:46.679832 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-c656644f9-tk9wq" Nov 04 04:47:46.682262 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:46.682204 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-c656644f9-tk9wq"] Nov 04 04:47:46.732813 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:46.732782 4509 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-2vp9t\"" Nov 04 04:47:46.732813 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:46.732783 4509 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Nov 04 04:47:46.733027 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:46.732785 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/67b9e6c7-278b-47dc-bad0-72d05c2c6427-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-c656644f9-tk9wq\" (UID: \"67b9e6c7-278b-47dc-bad0-72d05c2c6427\") " pod="openshift-monitoring/telemeter-client-c656644f9-tk9wq" Nov 04 04:47:46.733027 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:46.732904 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/67b9e6c7-278b-47dc-bad0-72d05c2c6427-serving-certs-ca-bundle\") pod \"telemeter-client-c656644f9-tk9wq\" (UID: \"67b9e6c7-278b-47dc-bad0-72d05c2c6427\") " pod="openshift-monitoring/telemeter-client-c656644f9-tk9wq" Nov 04 04:47:46.733027 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:46.732939 4509 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Nov 04 04:47:46.733027 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:46.732938 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/67b9e6c7-278b-47dc-bad0-72d05c2c6427-federate-client-tls\") pod \"telemeter-client-c656644f9-tk9wq\" (UID: \"67b9e6c7-278b-47dc-bad0-72d05c2c6427\") " pod="openshift-monitoring/telemeter-client-c656644f9-tk9wq" Nov 04 04:47:46.733027 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:46.733014 4509 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Nov 04 04:47:46.733027 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:46.733017 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/67b9e6c7-278b-47dc-bad0-72d05c2c6427-telemeter-trusted-ca-bundle\") pod \"telemeter-client-c656644f9-tk9wq\" (UID: \"67b9e6c7-278b-47dc-bad0-72d05c2c6427\") " pod="openshift-monitoring/telemeter-client-c656644f9-tk9wq" Nov 04 04:47:46.733252 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:46.733056 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/67b9e6c7-278b-47dc-bad0-72d05c2c6427-metrics-client-ca\") pod \"telemeter-client-c656644f9-tk9wq\" (UID: \"67b9e6c7-278b-47dc-bad0-72d05c2c6427\") " pod="openshift-monitoring/telemeter-client-c656644f9-tk9wq" Nov 04 04:47:46.733252 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:46.733087 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/67b9e6c7-278b-47dc-bad0-72d05c2c6427-telemeter-client-tls\") pod \"telemeter-client-c656644f9-tk9wq\" (UID: \"67b9e6c7-278b-47dc-bad0-72d05c2c6427\") " pod="openshift-monitoring/telemeter-client-c656644f9-tk9wq" Nov 04 04:47:46.733252 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:46.733183 4509 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Nov 04 04:47:46.733252 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:46.733194 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/67b9e6c7-278b-47dc-bad0-72d05c2c6427-secret-telemeter-client\") pod \"telemeter-client-c656644f9-tk9wq\" (UID: \"67b9e6c7-278b-47dc-bad0-72d05c2c6427\") " pod="openshift-monitoring/telemeter-client-c656644f9-tk9wq" Nov 04 04:47:46.733252 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:46.733223 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kq9vb\" (UniqueName: \"kubernetes.io/projected/67b9e6c7-278b-47dc-bad0-72d05c2c6427-kube-api-access-kq9vb\") pod \"telemeter-client-c656644f9-tk9wq\" (UID: \"67b9e6c7-278b-47dc-bad0-72d05c2c6427\") " pod="openshift-monitoring/telemeter-client-c656644f9-tk9wq" Nov 04 04:47:46.733440 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:46.733297 4509 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Nov 04 04:47:46.833606 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:46.833570 4509 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\"" Nov 04 04:47:46.833800 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:46.833657 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/67b9e6c7-278b-47dc-bad0-72d05c2c6427-telemeter-client-tls\") pod \"telemeter-client-c656644f9-tk9wq\" (UID: \"67b9e6c7-278b-47dc-bad0-72d05c2c6427\") " pod="openshift-monitoring/telemeter-client-c656644f9-tk9wq" Nov 04 04:47:46.833800 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:46.833731 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/67b9e6c7-278b-47dc-bad0-72d05c2c6427-secret-telemeter-client\") pod \"telemeter-client-c656644f9-tk9wq\" (UID: \"67b9e6c7-278b-47dc-bad0-72d05c2c6427\") " pod="openshift-monitoring/telemeter-client-c656644f9-tk9wq" Nov 04 04:47:46.833800 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:46.833770 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kq9vb\" (UniqueName: \"kubernetes.io/projected/67b9e6c7-278b-47dc-bad0-72d05c2c6427-kube-api-access-kq9vb\") pod \"telemeter-client-c656644f9-tk9wq\" (UID: \"67b9e6c7-278b-47dc-bad0-72d05c2c6427\") " pod="openshift-monitoring/telemeter-client-c656644f9-tk9wq" Nov 04 04:47:46.833949 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:46.833818 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/67b9e6c7-278b-47dc-bad0-72d05c2c6427-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-c656644f9-tk9wq\" (UID: \"67b9e6c7-278b-47dc-bad0-72d05c2c6427\") " pod="openshift-monitoring/telemeter-client-c656644f9-tk9wq" Nov 04 04:47:46.833949 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:46.833855 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/67b9e6c7-278b-47dc-bad0-72d05c2c6427-serving-certs-ca-bundle\") pod \"telemeter-client-c656644f9-tk9wq\" (UID: \"67b9e6c7-278b-47dc-bad0-72d05c2c6427\") " pod="openshift-monitoring/telemeter-client-c656644f9-tk9wq" Nov 04 04:47:46.833949 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:46.833880 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/67b9e6c7-278b-47dc-bad0-72d05c2c6427-federate-client-tls\") pod \"telemeter-client-c656644f9-tk9wq\" (UID: \"67b9e6c7-278b-47dc-bad0-72d05c2c6427\") " pod="openshift-monitoring/telemeter-client-c656644f9-tk9wq" Nov 04 04:47:46.833949 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:46.833912 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/67b9e6c7-278b-47dc-bad0-72d05c2c6427-telemeter-trusted-ca-bundle\") pod \"telemeter-client-c656644f9-tk9wq\" (UID: \"67b9e6c7-278b-47dc-bad0-72d05c2c6427\") " pod="openshift-monitoring/telemeter-client-c656644f9-tk9wq" Nov 04 04:47:46.834090 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:46.833954 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/67b9e6c7-278b-47dc-bad0-72d05c2c6427-metrics-client-ca\") pod \"telemeter-client-c656644f9-tk9wq\" (UID: \"67b9e6c7-278b-47dc-bad0-72d05c2c6427\") " pod="openshift-monitoring/telemeter-client-c656644f9-tk9wq" Nov 04 04:47:46.834650 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:46.834626 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/67b9e6c7-278b-47dc-bad0-72d05c2c6427-metrics-client-ca\") pod \"telemeter-client-c656644f9-tk9wq\" (UID: \"67b9e6c7-278b-47dc-bad0-72d05c2c6427\") " pod="openshift-monitoring/telemeter-client-c656644f9-tk9wq" Nov 04 04:47:46.835115 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:46.835095 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/67b9e6c7-278b-47dc-bad0-72d05c2c6427-serving-certs-ca-bundle\") pod \"telemeter-client-c656644f9-tk9wq\" (UID: \"67b9e6c7-278b-47dc-bad0-72d05c2c6427\") " pod="openshift-monitoring/telemeter-client-c656644f9-tk9wq" Nov 04 04:47:46.835735 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:46.835711 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/67b9e6c7-278b-47dc-bad0-72d05c2c6427-telemeter-trusted-ca-bundle\") pod \"telemeter-client-c656644f9-tk9wq\" (UID: \"67b9e6c7-278b-47dc-bad0-72d05c2c6427\") " pod="openshift-monitoring/telemeter-client-c656644f9-tk9wq" Nov 04 04:47:46.836750 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:46.836724 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/67b9e6c7-278b-47dc-bad0-72d05c2c6427-telemeter-client-tls\") pod \"telemeter-client-c656644f9-tk9wq\" (UID: \"67b9e6c7-278b-47dc-bad0-72d05c2c6427\") " pod="openshift-monitoring/telemeter-client-c656644f9-tk9wq" Nov 04 04:47:46.837612 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:46.837569 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/67b9e6c7-278b-47dc-bad0-72d05c2c6427-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-c656644f9-tk9wq\" (UID: \"67b9e6c7-278b-47dc-bad0-72d05c2c6427\") " pod="openshift-monitoring/telemeter-client-c656644f9-tk9wq" Nov 04 04:47:46.838278 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:46.838239 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/67b9e6c7-278b-47dc-bad0-72d05c2c6427-secret-telemeter-client\") pod \"telemeter-client-c656644f9-tk9wq\" (UID: \"67b9e6c7-278b-47dc-bad0-72d05c2c6427\") " pod="openshift-monitoring/telemeter-client-c656644f9-tk9wq" Nov 04 04:47:46.838632 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:46.838614 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/67b9e6c7-278b-47dc-bad0-72d05c2c6427-federate-client-tls\") pod \"telemeter-client-c656644f9-tk9wq\" (UID: \"67b9e6c7-278b-47dc-bad0-72d05c2c6427\") " pod="openshift-monitoring/telemeter-client-c656644f9-tk9wq" Nov 04 04:47:46.892514 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:46.892481 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kq9vb\" (UniqueName: \"kubernetes.io/projected/67b9e6c7-278b-47dc-bad0-72d05c2c6427-kube-api-access-kq9vb\") pod \"telemeter-client-c656644f9-tk9wq\" (UID: \"67b9e6c7-278b-47dc-bad0-72d05c2c6427\") " pod="openshift-monitoring/telemeter-client-c656644f9-tk9wq" Nov 04 04:47:47.021867 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:47.021830 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-c656644f9-tk9wq" Nov 04 04:47:47.521147 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:47.521111 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-c656644f9-tk9wq"] Nov 04 04:47:47.524617 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:47.524587 4509 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67b9e6c7_278b_47dc_bad0_72d05c2c6427.slice/crio-c47989203eea0536263e0fcb4cd9aeb3817b7c44a66bca2d253242a89ab41814 WatchSource:0}: Error finding container c47989203eea0536263e0fcb4cd9aeb3817b7c44a66bca2d253242a89ab41814: Status 404 returned error can't find the container with id c47989203eea0536263e0fcb4cd9aeb3817b7c44a66bca2d253242a89ab41814 Nov 04 04:47:48.431733 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:48.431646 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-c656644f9-tk9wq" event={"ID":"67b9e6c7-278b-47dc-bad0-72d05c2c6427","Type":"ContainerStarted","Data":"c47989203eea0536263e0fcb4cd9aeb3817b7c44a66bca2d253242a89ab41814"} Nov 04 04:47:48.433610 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:48.433581 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-kllmc" event={"ID":"50a99580-f574-461f-95e2-f6bc901f77ce","Type":"ContainerStarted","Data":"d29894418b80d8eb499a406d83bc7bdafafe67a0adc0bac8b32063b903b1eb53"} Nov 04 04:47:48.433878 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:48.433814 4509 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-kllmc" Nov 04 04:47:48.435136 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:48.435109 4509 patch_prober.go:29] interesting pod/downloads-6dd789c8b7-kllmc container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.133.0.10:8080/\": dial tcp 10.133.0.10:8080: connect: connection refused" start-of-body= Nov 04 04:47:48.435306 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:48.435164 4509 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-6dd789c8b7-kllmc" podUID="50a99580-f574-461f-95e2-f6bc901f77ce" containerName="download-server" probeResult="failure" output="Get \"http://10.133.0.10:8080/\": dial tcp 10.133.0.10:8080: connect: connection refused" Nov 04 04:47:48.550447 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:48.550380 4509 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-kllmc" podStartSLOduration=1.584521848 podStartE2EDuration="18.550361298s" totalImagesPullingTime="16.96583945s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 04:47:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:47:48.548923771 +0000 UTC m=+32.426647295" watchObservedRunningTime="2026-11-04 04:47:48.550361298 +0000 UTC m=+32.428084821" Nov 04 04:47:49.442276 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:49.442227 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-957r8" event={"ID":"841efaa4-fa65-4d0e-887f-7390c27db3f5","Type":"ContainerStarted","Data":"338011248c4af4c996b3a08a66750848b625851fa0c04ab636fd5367049c69c0"} Nov 04 04:47:49.443091 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:49.442833 4509 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-957r8" Nov 04 04:47:49.448086 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:49.448061 4509 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-957r8" Nov 04 04:47:49.456887 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:49.456866 4509 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-kllmc" Nov 04 04:47:49.556363 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:49.556282 4509 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-957r8" podStartSLOduration=2.281398087 podStartE2EDuration="4.556261357s" totalImagesPullingTime="2.27486327s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 04:47:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:47:49.554941699 +0000 UTC m=+33.432665220" watchObservedRunningTime="2026-11-04 04:47:49.556261357 +0000 UTC m=+33.433984879" Nov 04 04:47:50.459526 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:50.459493 4509 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-8hsp8" Nov 04 04:47:51.451490 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:51.451446 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-c656644f9-tk9wq" event={"ID":"67b9e6c7-278b-47dc-bad0-72d05c2c6427","Type":"ContainerStarted","Data":"0b2eaad50c284d2a79eb8adf491b1324301948e409967fbc288cb20208e0b583"} Nov 04 04:47:51.768400 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:51.768361 4509 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-648cd6c647-qqx85"] Nov 04 04:47:51.790001 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:51.789972 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-648cd6c647-qqx85" Nov 04 04:47:51.832025 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:51.831991 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-648cd6c647-qqx85"] Nov 04 04:47:51.881890 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:51.881848 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6c43eb65-2efa-445a-9cba-69be6820ed92-service-ca\") pod \"console-648cd6c647-qqx85\" (UID: \"6c43eb65-2efa-445a-9cba-69be6820ed92\") " pod="openshift-console/console-648cd6c647-qqx85" Nov 04 04:47:51.882075 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:51.881925 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6c43eb65-2efa-445a-9cba-69be6820ed92-trusted-ca-bundle\") pod \"console-648cd6c647-qqx85\" (UID: \"6c43eb65-2efa-445a-9cba-69be6820ed92\") " pod="openshift-console/console-648cd6c647-qqx85" Nov 04 04:47:51.882075 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:51.881998 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6c43eb65-2efa-445a-9cba-69be6820ed92-console-serving-cert\") pod \"console-648cd6c647-qqx85\" (UID: \"6c43eb65-2efa-445a-9cba-69be6820ed92\") " pod="openshift-console/console-648cd6c647-qqx85" Nov 04 04:47:51.882075 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:51.882026 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6c43eb65-2efa-445a-9cba-69be6820ed92-console-config\") pod \"console-648cd6c647-qqx85\" (UID: \"6c43eb65-2efa-445a-9cba-69be6820ed92\") " pod="openshift-console/console-648cd6c647-qqx85" Nov 04 04:47:51.882226 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:51.882114 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6c43eb65-2efa-445a-9cba-69be6820ed92-oauth-serving-cert\") pod \"console-648cd6c647-qqx85\" (UID: \"6c43eb65-2efa-445a-9cba-69be6820ed92\") " pod="openshift-console/console-648cd6c647-qqx85" Nov 04 04:47:51.882275 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:51.882256 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6c43eb65-2efa-445a-9cba-69be6820ed92-tmp\") pod \"console-648cd6c647-qqx85\" (UID: \"6c43eb65-2efa-445a-9cba-69be6820ed92\") " pod="openshift-console/console-648cd6c647-qqx85" Nov 04 04:47:51.882342 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:51.882308 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6c43eb65-2efa-445a-9cba-69be6820ed92-console-oauth-config\") pod \"console-648cd6c647-qqx85\" (UID: \"6c43eb65-2efa-445a-9cba-69be6820ed92\") " pod="openshift-console/console-648cd6c647-qqx85" Nov 04 04:47:51.882415 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:51.882397 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfwvz\" (UniqueName: \"kubernetes.io/projected/6c43eb65-2efa-445a-9cba-69be6820ed92-kube-api-access-dfwvz\") pod \"console-648cd6c647-qqx85\" (UID: \"6c43eb65-2efa-445a-9cba-69be6820ed92\") " pod="openshift-console/console-648cd6c647-qqx85" Nov 04 04:47:51.983284 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:51.983246 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6c43eb65-2efa-445a-9cba-69be6820ed92-console-oauth-config\") pod \"console-648cd6c647-qqx85\" (UID: \"6c43eb65-2efa-445a-9cba-69be6820ed92\") " pod="openshift-console/console-648cd6c647-qqx85" Nov 04 04:47:51.983512 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:51.983304 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfwvz\" (UniqueName: \"kubernetes.io/projected/6c43eb65-2efa-445a-9cba-69be6820ed92-kube-api-access-dfwvz\") pod \"console-648cd6c647-qqx85\" (UID: \"6c43eb65-2efa-445a-9cba-69be6820ed92\") " pod="openshift-console/console-648cd6c647-qqx85" Nov 04 04:47:51.983512 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:51.983367 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6c43eb65-2efa-445a-9cba-69be6820ed92-service-ca\") pod \"console-648cd6c647-qqx85\" (UID: \"6c43eb65-2efa-445a-9cba-69be6820ed92\") " pod="openshift-console/console-648cd6c647-qqx85" Nov 04 04:47:51.983512 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:51.983383 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6c43eb65-2efa-445a-9cba-69be6820ed92-trusted-ca-bundle\") pod \"console-648cd6c647-qqx85\" (UID: \"6c43eb65-2efa-445a-9cba-69be6820ed92\") " pod="openshift-console/console-648cd6c647-qqx85" Nov 04 04:47:51.983512 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:51.983410 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6c43eb65-2efa-445a-9cba-69be6820ed92-console-serving-cert\") pod \"console-648cd6c647-qqx85\" (UID: \"6c43eb65-2efa-445a-9cba-69be6820ed92\") " pod="openshift-console/console-648cd6c647-qqx85" Nov 04 04:47:51.983512 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:51.983432 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6c43eb65-2efa-445a-9cba-69be6820ed92-console-config\") pod \"console-648cd6c647-qqx85\" (UID: \"6c43eb65-2efa-445a-9cba-69be6820ed92\") " pod="openshift-console/console-648cd6c647-qqx85" Nov 04 04:47:51.983512 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:51.983463 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6c43eb65-2efa-445a-9cba-69be6820ed92-oauth-serving-cert\") pod \"console-648cd6c647-qqx85\" (UID: \"6c43eb65-2efa-445a-9cba-69be6820ed92\") " pod="openshift-console/console-648cd6c647-qqx85" Nov 04 04:47:51.983512 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:51.983493 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6c43eb65-2efa-445a-9cba-69be6820ed92-tmp\") pod \"console-648cd6c647-qqx85\" (UID: \"6c43eb65-2efa-445a-9cba-69be6820ed92\") " pod="openshift-console/console-648cd6c647-qqx85" Nov 04 04:47:51.984147 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:51.984117 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6c43eb65-2efa-445a-9cba-69be6820ed92-tmp\") pod \"console-648cd6c647-qqx85\" (UID: \"6c43eb65-2efa-445a-9cba-69be6820ed92\") " pod="openshift-console/console-648cd6c647-qqx85" Nov 04 04:47:51.984836 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:51.984785 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6c43eb65-2efa-445a-9cba-69be6820ed92-console-config\") pod \"console-648cd6c647-qqx85\" (UID: \"6c43eb65-2efa-445a-9cba-69be6820ed92\") " pod="openshift-console/console-648cd6c647-qqx85" Nov 04 04:47:51.984990 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:51.984969 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6c43eb65-2efa-445a-9cba-69be6820ed92-oauth-serving-cert\") pod \"console-648cd6c647-qqx85\" (UID: \"6c43eb65-2efa-445a-9cba-69be6820ed92\") " pod="openshift-console/console-648cd6c647-qqx85" Nov 04 04:47:51.985437 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:51.985392 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6c43eb65-2efa-445a-9cba-69be6820ed92-service-ca\") pod \"console-648cd6c647-qqx85\" (UID: \"6c43eb65-2efa-445a-9cba-69be6820ed92\") " pod="openshift-console/console-648cd6c647-qqx85" Nov 04 04:47:51.995100 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:51.986983 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6c43eb65-2efa-445a-9cba-69be6820ed92-console-serving-cert\") pod \"console-648cd6c647-qqx85\" (UID: \"6c43eb65-2efa-445a-9cba-69be6820ed92\") " pod="openshift-console/console-648cd6c647-qqx85" Nov 04 04:47:51.995100 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:51.987355 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6c43eb65-2efa-445a-9cba-69be6820ed92-console-oauth-config\") pod \"console-648cd6c647-qqx85\" (UID: \"6c43eb65-2efa-445a-9cba-69be6820ed92\") " pod="openshift-console/console-648cd6c647-qqx85" Nov 04 04:47:51.995100 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:51.993636 4509 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Nov 04 04:47:51.995751 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:51.995727 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6c43eb65-2efa-445a-9cba-69be6820ed92-trusted-ca-bundle\") pod \"console-648cd6c647-qqx85\" (UID: \"6c43eb65-2efa-445a-9cba-69be6820ed92\") " pod="openshift-console/console-648cd6c647-qqx85" Nov 04 04:47:52.042855 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:52.042755 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfwvz\" (UniqueName: \"kubernetes.io/projected/6c43eb65-2efa-445a-9cba-69be6820ed92-kube-api-access-dfwvz\") pod \"console-648cd6c647-qqx85\" (UID: \"6c43eb65-2efa-445a-9cba-69be6820ed92\") " pod="openshift-console/console-648cd6c647-qqx85" Nov 04 04:47:52.122572 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:52.122278 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-648cd6c647-qqx85" Nov 04 04:47:52.358536 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:52.358473 4509 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5dd6d64c77-mpvgf" Nov 04 04:47:52.532413 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:52.532368 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-648cd6c647-qqx85"] Nov 04 04:47:52.707112 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:52.707058 4509 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c43eb65_2efa_445a_9cba_69be6820ed92.slice/crio-978cc4b3c9596f868df9a8682c3e983ead9bb65f1b9283f29687923f3ad64cae WatchSource:0}: Error finding container 978cc4b3c9596f868df9a8682c3e983ead9bb65f1b9283f29687923f3ad64cae: Status 404 returned error can't find the container with id 978cc4b3c9596f868df9a8682c3e983ead9bb65f1b9283f29687923f3ad64cae Nov 04 04:47:53.461639 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:53.461601 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-c656644f9-tk9wq" event={"ID":"67b9e6c7-278b-47dc-bad0-72d05c2c6427","Type":"ContainerStarted","Data":"324c6032181a2153b8dcab8f337eb5fd2a27418a4aa3e4861d56cc619805b709"} Nov 04 04:47:53.462196 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:53.461673 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-c656644f9-tk9wq" event={"ID":"67b9e6c7-278b-47dc-bad0-72d05c2c6427","Type":"ContainerStarted","Data":"267f23ef1869be8682f34f2f1b35d3bedd945cf0f1f38773b578a3703bf5470d"} Nov 04 04:47:53.463591 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:53.463563 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-648cd6c647-qqx85" event={"ID":"6c43eb65-2efa-445a-9cba-69be6820ed92","Type":"ContainerStarted","Data":"12422da29f6a2bd1d8f2ec37660c037f809adfeaeaf2882a06fc7685094d9280"} Nov 04 04:47:53.463700 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:53.463614 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-648cd6c647-qqx85" event={"ID":"6c43eb65-2efa-445a-9cba-69be6820ed92","Type":"ContainerStarted","Data":"978cc4b3c9596f868df9a8682c3e983ead9bb65f1b9283f29687923f3ad64cae"} Nov 04 04:47:53.578875 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:53.578792 4509 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-c656644f9-tk9wq" podStartSLOduration=2.480431312 podStartE2EDuration="7.578774219s" totalImagesPullingTime="5.098342907s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 04:47:46 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:47:53.578246722 +0000 UTC m=+37.455970254" watchObservedRunningTime="2026-11-04 04:47:53.578774219 +0000 UTC m=+37.456497739" Nov 04 04:47:54.642849 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:54.642781 4509 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-648cd6c647-qqx85" podStartSLOduration=3.642763785 podStartE2EDuration="3.642763785s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 04:47:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:47:53.693508295 +0000 UTC m=+37.571231817" watchObservedRunningTime="2026-11-04 04:47:54.642763785 +0000 UTC m=+38.520487306" Nov 04 04:47:54.643546 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:54.643527 4509 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-648cd6c647-qqx85"] Nov 04 04:47:54.672390 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:54.672365 4509 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-79df8c9c48-vtqmq"] Nov 04 04:47:54.713823 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:54.713791 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-79df8c9c48-vtqmq" Nov 04 04:47:54.754846 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:54.754811 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-79df8c9c48-vtqmq"] Nov 04 04:47:54.810041 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:54.810006 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/45d89e4a-75ca-45b8-bab9-e392b3f65efd-console-oauth-config\") pod \"console-79df8c9c48-vtqmq\" (UID: \"45d89e4a-75ca-45b8-bab9-e392b3f65efd\") " pod="openshift-console/console-79df8c9c48-vtqmq" Nov 04 04:47:54.810227 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:54.810049 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45d89e4a-75ca-45b8-bab9-e392b3f65efd-trusted-ca-bundle\") pod \"console-79df8c9c48-vtqmq\" (UID: \"45d89e4a-75ca-45b8-bab9-e392b3f65efd\") " pod="openshift-console/console-79df8c9c48-vtqmq" Nov 04 04:47:54.810227 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:54.810159 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/45d89e4a-75ca-45b8-bab9-e392b3f65efd-service-ca\") pod \"console-79df8c9c48-vtqmq\" (UID: \"45d89e4a-75ca-45b8-bab9-e392b3f65efd\") " pod="openshift-console/console-79df8c9c48-vtqmq" Nov 04 04:47:54.810227 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:54.810200 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/45d89e4a-75ca-45b8-bab9-e392b3f65efd-tmp\") pod \"console-79df8c9c48-vtqmq\" (UID: \"45d89e4a-75ca-45b8-bab9-e392b3f65efd\") " pod="openshift-console/console-79df8c9c48-vtqmq" Nov 04 04:47:54.810398 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:54.810289 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/45d89e4a-75ca-45b8-bab9-e392b3f65efd-oauth-serving-cert\") pod \"console-79df8c9c48-vtqmq\" (UID: \"45d89e4a-75ca-45b8-bab9-e392b3f65efd\") " pod="openshift-console/console-79df8c9c48-vtqmq" Nov 04 04:47:54.810398 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:54.810313 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lk57n\" (UniqueName: \"kubernetes.io/projected/45d89e4a-75ca-45b8-bab9-e392b3f65efd-kube-api-access-lk57n\") pod \"console-79df8c9c48-vtqmq\" (UID: \"45d89e4a-75ca-45b8-bab9-e392b3f65efd\") " pod="openshift-console/console-79df8c9c48-vtqmq" Nov 04 04:47:54.810499 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:54.810420 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/45d89e4a-75ca-45b8-bab9-e392b3f65efd-console-serving-cert\") pod \"console-79df8c9c48-vtqmq\" (UID: \"45d89e4a-75ca-45b8-bab9-e392b3f65efd\") " pod="openshift-console/console-79df8c9c48-vtqmq" Nov 04 04:47:54.810499 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:54.810444 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/45d89e4a-75ca-45b8-bab9-e392b3f65efd-console-config\") pod \"console-79df8c9c48-vtqmq\" (UID: \"45d89e4a-75ca-45b8-bab9-e392b3f65efd\") " pod="openshift-console/console-79df8c9c48-vtqmq" Nov 04 04:47:54.910968 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:54.910864 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/45d89e4a-75ca-45b8-bab9-e392b3f65efd-service-ca\") pod \"console-79df8c9c48-vtqmq\" (UID: \"45d89e4a-75ca-45b8-bab9-e392b3f65efd\") " pod="openshift-console/console-79df8c9c48-vtqmq" Nov 04 04:47:54.910968 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:54.910920 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/45d89e4a-75ca-45b8-bab9-e392b3f65efd-tmp\") pod \"console-79df8c9c48-vtqmq\" (UID: \"45d89e4a-75ca-45b8-bab9-e392b3f65efd\") " pod="openshift-console/console-79df8c9c48-vtqmq" Nov 04 04:47:54.910968 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:54.910958 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/45d89e4a-75ca-45b8-bab9-e392b3f65efd-oauth-serving-cert\") pod \"console-79df8c9c48-vtqmq\" (UID: \"45d89e4a-75ca-45b8-bab9-e392b3f65efd\") " pod="openshift-console/console-79df8c9c48-vtqmq" Nov 04 04:47:54.911284 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:54.910976 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lk57n\" (UniqueName: \"kubernetes.io/projected/45d89e4a-75ca-45b8-bab9-e392b3f65efd-kube-api-access-lk57n\") pod \"console-79df8c9c48-vtqmq\" (UID: \"45d89e4a-75ca-45b8-bab9-e392b3f65efd\") " pod="openshift-console/console-79df8c9c48-vtqmq" Nov 04 04:47:54.911284 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:54.911006 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/45d89e4a-75ca-45b8-bab9-e392b3f65efd-console-serving-cert\") pod \"console-79df8c9c48-vtqmq\" (UID: \"45d89e4a-75ca-45b8-bab9-e392b3f65efd\") " pod="openshift-console/console-79df8c9c48-vtqmq" Nov 04 04:47:54.911284 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:54.911031 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/45d89e4a-75ca-45b8-bab9-e392b3f65efd-console-config\") pod \"console-79df8c9c48-vtqmq\" (UID: \"45d89e4a-75ca-45b8-bab9-e392b3f65efd\") " pod="openshift-console/console-79df8c9c48-vtqmq" Nov 04 04:47:54.911284 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:54.911094 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/45d89e4a-75ca-45b8-bab9-e392b3f65efd-console-oauth-config\") pod \"console-79df8c9c48-vtqmq\" (UID: \"45d89e4a-75ca-45b8-bab9-e392b3f65efd\") " pod="openshift-console/console-79df8c9c48-vtqmq" Nov 04 04:47:54.911284 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:54.911120 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45d89e4a-75ca-45b8-bab9-e392b3f65efd-trusted-ca-bundle\") pod \"console-79df8c9c48-vtqmq\" (UID: \"45d89e4a-75ca-45b8-bab9-e392b3f65efd\") " pod="openshift-console/console-79df8c9c48-vtqmq" Nov 04 04:47:54.911554 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:54.911493 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/45d89e4a-75ca-45b8-bab9-e392b3f65efd-tmp\") pod \"console-79df8c9c48-vtqmq\" (UID: \"45d89e4a-75ca-45b8-bab9-e392b3f65efd\") " pod="openshift-console/console-79df8c9c48-vtqmq" Nov 04 04:47:54.911889 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:54.911839 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/45d89e4a-75ca-45b8-bab9-e392b3f65efd-oauth-serving-cert\") pod \"console-79df8c9c48-vtqmq\" (UID: \"45d89e4a-75ca-45b8-bab9-e392b3f65efd\") " pod="openshift-console/console-79df8c9c48-vtqmq" Nov 04 04:47:54.912062 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:54.912038 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/45d89e4a-75ca-45b8-bab9-e392b3f65efd-console-config\") pod \"console-79df8c9c48-vtqmq\" (UID: \"45d89e4a-75ca-45b8-bab9-e392b3f65efd\") " pod="openshift-console/console-79df8c9c48-vtqmq" Nov 04 04:47:54.912143 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:54.912060 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45d89e4a-75ca-45b8-bab9-e392b3f65efd-trusted-ca-bundle\") pod \"console-79df8c9c48-vtqmq\" (UID: \"45d89e4a-75ca-45b8-bab9-e392b3f65efd\") " pod="openshift-console/console-79df8c9c48-vtqmq" Nov 04 04:47:54.912586 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:54.912562 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/45d89e4a-75ca-45b8-bab9-e392b3f65efd-service-ca\") pod \"console-79df8c9c48-vtqmq\" (UID: \"45d89e4a-75ca-45b8-bab9-e392b3f65efd\") " pod="openshift-console/console-79df8c9c48-vtqmq" Nov 04 04:47:54.914175 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:54.914151 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/45d89e4a-75ca-45b8-bab9-e392b3f65efd-console-serving-cert\") pod \"console-79df8c9c48-vtqmq\" (UID: \"45d89e4a-75ca-45b8-bab9-e392b3f65efd\") " pod="openshift-console/console-79df8c9c48-vtqmq" Nov 04 04:47:54.914257 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:54.914189 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/45d89e4a-75ca-45b8-bab9-e392b3f65efd-console-oauth-config\") pod \"console-79df8c9c48-vtqmq\" (UID: \"45d89e4a-75ca-45b8-bab9-e392b3f65efd\") " pod="openshift-console/console-79df8c9c48-vtqmq" Nov 04 04:47:54.968550 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:54.968520 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lk57n\" (UniqueName: \"kubernetes.io/projected/45d89e4a-75ca-45b8-bab9-e392b3f65efd-kube-api-access-lk57n\") pod \"console-79df8c9c48-vtqmq\" (UID: \"45d89e4a-75ca-45b8-bab9-e392b3f65efd\") " pod="openshift-console/console-79df8c9c48-vtqmq" Nov 04 04:47:55.044315 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:55.044283 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-79df8c9c48-vtqmq" Nov 04 04:47:55.452259 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:55.451977 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-79df8c9c48-vtqmq"] Nov 04 04:47:55.455111 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:47:55.455083 4509 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45d89e4a_75ca_45b8_bab9_e392b3f65efd.slice/crio-86ab0f7f989fa3ebc35e7ee90bae06b4072249e3374cd853a59d6c459b30ef5b WatchSource:0}: Error finding container 86ab0f7f989fa3ebc35e7ee90bae06b4072249e3374cd853a59d6c459b30ef5b: Status 404 returned error can't find the container with id 86ab0f7f989fa3ebc35e7ee90bae06b4072249e3374cd853a59d6c459b30ef5b Nov 04 04:47:55.473652 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:55.473620 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-79df8c9c48-vtqmq" event={"ID":"45d89e4a-75ca-45b8-bab9-e392b3f65efd","Type":"ContainerStarted","Data":"86ab0f7f989fa3ebc35e7ee90bae06b4072249e3374cd853a59d6c459b30ef5b"} Nov 04 04:47:56.479165 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:56.479125 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-79df8c9c48-vtqmq" event={"ID":"45d89e4a-75ca-45b8-bab9-e392b3f65efd","Type":"ContainerStarted","Data":"d1c467f196568547836faae0995f9e996bc117c49c4b29c06f21c698b34b7283"} Nov 04 04:47:56.595537 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:47:56.595444 4509 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-79df8c9c48-vtqmq" podStartSLOduration=2.59543038 podStartE2EDuration="2.59543038s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 04:47:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:47:56.595136786 +0000 UTC m=+40.472860306" watchObservedRunningTime="2026-11-04 04:47:56.59543038 +0000 UTC m=+40.473153900" Nov 04 04:48:02.123176 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:48:02.123136 4509 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-648cd6c647-qqx85" Nov 04 04:48:05.045435 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:48:05.045400 4509 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-79df8c9c48-vtqmq" Nov 04 04:48:05.045807 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:48:05.045482 4509 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-79df8c9c48-vtqmq" Nov 04 04:48:05.049965 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:48:05.049946 4509 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-79df8c9c48-vtqmq" Nov 04 04:48:05.513657 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:48:05.513633 4509 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-79df8c9c48-vtqmq" Nov 04 04:48:05.655095 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:48:05.655063 4509 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7567f6b9c5-nztb2"] Nov 04 04:48:14.452773 ip-10-0-133-180 kubenswrapper[4509]: E1104 04:48:14.452735 4509 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmdhl" containerName="extractor" Nov 04 04:48:20.494904 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:48:20.494841 4509 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-648cd6c647-qqx85" podUID="6c43eb65-2efa-445a-9cba-69be6820ed92" containerName="console" containerID="cri-o://12422da29f6a2bd1d8f2ec37660c037f809adfeaeaf2882a06fc7685094d9280" gracePeriod=15 Nov 04 04:48:20.782073 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:48:20.782041 4509 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-648cd6c647-qqx85" Nov 04 04:48:20.816666 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:48:20.816640 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6c43eb65-2efa-445a-9cba-69be6820ed92-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6c43eb65-2efa-445a-9cba-69be6820ed92-console-oauth-config\") pod \"6c43eb65-2efa-445a-9cba-69be6820ed92\" (UID: \"6c43eb65-2efa-445a-9cba-69be6820ed92\") " Nov 04 04:48:20.816798 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:48:20.816682 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6c43eb65-2efa-445a-9cba-69be6820ed92-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6c43eb65-2efa-445a-9cba-69be6820ed92-console-serving-cert\") pod \"6c43eb65-2efa-445a-9cba-69be6820ed92\" (UID: \"6c43eb65-2efa-445a-9cba-69be6820ed92\") " Nov 04 04:48:20.816798 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:48:20.816708 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c43eb65-2efa-445a-9cba-69be6820ed92-kube-api-access-dfwvz\" (UniqueName: \"kubernetes.io/projected/6c43eb65-2efa-445a-9cba-69be6820ed92-kube-api-access-dfwvz\") pod \"6c43eb65-2efa-445a-9cba-69be6820ed92\" (UID: \"6c43eb65-2efa-445a-9cba-69be6820ed92\") " Nov 04 04:48:20.816905 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:48:20.816871 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c43eb65-2efa-445a-9cba-69be6820ed92-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6c43eb65-2efa-445a-9cba-69be6820ed92-trusted-ca-bundle\") pod \"6c43eb65-2efa-445a-9cba-69be6820ed92\" (UID: \"6c43eb65-2efa-445a-9cba-69be6820ed92\") " Nov 04 04:48:20.816905 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:48:20.816903 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c43eb65-2efa-445a-9cba-69be6820ed92-console-config\" (UniqueName: \"kubernetes.io/configmap/6c43eb65-2efa-445a-9cba-69be6820ed92-console-config\") pod \"6c43eb65-2efa-445a-9cba-69be6820ed92\" (UID: \"6c43eb65-2efa-445a-9cba-69be6820ed92\") " Nov 04 04:48:20.817008 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:48:20.816967 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c43eb65-2efa-445a-9cba-69be6820ed92-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6c43eb65-2efa-445a-9cba-69be6820ed92-oauth-serving-cert\") pod \"6c43eb65-2efa-445a-9cba-69be6820ed92\" (UID: \"6c43eb65-2efa-445a-9cba-69be6820ed92\") " Nov 04 04:48:20.817065 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:48:20.817006 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c43eb65-2efa-445a-9cba-69be6820ed92-tmp\" (UniqueName: \"kubernetes.io/empty-dir/6c43eb65-2efa-445a-9cba-69be6820ed92-tmp\") pod \"6c43eb65-2efa-445a-9cba-69be6820ed92\" (UID: \"6c43eb65-2efa-445a-9cba-69be6820ed92\") " Nov 04 04:48:20.817065 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:48:20.817045 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c43eb65-2efa-445a-9cba-69be6820ed92-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c43eb65-2efa-445a-9cba-69be6820ed92-service-ca\") pod \"6c43eb65-2efa-445a-9cba-69be6820ed92\" (UID: \"6c43eb65-2efa-445a-9cba-69be6820ed92\") " Nov 04 04:48:20.817281 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:48:20.817252 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c43eb65-2efa-445a-9cba-69be6820ed92-trusted-ca-bundle" pod "6c43eb65-2efa-445a-9cba-69be6820ed92" (UID: "6c43eb65-2efa-445a-9cba-69be6820ed92"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 04:48:20.817539 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:48:20.817498 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c43eb65-2efa-445a-9cba-69be6820ed92-console-config" pod "6c43eb65-2efa-445a-9cba-69be6820ed92" (UID: "6c43eb65-2efa-445a-9cba-69be6820ed92"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 04:48:20.817688 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:48:20.817664 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c43eb65-2efa-445a-9cba-69be6820ed92-tmp" pod "6c43eb65-2efa-445a-9cba-69be6820ed92" (UID: "6c43eb65-2efa-445a-9cba-69be6820ed92"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 04:48:20.817755 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:48:20.817698 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c43eb65-2efa-445a-9cba-69be6820ed92-service-ca" pod "6c43eb65-2efa-445a-9cba-69be6820ed92" (UID: "6c43eb65-2efa-445a-9cba-69be6820ed92"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 04:48:20.817885 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:48:20.817860 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c43eb65-2efa-445a-9cba-69be6820ed92-oauth-serving-cert" pod "6c43eb65-2efa-445a-9cba-69be6820ed92" (UID: "6c43eb65-2efa-445a-9cba-69be6820ed92"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 04:48:20.819152 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:48:20.819124 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c43eb65-2efa-445a-9cba-69be6820ed92-console-serving-cert" pod "6c43eb65-2efa-445a-9cba-69be6820ed92" (UID: "6c43eb65-2efa-445a-9cba-69be6820ed92"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 04:48:20.819247 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:48:20.819172 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c43eb65-2efa-445a-9cba-69be6820ed92-kube-api-access-dfwvz" pod "6c43eb65-2efa-445a-9cba-69be6820ed92" (UID: "6c43eb65-2efa-445a-9cba-69be6820ed92"). InnerVolumeSpecName "kube-api-access-dfwvz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 04:48:20.819247 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:48:20.819196 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c43eb65-2efa-445a-9cba-69be6820ed92-console-oauth-config" pod "6c43eb65-2efa-445a-9cba-69be6820ed92" (UID: "6c43eb65-2efa-445a-9cba-69be6820ed92"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 04:48:20.918408 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:48:20.918371 4509 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6c43eb65-2efa-445a-9cba-69be6820ed92-oauth-serving-cert\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 04:48:20.918408 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:48:20.918408 4509 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6c43eb65-2efa-445a-9cba-69be6820ed92-tmp\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 04:48:20.918584 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:48:20.918424 4509 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6c43eb65-2efa-445a-9cba-69be6820ed92-service-ca\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 04:48:20.918584 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:48:20.918448 4509 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6c43eb65-2efa-445a-9cba-69be6820ed92-console-oauth-config\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 04:48:20.918584 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:48:20.918462 4509 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6c43eb65-2efa-445a-9cba-69be6820ed92-console-serving-cert\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 04:48:20.918584 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:48:20.918475 4509 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dfwvz\" (UniqueName: \"kubernetes.io/projected/6c43eb65-2efa-445a-9cba-69be6820ed92-kube-api-access-dfwvz\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 04:48:20.918584 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:48:20.918490 4509 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6c43eb65-2efa-445a-9cba-69be6820ed92-trusted-ca-bundle\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 04:48:20.918584 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:48:20.918503 4509 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6c43eb65-2efa-445a-9cba-69be6820ed92-console-config\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 04:48:21.564272 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:48:21.564234 4509 generic.go:356] "Generic (PLEG): container finished" podID="6c43eb65-2efa-445a-9cba-69be6820ed92" containerID="12422da29f6a2bd1d8f2ec37660c037f809adfeaeaf2882a06fc7685094d9280" exitCode=0 Nov 04 04:48:21.564675 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:48:21.564292 4509 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-648cd6c647-qqx85" Nov 04 04:48:21.564675 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:48:21.564339 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-648cd6c647-qqx85" event={"ID":"6c43eb65-2efa-445a-9cba-69be6820ed92","Type":"ContainerDied","Data":"12422da29f6a2bd1d8f2ec37660c037f809adfeaeaf2882a06fc7685094d9280"} Nov 04 04:48:21.564675 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:48:21.564392 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-648cd6c647-qqx85" event={"ID":"6c43eb65-2efa-445a-9cba-69be6820ed92","Type":"ContainerDied","Data":"978cc4b3c9596f868df9a8682c3e983ead9bb65f1b9283f29687923f3ad64cae"} Nov 04 04:48:21.564675 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:48:21.564426 4509 scope.go:122] "RemoveContainer" containerID="12422da29f6a2bd1d8f2ec37660c037f809adfeaeaf2882a06fc7685094d9280" Nov 04 04:48:21.572925 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:48:21.572904 4509 scope.go:122] "RemoveContainer" containerID="12422da29f6a2bd1d8f2ec37660c037f809adfeaeaf2882a06fc7685094d9280" Nov 04 04:48:21.573216 ip-10-0-133-180 kubenswrapper[4509]: E1104 04:48:21.573199 4509 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12422da29f6a2bd1d8f2ec37660c037f809adfeaeaf2882a06fc7685094d9280\": container with ID starting with 12422da29f6a2bd1d8f2ec37660c037f809adfeaeaf2882a06fc7685094d9280 not found: ID does not exist" containerID="12422da29f6a2bd1d8f2ec37660c037f809adfeaeaf2882a06fc7685094d9280" Nov 04 04:48:21.573259 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:48:21.573227 4509 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12422da29f6a2bd1d8f2ec37660c037f809adfeaeaf2882a06fc7685094d9280"} err="failed to get container status \"12422da29f6a2bd1d8f2ec37660c037f809adfeaeaf2882a06fc7685094d9280\": rpc error: code = NotFound desc = could not find container \"12422da29f6a2bd1d8f2ec37660c037f809adfeaeaf2882a06fc7685094d9280\": container with ID starting with 12422da29f6a2bd1d8f2ec37660c037f809adfeaeaf2882a06fc7685094d9280 not found: ID does not exist" Nov 04 04:48:21.728771 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:48:21.728739 4509 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-648cd6c647-qqx85"] Nov 04 04:48:21.778759 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:48:21.778729 4509 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-648cd6c647-qqx85"] Nov 04 04:48:22.260104 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:48:22.260077 4509 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c43eb65-2efa-445a-9cba-69be6820ed92" path="/var/lib/kubelet/pods/6c43eb65-2efa-445a-9cba-69be6820ed92/volumes" Nov 04 04:48:30.673973 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:48:30.673907 4509 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-7567f6b9c5-nztb2" podUID="50622b62-1450-4af2-80fc-d2f3d8f82d27" containerName="console" containerID="cri-o://ae73a568e8b235ae34d54d5babea6cb6214bdf6a364b968e218ff745296ef536" gracePeriod=15 Nov 04 04:48:30.956418 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:48:30.956392 4509 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7567f6b9c5-nztb2" Nov 04 04:48:30.991876 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:48:30.991849 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50622b62-1450-4af2-80fc-d2f3d8f82d27-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/50622b62-1450-4af2-80fc-d2f3d8f82d27-oauth-serving-cert\") pod \"50622b62-1450-4af2-80fc-d2f3d8f82d27\" (UID: \"50622b62-1450-4af2-80fc-d2f3d8f82d27\") " Nov 04 04:48:30.991992 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:48:30.991887 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/50622b62-1450-4af2-80fc-d2f3d8f82d27-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/50622b62-1450-4af2-80fc-d2f3d8f82d27-console-serving-cert\") pod \"50622b62-1450-4af2-80fc-d2f3d8f82d27\" (UID: \"50622b62-1450-4af2-80fc-d2f3d8f82d27\") " Nov 04 04:48:30.991992 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:48:30.991915 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50622b62-1450-4af2-80fc-d2f3d8f82d27-kube-api-access-nlnb5\" (UniqueName: \"kubernetes.io/projected/50622b62-1450-4af2-80fc-d2f3d8f82d27-kube-api-access-nlnb5\") pod \"50622b62-1450-4af2-80fc-d2f3d8f82d27\" (UID: \"50622b62-1450-4af2-80fc-d2f3d8f82d27\") " Nov 04 04:48:30.992160 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:48:30.992097 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50622b62-1450-4af2-80fc-d2f3d8f82d27-service-ca\" (UniqueName: \"kubernetes.io/configmap/50622b62-1450-4af2-80fc-d2f3d8f82d27-service-ca\") pod \"50622b62-1450-4af2-80fc-d2f3d8f82d27\" (UID: \"50622b62-1450-4af2-80fc-d2f3d8f82d27\") " Nov 04 04:48:30.992160 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:48:30.992139 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/50622b62-1450-4af2-80fc-d2f3d8f82d27-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/50622b62-1450-4af2-80fc-d2f3d8f82d27-console-oauth-config\") pod \"50622b62-1450-4af2-80fc-d2f3d8f82d27\" (UID: \"50622b62-1450-4af2-80fc-d2f3d8f82d27\") " Nov 04 04:48:30.992261 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:48:30.992163 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50622b62-1450-4af2-80fc-d2f3d8f82d27-console-config\" (UniqueName: \"kubernetes.io/configmap/50622b62-1450-4af2-80fc-d2f3d8f82d27-console-config\") pod \"50622b62-1450-4af2-80fc-d2f3d8f82d27\" (UID: \"50622b62-1450-4af2-80fc-d2f3d8f82d27\") " Nov 04 04:48:30.992261 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:48:30.992210 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50622b62-1450-4af2-80fc-d2f3d8f82d27-tmp\" (UniqueName: \"kubernetes.io/empty-dir/50622b62-1450-4af2-80fc-d2f3d8f82d27-tmp\") pod \"50622b62-1450-4af2-80fc-d2f3d8f82d27\" (UID: \"50622b62-1450-4af2-80fc-d2f3d8f82d27\") " Nov 04 04:48:30.992261 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:48:30.992232 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50622b62-1450-4af2-80fc-d2f3d8f82d27-oauth-serving-cert" pod "50622b62-1450-4af2-80fc-d2f3d8f82d27" (UID: "50622b62-1450-4af2-80fc-d2f3d8f82d27"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 04:48:30.992481 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:48:30.992453 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50622b62-1450-4af2-80fc-d2f3d8f82d27-service-ca" pod "50622b62-1450-4af2-80fc-d2f3d8f82d27" (UID: "50622b62-1450-4af2-80fc-d2f3d8f82d27"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 04:48:30.992548 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:48:30.992512 4509 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/50622b62-1450-4af2-80fc-d2f3d8f82d27-oauth-serving-cert\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 04:48:30.992605 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:48:30.992552 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50622b62-1450-4af2-80fc-d2f3d8f82d27-tmp" pod "50622b62-1450-4af2-80fc-d2f3d8f82d27" (UID: "50622b62-1450-4af2-80fc-d2f3d8f82d27"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 04:48:30.992605 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:48:30.992573 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50622b62-1450-4af2-80fc-d2f3d8f82d27-console-config" pod "50622b62-1450-4af2-80fc-d2f3d8f82d27" (UID: "50622b62-1450-4af2-80fc-d2f3d8f82d27"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 04:48:30.994065 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:48:30.994040 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50622b62-1450-4af2-80fc-d2f3d8f82d27-console-serving-cert" pod "50622b62-1450-4af2-80fc-d2f3d8f82d27" (UID: "50622b62-1450-4af2-80fc-d2f3d8f82d27"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 04:48:30.994164 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:48:30.994078 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50622b62-1450-4af2-80fc-d2f3d8f82d27-kube-api-access-nlnb5" pod "50622b62-1450-4af2-80fc-d2f3d8f82d27" (UID: "50622b62-1450-4af2-80fc-d2f3d8f82d27"). InnerVolumeSpecName "kube-api-access-nlnb5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 04:48:30.994460 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:48:30.994440 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50622b62-1450-4af2-80fc-d2f3d8f82d27-console-oauth-config" pod "50622b62-1450-4af2-80fc-d2f3d8f82d27" (UID: "50622b62-1450-4af2-80fc-d2f3d8f82d27"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 04:48:31.093596 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:48:31.093568 4509 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/50622b62-1450-4af2-80fc-d2f3d8f82d27-console-serving-cert\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 04:48:31.093596 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:48:31.093593 4509 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlnb5\" (UniqueName: \"kubernetes.io/projected/50622b62-1450-4af2-80fc-d2f3d8f82d27-kube-api-access-nlnb5\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 04:48:31.093783 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:48:31.093605 4509 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/50622b62-1450-4af2-80fc-d2f3d8f82d27-service-ca\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 04:48:31.093783 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:48:31.093613 4509 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/50622b62-1450-4af2-80fc-d2f3d8f82d27-console-oauth-config\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 04:48:31.093783 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:48:31.093624 4509 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/50622b62-1450-4af2-80fc-d2f3d8f82d27-console-config\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 04:48:31.093783 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:48:31.093633 4509 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/50622b62-1450-4af2-80fc-d2f3d8f82d27-tmp\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 04:48:31.443472 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:48:31.443435 4509 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-796467b4dc-mrjk9"] Nov 04 04:48:31.443719 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:48:31.443707 4509 cpu_manager.go:414] "containerMap: removing container" podUID="6c43eb65-2efa-445a-9cba-69be6820ed92" containerName="console" Nov 04 04:48:31.443762 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:48:31.443721 4509 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c43eb65-2efa-445a-9cba-69be6820ed92" containerName="console" Nov 04 04:48:31.443762 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:48:31.443730 4509 cpu_manager.go:414] "containerMap: removing container" podUID="50622b62-1450-4af2-80fc-d2f3d8f82d27" containerName="console" Nov 04 04:48:31.443762 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:48:31.443734 4509 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50622b62-1450-4af2-80fc-d2f3d8f82d27" containerName="console" Nov 04 04:48:31.443847 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:48:31.443786 4509 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c43eb65-2efa-445a-9cba-69be6820ed92" containerName="console" Nov 04 04:48:31.443847 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:48:31.443793 4509 memory_manager.go:365] "RemoveStaleState removing state" podUID="50622b62-1450-4af2-80fc-d2f3d8f82d27" containerName="console" Nov 04 04:48:31.461952 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:48:31.461934 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-796467b4dc-mrjk9" Nov 04 04:48:31.496332 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:48:31.496285 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgdnr\" (UniqueName: \"kubernetes.io/projected/1d63899a-ef23-444e-9089-5c40de8a3817-kube-api-access-wgdnr\") pod \"console-796467b4dc-mrjk9\" (UID: \"1d63899a-ef23-444e-9089-5c40de8a3817\") " pod="openshift-console/console-796467b4dc-mrjk9" Nov 04 04:48:31.496332 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:48:31.496335 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1d63899a-ef23-444e-9089-5c40de8a3817-console-oauth-config\") pod \"console-796467b4dc-mrjk9\" (UID: \"1d63899a-ef23-444e-9089-5c40de8a3817\") " pod="openshift-console/console-796467b4dc-mrjk9" Nov 04 04:48:31.496529 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:48:31.496403 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1d63899a-ef23-444e-9089-5c40de8a3817-oauth-serving-cert\") pod \"console-796467b4dc-mrjk9\" (UID: \"1d63899a-ef23-444e-9089-5c40de8a3817\") " pod="openshift-console/console-796467b4dc-mrjk9" Nov 04 04:48:31.496529 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:48:31.496438 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d63899a-ef23-444e-9089-5c40de8a3817-trusted-ca-bundle\") pod \"console-796467b4dc-mrjk9\" (UID: \"1d63899a-ef23-444e-9089-5c40de8a3817\") " pod="openshift-console/console-796467b4dc-mrjk9" Nov 04 04:48:31.496529 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:48:31.496516 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1d63899a-ef23-444e-9089-5c40de8a3817-console-serving-cert\") pod \"console-796467b4dc-mrjk9\" (UID: \"1d63899a-ef23-444e-9089-5c40de8a3817\") " pod="openshift-console/console-796467b4dc-mrjk9" Nov 04 04:48:31.496624 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:48:31.496558 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1d63899a-ef23-444e-9089-5c40de8a3817-service-ca\") pod \"console-796467b4dc-mrjk9\" (UID: \"1d63899a-ef23-444e-9089-5c40de8a3817\") " pod="openshift-console/console-796467b4dc-mrjk9" Nov 04 04:48:31.496656 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:48:31.496624 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1d63899a-ef23-444e-9089-5c40de8a3817-console-config\") pod \"console-796467b4dc-mrjk9\" (UID: \"1d63899a-ef23-444e-9089-5c40de8a3817\") " pod="openshift-console/console-796467b4dc-mrjk9" Nov 04 04:48:31.496656 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:48:31.496643 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1d63899a-ef23-444e-9089-5c40de8a3817-tmp\") pod \"console-796467b4dc-mrjk9\" (UID: \"1d63899a-ef23-444e-9089-5c40de8a3817\") " pod="openshift-console/console-796467b4dc-mrjk9" Nov 04 04:48:31.547652 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:48:31.547613 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-796467b4dc-mrjk9"] Nov 04 04:48:31.596972 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:48:31.596932 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d63899a-ef23-444e-9089-5c40de8a3817-trusted-ca-bundle\") pod \"console-796467b4dc-mrjk9\" (UID: \"1d63899a-ef23-444e-9089-5c40de8a3817\") " pod="openshift-console/console-796467b4dc-mrjk9" Nov 04 04:48:31.596972 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:48:31.596966 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1d63899a-ef23-444e-9089-5c40de8a3817-console-serving-cert\") pod \"console-796467b4dc-mrjk9\" (UID: \"1d63899a-ef23-444e-9089-5c40de8a3817\") " pod="openshift-console/console-796467b4dc-mrjk9" Nov 04 04:48:31.597211 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:48:31.597011 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1d63899a-ef23-444e-9089-5c40de8a3817-service-ca\") pod \"console-796467b4dc-mrjk9\" (UID: \"1d63899a-ef23-444e-9089-5c40de8a3817\") " pod="openshift-console/console-796467b4dc-mrjk9" Nov 04 04:48:31.597211 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:48:31.597037 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1d63899a-ef23-444e-9089-5c40de8a3817-console-config\") pod \"console-796467b4dc-mrjk9\" (UID: \"1d63899a-ef23-444e-9089-5c40de8a3817\") " pod="openshift-console/console-796467b4dc-mrjk9" Nov 04 04:48:31.597211 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:48:31.597192 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1d63899a-ef23-444e-9089-5c40de8a3817-tmp\") pod \"console-796467b4dc-mrjk9\" (UID: \"1d63899a-ef23-444e-9089-5c40de8a3817\") " pod="openshift-console/console-796467b4dc-mrjk9" Nov 04 04:48:31.597403 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:48:31.597236 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgdnr\" (UniqueName: \"kubernetes.io/projected/1d63899a-ef23-444e-9089-5c40de8a3817-kube-api-access-wgdnr\") pod \"console-796467b4dc-mrjk9\" (UID: \"1d63899a-ef23-444e-9089-5c40de8a3817\") " pod="openshift-console/console-796467b4dc-mrjk9" Nov 04 04:48:31.597403 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:48:31.597259 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1d63899a-ef23-444e-9089-5c40de8a3817-console-oauth-config\") pod \"console-796467b4dc-mrjk9\" (UID: \"1d63899a-ef23-444e-9089-5c40de8a3817\") " pod="openshift-console/console-796467b4dc-mrjk9" Nov 04 04:48:31.597403 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:48:31.597280 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1d63899a-ef23-444e-9089-5c40de8a3817-oauth-serving-cert\") pod \"console-796467b4dc-mrjk9\" (UID: \"1d63899a-ef23-444e-9089-5c40de8a3817\") " pod="openshift-console/console-796467b4dc-mrjk9" Nov 04 04:48:31.598104 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:48:31.597781 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1d63899a-ef23-444e-9089-5c40de8a3817-service-ca\") pod \"console-796467b4dc-mrjk9\" (UID: \"1d63899a-ef23-444e-9089-5c40de8a3817\") " pod="openshift-console/console-796467b4dc-mrjk9" Nov 04 04:48:31.598104 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:48:31.597810 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1d63899a-ef23-444e-9089-5c40de8a3817-console-config\") pod \"console-796467b4dc-mrjk9\" (UID: \"1d63899a-ef23-444e-9089-5c40de8a3817\") " pod="openshift-console/console-796467b4dc-mrjk9" Nov 04 04:48:31.598104 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:48:31.597886 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1d63899a-ef23-444e-9089-5c40de8a3817-oauth-serving-cert\") pod \"console-796467b4dc-mrjk9\" (UID: \"1d63899a-ef23-444e-9089-5c40de8a3817\") " pod="openshift-console/console-796467b4dc-mrjk9" Nov 04 04:48:31.598104 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:48:31.598069 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1d63899a-ef23-444e-9089-5c40de8a3817-tmp\") pod \"console-796467b4dc-mrjk9\" (UID: \"1d63899a-ef23-444e-9089-5c40de8a3817\") " pod="openshift-console/console-796467b4dc-mrjk9" Nov 04 04:48:31.598104 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:48:31.598072 4509 generic.go:356] "Generic (PLEG): container finished" podID="50622b62-1450-4af2-80fc-d2f3d8f82d27" containerID="ae73a568e8b235ae34d54d5babea6cb6214bdf6a364b968e218ff745296ef536" exitCode=0 Nov 04 04:48:31.598104 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:48:31.598097 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7567f6b9c5-nztb2" event={"ID":"50622b62-1450-4af2-80fc-d2f3d8f82d27","Type":"ContainerDied","Data":"ae73a568e8b235ae34d54d5babea6cb6214bdf6a364b968e218ff745296ef536"} Nov 04 04:48:31.598401 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:48:31.598126 4509 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7567f6b9c5-nztb2" Nov 04 04:48:31.598401 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:48:31.598137 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d63899a-ef23-444e-9089-5c40de8a3817-trusted-ca-bundle\") pod \"console-796467b4dc-mrjk9\" (UID: \"1d63899a-ef23-444e-9089-5c40de8a3817\") " pod="openshift-console/console-796467b4dc-mrjk9" Nov 04 04:48:31.598401 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:48:31.598144 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7567f6b9c5-nztb2" event={"ID":"50622b62-1450-4af2-80fc-d2f3d8f82d27","Type":"ContainerDied","Data":"a06bc4abd29f912f1c40df9ee4858acca7788369b2e169d2df97c576ed1bd9c9"} Nov 04 04:48:31.598401 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:48:31.598172 4509 scope.go:122] "RemoveContainer" containerID="ae73a568e8b235ae34d54d5babea6cb6214bdf6a364b968e218ff745296ef536" Nov 04 04:48:31.600111 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:48:31.600086 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1d63899a-ef23-444e-9089-5c40de8a3817-console-serving-cert\") pod \"console-796467b4dc-mrjk9\" (UID: \"1d63899a-ef23-444e-9089-5c40de8a3817\") " pod="openshift-console/console-796467b4dc-mrjk9" Nov 04 04:48:31.600226 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:48:31.600131 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1d63899a-ef23-444e-9089-5c40de8a3817-console-oauth-config\") pod \"console-796467b4dc-mrjk9\" (UID: \"1d63899a-ef23-444e-9089-5c40de8a3817\") " pod="openshift-console/console-796467b4dc-mrjk9" Nov 04 04:48:31.606791 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:48:31.606777 4509 scope.go:122] "RemoveContainer" containerID="ae73a568e8b235ae34d54d5babea6cb6214bdf6a364b968e218ff745296ef536" Nov 04 04:48:31.607073 ip-10-0-133-180 kubenswrapper[4509]: E1104 04:48:31.607054 4509 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae73a568e8b235ae34d54d5babea6cb6214bdf6a364b968e218ff745296ef536\": container with ID starting with ae73a568e8b235ae34d54d5babea6cb6214bdf6a364b968e218ff745296ef536 not found: ID does not exist" containerID="ae73a568e8b235ae34d54d5babea6cb6214bdf6a364b968e218ff745296ef536" Nov 04 04:48:31.607147 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:48:31.607084 4509 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae73a568e8b235ae34d54d5babea6cb6214bdf6a364b968e218ff745296ef536"} err="failed to get container status \"ae73a568e8b235ae34d54d5babea6cb6214bdf6a364b968e218ff745296ef536\": rpc error: code = NotFound desc = could not find container \"ae73a568e8b235ae34d54d5babea6cb6214bdf6a364b968e218ff745296ef536\": container with ID starting with ae73a568e8b235ae34d54d5babea6cb6214bdf6a364b968e218ff745296ef536 not found: ID does not exist" Nov 04 04:48:31.654837 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:48:31.654806 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgdnr\" (UniqueName: \"kubernetes.io/projected/1d63899a-ef23-444e-9089-5c40de8a3817-kube-api-access-wgdnr\") pod \"console-796467b4dc-mrjk9\" (UID: \"1d63899a-ef23-444e-9089-5c40de8a3817\") " pod="openshift-console/console-796467b4dc-mrjk9" Nov 04 04:48:31.761303 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:48:31.761268 4509 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7567f6b9c5-nztb2"] Nov 04 04:48:31.764743 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:48:31.764720 4509 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7567f6b9c5-nztb2"] Nov 04 04:48:31.785210 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:48:31.785192 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-796467b4dc-mrjk9" Nov 04 04:48:32.186368 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:48:32.186263 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-796467b4dc-mrjk9"] Nov 04 04:48:32.189498 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:48:32.189461 4509 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d63899a_ef23_444e_9089_5c40de8a3817.slice/crio-a63cde33f99aee5e83398cbd2bfade910d0d6eb79c5914b923bfaef902f39d1b WatchSource:0}: Error finding container a63cde33f99aee5e83398cbd2bfade910d0d6eb79c5914b923bfaef902f39d1b: Status 404 returned error can't find the container with id a63cde33f99aee5e83398cbd2bfade910d0d6eb79c5914b923bfaef902f39d1b Nov 04 04:48:32.261693 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:48:32.261667 4509 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50622b62-1450-4af2-80fc-d2f3d8f82d27" path="/var/lib/kubelet/pods/50622b62-1450-4af2-80fc-d2f3d8f82d27/volumes" Nov 04 04:48:32.604300 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:48:32.604265 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-796467b4dc-mrjk9" event={"ID":"1d63899a-ef23-444e-9089-5c40de8a3817","Type":"ContainerStarted","Data":"40d87504252369b6b740e01f535ce89db2c045c21d48f343db2a99fb79a93088"} Nov 04 04:48:32.604468 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:48:32.604311 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-796467b4dc-mrjk9" event={"ID":"1d63899a-ef23-444e-9089-5c40de8a3817","Type":"ContainerStarted","Data":"a63cde33f99aee5e83398cbd2bfade910d0d6eb79c5914b923bfaef902f39d1b"} Nov 04 04:48:32.721979 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:48:32.721918 4509 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-796467b4dc-mrjk9" podStartSLOduration=1.7219050390000001 podStartE2EDuration="1.721905039s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 04:48:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:48:32.721513969 +0000 UTC m=+76.599237491" watchObservedRunningTime="2026-11-04 04:48:32.721905039 +0000 UTC m=+76.599628560" Nov 04 04:48:41.785385 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:48:41.785342 4509 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-796467b4dc-mrjk9" Nov 04 04:48:41.785962 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:48:41.785437 4509 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-796467b4dc-mrjk9" Nov 04 04:48:41.790878 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:48:41.790857 4509 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-796467b4dc-mrjk9" Nov 04 04:48:42.640987 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:48:42.640962 4509 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-796467b4dc-mrjk9" Nov 04 04:48:42.809208 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:48:42.809177 4509 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-79df8c9c48-vtqmq"] Nov 04 04:48:44.452758 ip-10-0-133-180 kubenswrapper[4509]: E1104 04:48:44.452721 4509 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmdhl" containerName="extractor" Nov 04 04:49:07.828438 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:49:07.828374 4509 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-79df8c9c48-vtqmq" podUID="45d89e4a-75ca-45b8-bab9-e392b3f65efd" containerName="console" containerID="cri-o://d1c467f196568547836faae0995f9e996bc117c49c4b29c06f21c698b34b7283" gracePeriod=15 Nov 04 04:49:08.073275 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:49:08.073249 4509 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-79df8c9c48-vtqmq" Nov 04 04:49:08.158359 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:49:08.158272 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45d89e4a-75ca-45b8-bab9-e392b3f65efd-service-ca\" (UniqueName: \"kubernetes.io/configmap/45d89e4a-75ca-45b8-bab9-e392b3f65efd-service-ca\") pod \"45d89e4a-75ca-45b8-bab9-e392b3f65efd\" (UID: \"45d89e4a-75ca-45b8-bab9-e392b3f65efd\") " Nov 04 04:49:08.158359 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:49:08.158315 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/45d89e4a-75ca-45b8-bab9-e392b3f65efd-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/45d89e4a-75ca-45b8-bab9-e392b3f65efd-console-oauth-config\") pod \"45d89e4a-75ca-45b8-bab9-e392b3f65efd\" (UID: \"45d89e4a-75ca-45b8-bab9-e392b3f65efd\") " Nov 04 04:49:08.158545 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:49:08.158369 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45d89e4a-75ca-45b8-bab9-e392b3f65efd-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/45d89e4a-75ca-45b8-bab9-e392b3f65efd-oauth-serving-cert\") pod \"45d89e4a-75ca-45b8-bab9-e392b3f65efd\" (UID: \"45d89e4a-75ca-45b8-bab9-e392b3f65efd\") " Nov 04 04:49:08.158545 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:49:08.158402 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45d89e4a-75ca-45b8-bab9-e392b3f65efd-tmp\" (UniqueName: \"kubernetes.io/empty-dir/45d89e4a-75ca-45b8-bab9-e392b3f65efd-tmp\") pod \"45d89e4a-75ca-45b8-bab9-e392b3f65efd\" (UID: \"45d89e4a-75ca-45b8-bab9-e392b3f65efd\") " Nov 04 04:49:08.158545 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:49:08.158428 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/45d89e4a-75ca-45b8-bab9-e392b3f65efd-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/45d89e4a-75ca-45b8-bab9-e392b3f65efd-console-serving-cert\") pod \"45d89e4a-75ca-45b8-bab9-e392b3f65efd\" (UID: \"45d89e4a-75ca-45b8-bab9-e392b3f65efd\") " Nov 04 04:49:08.158545 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:49:08.158455 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45d89e4a-75ca-45b8-bab9-e392b3f65efd-console-config\" (UniqueName: \"kubernetes.io/configmap/45d89e4a-75ca-45b8-bab9-e392b3f65efd-console-config\") pod \"45d89e4a-75ca-45b8-bab9-e392b3f65efd\" (UID: \"45d89e4a-75ca-45b8-bab9-e392b3f65efd\") " Nov 04 04:49:08.158545 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:49:08.158486 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45d89e4a-75ca-45b8-bab9-e392b3f65efd-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45d89e4a-75ca-45b8-bab9-e392b3f65efd-trusted-ca-bundle\") pod \"45d89e4a-75ca-45b8-bab9-e392b3f65efd\" (UID: \"45d89e4a-75ca-45b8-bab9-e392b3f65efd\") " Nov 04 04:49:08.158545 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:49:08.158542 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45d89e4a-75ca-45b8-bab9-e392b3f65efd-kube-api-access-lk57n\" (UniqueName: \"kubernetes.io/projected/45d89e4a-75ca-45b8-bab9-e392b3f65efd-kube-api-access-lk57n\") pod \"45d89e4a-75ca-45b8-bab9-e392b3f65efd\" (UID: \"45d89e4a-75ca-45b8-bab9-e392b3f65efd\") " Nov 04 04:49:08.158823 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:49:08.158686 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45d89e4a-75ca-45b8-bab9-e392b3f65efd-service-ca" pod "45d89e4a-75ca-45b8-bab9-e392b3f65efd" (UID: "45d89e4a-75ca-45b8-bab9-e392b3f65efd"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 04:49:08.158823 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:49:08.158740 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45d89e4a-75ca-45b8-bab9-e392b3f65efd-tmp" pod "45d89e4a-75ca-45b8-bab9-e392b3f65efd" (UID: "45d89e4a-75ca-45b8-bab9-e392b3f65efd"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 04:49:08.158961 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:49:08.158813 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45d89e4a-75ca-45b8-bab9-e392b3f65efd-oauth-serving-cert" pod "45d89e4a-75ca-45b8-bab9-e392b3f65efd" (UID: "45d89e4a-75ca-45b8-bab9-e392b3f65efd"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 04:49:08.159072 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:49:08.159046 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45d89e4a-75ca-45b8-bab9-e392b3f65efd-trusted-ca-bundle" pod "45d89e4a-75ca-45b8-bab9-e392b3f65efd" (UID: "45d89e4a-75ca-45b8-bab9-e392b3f65efd"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 04:49:08.159129 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:49:08.159066 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45d89e4a-75ca-45b8-bab9-e392b3f65efd-console-config" pod "45d89e4a-75ca-45b8-bab9-e392b3f65efd" (UID: "45d89e4a-75ca-45b8-bab9-e392b3f65efd"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 04:49:08.160574 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:49:08.160540 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45d89e4a-75ca-45b8-bab9-e392b3f65efd-console-oauth-config" pod "45d89e4a-75ca-45b8-bab9-e392b3f65efd" (UID: "45d89e4a-75ca-45b8-bab9-e392b3f65efd"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 04:49:08.160675 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:49:08.160577 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45d89e4a-75ca-45b8-bab9-e392b3f65efd-console-serving-cert" pod "45d89e4a-75ca-45b8-bab9-e392b3f65efd" (UID: "45d89e4a-75ca-45b8-bab9-e392b3f65efd"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 04:49:08.160675 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:49:08.160670 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45d89e4a-75ca-45b8-bab9-e392b3f65efd-kube-api-access-lk57n" pod "45d89e4a-75ca-45b8-bab9-e392b3f65efd" (UID: "45d89e4a-75ca-45b8-bab9-e392b3f65efd"). InnerVolumeSpecName "kube-api-access-lk57n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 04:49:08.259474 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:49:08.259447 4509 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/45d89e4a-75ca-45b8-bab9-e392b3f65efd-service-ca\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 04:49:08.259474 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:49:08.259469 4509 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/45d89e4a-75ca-45b8-bab9-e392b3f65efd-console-oauth-config\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 04:49:08.259474 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:49:08.259478 4509 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/45d89e4a-75ca-45b8-bab9-e392b3f65efd-oauth-serving-cert\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 04:49:08.259739 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:49:08.259486 4509 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/45d89e4a-75ca-45b8-bab9-e392b3f65efd-tmp\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 04:49:08.259739 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:49:08.259495 4509 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/45d89e4a-75ca-45b8-bab9-e392b3f65efd-console-serving-cert\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 04:49:08.259739 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:49:08.259503 4509 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/45d89e4a-75ca-45b8-bab9-e392b3f65efd-console-config\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 04:49:08.259739 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:49:08.259511 4509 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45d89e4a-75ca-45b8-bab9-e392b3f65efd-trusted-ca-bundle\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 04:49:08.259739 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:49:08.259520 4509 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lk57n\" (UniqueName: \"kubernetes.io/projected/45d89e4a-75ca-45b8-bab9-e392b3f65efd-kube-api-access-lk57n\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 04:49:08.735580 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:49:08.735541 4509 generic.go:356] "Generic (PLEG): container finished" podID="45d89e4a-75ca-45b8-bab9-e392b3f65efd" containerID="d1c467f196568547836faae0995f9e996bc117c49c4b29c06f21c698b34b7283" exitCode=0 Nov 04 04:49:08.735737 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:49:08.735590 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-79df8c9c48-vtqmq" event={"ID":"45d89e4a-75ca-45b8-bab9-e392b3f65efd","Type":"ContainerDied","Data":"d1c467f196568547836faae0995f9e996bc117c49c4b29c06f21c698b34b7283"} Nov 04 04:49:08.735737 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:49:08.735607 4509 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-79df8c9c48-vtqmq" Nov 04 04:49:08.735737 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:49:08.735664 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-79df8c9c48-vtqmq" event={"ID":"45d89e4a-75ca-45b8-bab9-e392b3f65efd","Type":"ContainerDied","Data":"86ab0f7f989fa3ebc35e7ee90bae06b4072249e3374cd853a59d6c459b30ef5b"} Nov 04 04:49:08.735737 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:49:08.735686 4509 scope.go:122] "RemoveContainer" containerID="d1c467f196568547836faae0995f9e996bc117c49c4b29c06f21c698b34b7283" Nov 04 04:49:08.743197 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:49:08.743181 4509 scope.go:122] "RemoveContainer" containerID="d1c467f196568547836faae0995f9e996bc117c49c4b29c06f21c698b34b7283" Nov 04 04:49:08.743464 ip-10-0-133-180 kubenswrapper[4509]: E1104 04:49:08.743444 4509 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1c467f196568547836faae0995f9e996bc117c49c4b29c06f21c698b34b7283\": container with ID starting with d1c467f196568547836faae0995f9e996bc117c49c4b29c06f21c698b34b7283 not found: ID does not exist" containerID="d1c467f196568547836faae0995f9e996bc117c49c4b29c06f21c698b34b7283" Nov 04 04:49:08.743512 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:49:08.743473 4509 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1c467f196568547836faae0995f9e996bc117c49c4b29c06f21c698b34b7283"} err="failed to get container status \"d1c467f196568547836faae0995f9e996bc117c49c4b29c06f21c698b34b7283\": rpc error: code = NotFound desc = could not find container \"d1c467f196568547836faae0995f9e996bc117c49c4b29c06f21c698b34b7283\": container with ID starting with d1c467f196568547836faae0995f9e996bc117c49c4b29c06f21c698b34b7283 not found: ID does not exist" Nov 04 04:49:08.899784 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:49:08.899755 4509 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-79df8c9c48-vtqmq"] Nov 04 04:49:08.949668 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:49:08.949644 4509 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-79df8c9c48-vtqmq"] Nov 04 04:49:10.258649 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:49:10.258618 4509 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45d89e4a-75ca-45b8-bab9-e392b3f65efd" path="/var/lib/kubelet/pods/45d89e4a-75ca-45b8-bab9-e392b3f65efd/volumes" Nov 04 04:49:14.452397 ip-10-0-133-180 kubenswrapper[4509]: E1104 04:49:14.452360 4509 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmdhl" containerName="extractor" Nov 04 04:49:44.452310 ip-10-0-133-180 kubenswrapper[4509]: E1104 04:49:44.452231 4509 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmdhl" containerName="extractor" Nov 04 04:50:14.453147 ip-10-0-133-180 kubenswrapper[4509]: E1104 04:50:14.453104 4509 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmdhl" containerName="extractor" Nov 04 04:50:44.453148 ip-10-0-133-180 kubenswrapper[4509]: E1104 04:50:44.453114 4509 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmdhl" containerName="extractor" Nov 04 04:50:52.309369 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:50:52.309337 4509 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651chrrjb"] Nov 04 04:50:52.309813 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:50:52.309584 4509 cpu_manager.go:414] "containerMap: removing container" podUID="45d89e4a-75ca-45b8-bab9-e392b3f65efd" containerName="console" Nov 04 04:50:52.309813 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:50:52.309594 4509 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45d89e4a-75ca-45b8-bab9-e392b3f65efd" containerName="console" Nov 04 04:50:52.309813 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:50:52.309645 4509 memory_manager.go:365] "RemoveStaleState removing state" podUID="45d89e4a-75ca-45b8-bab9-e392b3f65efd" containerName="console" Nov 04 04:50:52.311288 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:50:52.311273 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651chrrjb" Nov 04 04:50:52.414612 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:50:52.414578 4509 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Nov 04 04:50:52.414612 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:50:52.414581 4509 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-smwxx\"" Nov 04 04:50:52.414809 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:50:52.414663 4509 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Nov 04 04:50:52.415693 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:50:52.415674 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651chrrjb"] Nov 04 04:50:52.462159 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:50:52.462132 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a922ad13-7bab-4b26-b6e4-459c340b8256-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651chrrjb\" (UID: \"a922ad13-7bab-4b26-b6e4-459c340b8256\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651chrrjb" Nov 04 04:50:52.462252 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:50:52.462159 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a922ad13-7bab-4b26-b6e4-459c340b8256-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651chrrjb\" (UID: \"a922ad13-7bab-4b26-b6e4-459c340b8256\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651chrrjb" Nov 04 04:50:52.462290 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:50:52.462267 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkbc8\" (UniqueName: \"kubernetes.io/projected/a922ad13-7bab-4b26-b6e4-459c340b8256-kube-api-access-xkbc8\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651chrrjb\" (UID: \"a922ad13-7bab-4b26-b6e4-459c340b8256\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651chrrjb" Nov 04 04:50:52.562861 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:50:52.562770 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a922ad13-7bab-4b26-b6e4-459c340b8256-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651chrrjb\" (UID: \"a922ad13-7bab-4b26-b6e4-459c340b8256\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651chrrjb" Nov 04 04:50:52.562861 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:50:52.562808 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a922ad13-7bab-4b26-b6e4-459c340b8256-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651chrrjb\" (UID: \"a922ad13-7bab-4b26-b6e4-459c340b8256\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651chrrjb" Nov 04 04:50:52.562861 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:50:52.562849 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkbc8\" (UniqueName: \"kubernetes.io/projected/a922ad13-7bab-4b26-b6e4-459c340b8256-kube-api-access-xkbc8\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651chrrjb\" (UID: \"a922ad13-7bab-4b26-b6e4-459c340b8256\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651chrrjb" Nov 04 04:50:52.563260 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:50:52.563235 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a922ad13-7bab-4b26-b6e4-459c340b8256-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651chrrjb\" (UID: \"a922ad13-7bab-4b26-b6e4-459c340b8256\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651chrrjb" Nov 04 04:50:52.563308 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:50:52.563292 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a922ad13-7bab-4b26-b6e4-459c340b8256-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651chrrjb\" (UID: \"a922ad13-7bab-4b26-b6e4-459c340b8256\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651chrrjb" Nov 04 04:50:52.620229 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:50:52.620195 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkbc8\" (UniqueName: \"kubernetes.io/projected/a922ad13-7bab-4b26-b6e4-459c340b8256-kube-api-access-xkbc8\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651chrrjb\" (UID: \"a922ad13-7bab-4b26-b6e4-459c340b8256\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651chrrjb" Nov 04 04:50:52.624810 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:50:52.624784 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651chrrjb" Nov 04 04:50:52.674082 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:50:52.674019 4509 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-g8rd9"] Nov 04 04:50:52.677275 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:50:52.677256 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-g8rd9" Nov 04 04:50:52.735105 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:50:52.735081 4509 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 04 04:50:52.735284 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:50:52.735202 4509 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 04 04:50:52.735284 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:50:52.735237 4509 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 04 04:50:52.739534 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:50:52.739509 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-g8rd9"] Nov 04 04:50:52.765345 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:50:52.765290 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce33eec0-7da6-4daa-a122-3b14aab546bb-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-g8rd9\" (UID: \"ce33eec0-7da6-4daa-a122-3b14aab546bb\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-g8rd9" Nov 04 04:50:52.765465 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:50:52.765356 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce33eec0-7da6-4daa-a122-3b14aab546bb-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-g8rd9\" (UID: \"ce33eec0-7da6-4daa-a122-3b14aab546bb\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-g8rd9" Nov 04 04:50:52.765465 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:50:52.765448 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ce33eec0-7da6-4daa-a122-3b14aab546bb-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-g8rd9\" (UID: \"ce33eec0-7da6-4daa-a122-3b14aab546bb\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-g8rd9" Nov 04 04:50:52.765557 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:50:52.765477 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ce33eec0-7da6-4daa-a122-3b14aab546bb-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-g8rd9\" (UID: \"ce33eec0-7da6-4daa-a122-3b14aab546bb\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-g8rd9" Nov 04 04:50:52.765557 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:50:52.765499 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ce33eec0-7da6-4daa-a122-3b14aab546bb-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-g8rd9\" (UID: \"ce33eec0-7da6-4daa-a122-3b14aab546bb\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-g8rd9" Nov 04 04:50:52.765662 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:50:52.765644 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ce33eec0-7da6-4daa-a122-3b14aab546bb-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-g8rd9\" (UID: \"ce33eec0-7da6-4daa-a122-3b14aab546bb\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-g8rd9" Nov 04 04:50:52.765718 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:50:52.765681 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98ckb\" (UniqueName: \"kubernetes.io/projected/ce33eec0-7da6-4daa-a122-3b14aab546bb-kube-api-access-98ckb\") pod \"istiod-openshift-gateway-7876bfcb74-g8rd9\" (UID: \"ce33eec0-7da6-4daa-a122-3b14aab546bb\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-g8rd9" Nov 04 04:50:52.825258 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:50:52.825163 4509 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Nov 04 04:50:52.825258 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:50:52.825215 4509 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Nov 04 04:50:52.825258 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:50:52.825255 4509 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Nov 04 04:50:52.825522 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:50:52.825377 4509 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-v5p6h\"" Nov 04 04:50:52.866680 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:50:52.866647 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ce33eec0-7da6-4daa-a122-3b14aab546bb-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-g8rd9\" (UID: \"ce33eec0-7da6-4daa-a122-3b14aab546bb\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-g8rd9" Nov 04 04:50:52.866860 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:50:52.866683 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98ckb\" (UniqueName: \"kubernetes.io/projected/ce33eec0-7da6-4daa-a122-3b14aab546bb-kube-api-access-98ckb\") pod \"istiod-openshift-gateway-7876bfcb74-g8rd9\" (UID: \"ce33eec0-7da6-4daa-a122-3b14aab546bb\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-g8rd9" Nov 04 04:50:52.866860 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:50:52.866709 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce33eec0-7da6-4daa-a122-3b14aab546bb-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-g8rd9\" (UID: \"ce33eec0-7da6-4daa-a122-3b14aab546bb\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-g8rd9" Nov 04 04:50:52.866860 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:50:52.866728 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce33eec0-7da6-4daa-a122-3b14aab546bb-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-g8rd9\" (UID: \"ce33eec0-7da6-4daa-a122-3b14aab546bb\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-g8rd9" Nov 04 04:50:52.867010 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:50:52.866863 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ce33eec0-7da6-4daa-a122-3b14aab546bb-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-g8rd9\" (UID: \"ce33eec0-7da6-4daa-a122-3b14aab546bb\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-g8rd9" Nov 04 04:50:52.867010 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:50:52.866902 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ce33eec0-7da6-4daa-a122-3b14aab546bb-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-g8rd9\" (UID: \"ce33eec0-7da6-4daa-a122-3b14aab546bb\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-g8rd9" Nov 04 04:50:52.867010 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:50:52.866938 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ce33eec0-7da6-4daa-a122-3b14aab546bb-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-g8rd9\" (UID: \"ce33eec0-7da6-4daa-a122-3b14aab546bb\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-g8rd9" Nov 04 04:50:52.867767 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:50:52.867740 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ce33eec0-7da6-4daa-a122-3b14aab546bb-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-g8rd9\" (UID: \"ce33eec0-7da6-4daa-a122-3b14aab546bb\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-g8rd9" Nov 04 04:50:52.869110 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:50:52.869085 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce33eec0-7da6-4daa-a122-3b14aab546bb-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-g8rd9\" (UID: \"ce33eec0-7da6-4daa-a122-3b14aab546bb\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-g8rd9" Nov 04 04:50:52.869303 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:50:52.869285 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ce33eec0-7da6-4daa-a122-3b14aab546bb-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-g8rd9\" (UID: \"ce33eec0-7da6-4daa-a122-3b14aab546bb\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-g8rd9" Nov 04 04:50:52.869382 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:50:52.869294 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ce33eec0-7da6-4daa-a122-3b14aab546bb-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-g8rd9\" (UID: \"ce33eec0-7da6-4daa-a122-3b14aab546bb\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-g8rd9" Nov 04 04:50:52.869382 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:50:52.869364 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ce33eec0-7da6-4daa-a122-3b14aab546bb-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-g8rd9\" (UID: \"ce33eec0-7da6-4daa-a122-3b14aab546bb\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-g8rd9" Nov 04 04:50:52.924501 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:50:52.924459 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce33eec0-7da6-4daa-a122-3b14aab546bb-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-g8rd9\" (UID: \"ce33eec0-7da6-4daa-a122-3b14aab546bb\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-g8rd9" Nov 04 04:50:52.924628 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:50:52.924510 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98ckb\" (UniqueName: \"kubernetes.io/projected/ce33eec0-7da6-4daa-a122-3b14aab546bb-kube-api-access-98ckb\") pod \"istiod-openshift-gateway-7876bfcb74-g8rd9\" (UID: \"ce33eec0-7da6-4daa-a122-3b14aab546bb\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-g8rd9" Nov 04 04:50:52.992205 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:50:52.992174 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-g8rd9" Nov 04 04:50:52.997046 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:50:52.997023 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651chrrjb"] Nov 04 04:50:53.000126 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:50:53.000104 4509 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda922ad13_7bab_4b26_b6e4_459c340b8256.slice/crio-44b645b6dac6e2e29c9f8f809222c222967c5fc323d96f168327ebe43a017a05 WatchSource:0}: Error finding container 44b645b6dac6e2e29c9f8f809222c222967c5fc323d96f168327ebe43a017a05: Status 404 returned error can't find the container with id 44b645b6dac6e2e29c9f8f809222c222967c5fc323d96f168327ebe43a017a05 Nov 04 04:50:53.067029 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:50:53.066995 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651chrrjb" event={"ID":"a922ad13-7bab-4b26-b6e4-459c340b8256","Type":"ContainerStarted","Data":"44b645b6dac6e2e29c9f8f809222c222967c5fc323d96f168327ebe43a017a05"} Nov 04 04:50:53.360971 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:50:53.360938 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-g8rd9"] Nov 04 04:50:53.364790 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:50:53.364764 4509 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce33eec0_7da6_4daa_a122_3b14aab546bb.slice/crio-2500b21e560f4ffbe0b8e2425204365fb5e1f7ca774e1f3fb07d1196a97a16e3 WatchSource:0}: Error finding container 2500b21e560f4ffbe0b8e2425204365fb5e1f7ca774e1f3fb07d1196a97a16e3: Status 404 returned error can't find the container with id 2500b21e560f4ffbe0b8e2425204365fb5e1f7ca774e1f3fb07d1196a97a16e3 Nov 04 04:50:54.072403 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:50:54.072358 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-g8rd9" event={"ID":"ce33eec0-7da6-4daa-a122-3b14aab546bb","Type":"ContainerStarted","Data":"2500b21e560f4ffbe0b8e2425204365fb5e1f7ca774e1f3fb07d1196a97a16e3"} Nov 04 04:50:57.478339 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:50:57.478286 4509 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 04 04:50:57.478708 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:50:57.478379 4509 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 04 04:50:58.091843 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:50:58.091799 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-g8rd9" event={"ID":"ce33eec0-7da6-4daa-a122-3b14aab546bb","Type":"ContainerStarted","Data":"1451d99d41659d2a36809a9936e812aec4d65f72bd8f62ee5ec10850eb039bd1"} Nov 04 04:50:58.092049 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:50:58.092017 4509 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-g8rd9" Nov 04 04:50:58.093880 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:50:58.093843 4509 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-g8rd9 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 04:50:58.094022 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:50:58.093909 4509 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-g8rd9" podUID="ce33eec0-7da6-4daa-a122-3b14aab546bb" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 04:50:58.301791 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:50:58.301724 4509 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-g8rd9" podStartSLOduration=2.190844945 podStartE2EDuration="6.301702603s" totalImagesPullingTime="4.110857658s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 04:50:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:50:58.259268599 +0000 UTC m=+222.136992119" watchObservedRunningTime="2026-11-04 04:50:58.301702603 +0000 UTC m=+222.179426126" Nov 04 04:50:59.096381 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:50:59.096345 4509 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-g8rd9 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 04:50:59.096812 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:50:59.096420 4509 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-g8rd9" podUID="ce33eec0-7da6-4daa-a122-3b14aab546bb" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 04:51:00.100838 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:51:00.100805 4509 generic.go:356] "Generic (PLEG): container finished" podID="a922ad13-7bab-4b26-b6e4-459c340b8256" containerID="997bcd2705f16c6f86bd4f52d9679f7934cc663ee0a0254a01a9dace57116947" exitCode=0 Nov 04 04:51:00.101223 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:51:00.100873 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651chrrjb" event={"ID":"a922ad13-7bab-4b26-b6e4-459c340b8256","Type":"ContainerDied","Data":"997bcd2705f16c6f86bd4f52d9679f7934cc663ee0a0254a01a9dace57116947"} Nov 04 04:51:02.096505 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:51:02.096470 4509 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-g8rd9" Nov 04 04:51:04.116240 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:51:04.116204 4509 generic.go:356] "Generic (PLEG): container finished" podID="a922ad13-7bab-4b26-b6e4-459c340b8256" containerID="2e4338d34c02faaa4bc66ac7a2c3e7360b23c43064fcd2c1418fb7d3fa8cdc5a" exitCode=0 Nov 04 04:51:04.116726 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:51:04.116256 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651chrrjb" event={"ID":"a922ad13-7bab-4b26-b6e4-459c340b8256","Type":"ContainerDied","Data":"2e4338d34c02faaa4bc66ac7a2c3e7360b23c43064fcd2c1418fb7d3fa8cdc5a"} Nov 04 04:51:11.144217 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:51:11.144189 4509 generic.go:356] "Generic (PLEG): container finished" podID="a922ad13-7bab-4b26-b6e4-459c340b8256" containerID="7c62270369de7b12ccb7e1cfb1bdcd31e48299a19f190149a590c277f77457db" exitCode=0 Nov 04 04:51:11.144549 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:51:11.144245 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651chrrjb" event={"ID":"a922ad13-7bab-4b26-b6e4-459c340b8256","Type":"ContainerDied","Data":"7c62270369de7b12ccb7e1cfb1bdcd31e48299a19f190149a590c277f77457db"} Nov 04 04:51:12.259956 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:51:12.259933 4509 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651chrrjb" Nov 04 04:51:12.428986 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:51:12.428905 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a922ad13-7bab-4b26-b6e4-459c340b8256-kube-api-access-xkbc8\" (UniqueName: \"kubernetes.io/projected/a922ad13-7bab-4b26-b6e4-459c340b8256-kube-api-access-xkbc8\") pod \"a922ad13-7bab-4b26-b6e4-459c340b8256\" (UID: \"a922ad13-7bab-4b26-b6e4-459c340b8256\") " Nov 04 04:51:12.429137 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:51:12.428995 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a922ad13-7bab-4b26-b6e4-459c340b8256-util\" (UniqueName: \"kubernetes.io/empty-dir/a922ad13-7bab-4b26-b6e4-459c340b8256-util\") pod \"a922ad13-7bab-4b26-b6e4-459c340b8256\" (UID: \"a922ad13-7bab-4b26-b6e4-459c340b8256\") " Nov 04 04:51:12.429137 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:51:12.429014 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a922ad13-7bab-4b26-b6e4-459c340b8256-bundle\" (UniqueName: \"kubernetes.io/empty-dir/a922ad13-7bab-4b26-b6e4-459c340b8256-bundle\") pod \"a922ad13-7bab-4b26-b6e4-459c340b8256\" (UID: \"a922ad13-7bab-4b26-b6e4-459c340b8256\") " Nov 04 04:51:12.429978 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:51:12.429948 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a922ad13-7bab-4b26-b6e4-459c340b8256-bundle" pod "a922ad13-7bab-4b26-b6e4-459c340b8256" (UID: "a922ad13-7bab-4b26-b6e4-459c340b8256"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 04:51:12.431353 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:51:12.431294 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a922ad13-7bab-4b26-b6e4-459c340b8256-kube-api-access-xkbc8" pod "a922ad13-7bab-4b26-b6e4-459c340b8256" (UID: "a922ad13-7bab-4b26-b6e4-459c340b8256"). InnerVolumeSpecName "kube-api-access-xkbc8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 04:51:12.433355 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:51:12.433315 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a922ad13-7bab-4b26-b6e4-459c340b8256-util" pod "a922ad13-7bab-4b26-b6e4-459c340b8256" (UID: "a922ad13-7bab-4b26-b6e4-459c340b8256"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 04:51:12.529890 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:51:12.529867 4509 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkbc8\" (UniqueName: \"kubernetes.io/projected/a922ad13-7bab-4b26-b6e4-459c340b8256-kube-api-access-xkbc8\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 04:51:12.529890 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:51:12.529888 4509 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a922ad13-7bab-4b26-b6e4-459c340b8256-util\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 04:51:12.530003 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:51:12.529897 4509 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a922ad13-7bab-4b26-b6e4-459c340b8256-bundle\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 04:51:13.152602 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:51:13.152573 4509 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651chrrjb" Nov 04 04:51:13.152602 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:51:13.152580 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651chrrjb" event={"ID":"a922ad13-7bab-4b26-b6e4-459c340b8256","Type":"ContainerDied","Data":"44b645b6dac6e2e29c9f8f809222c222967c5fc323d96f168327ebe43a017a05"} Nov 04 04:51:13.152790 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:51:13.152621 4509 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="44b645b6dac6e2e29c9f8f809222c222967c5fc323d96f168327ebe43a017a05" Nov 04 04:51:14.452523 ip-10-0-133-180 kubenswrapper[4509]: E1104 04:51:14.452487 4509 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmdhl" containerName="extractor" Nov 04 04:51:44.454359 ip-10-0-133-180 kubenswrapper[4509]: E1104 04:51:44.454273 4509 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmdhl" containerName="extractor" Nov 04 04:51:49.319529 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:51:49.319492 4509 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-84f7664b48-r9wgb"] Nov 04 04:51:49.319905 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:51:49.319750 4509 cpu_manager.go:414] "containerMap: removing container" podUID="a922ad13-7bab-4b26-b6e4-459c340b8256" containerName="pull" Nov 04 04:51:49.319905 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:51:49.319760 4509 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a922ad13-7bab-4b26-b6e4-459c340b8256" containerName="pull" Nov 04 04:51:49.319905 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:51:49.319776 4509 cpu_manager.go:414] "containerMap: removing container" podUID="a922ad13-7bab-4b26-b6e4-459c340b8256" containerName="util" Nov 04 04:51:49.319905 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:51:49.319781 4509 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a922ad13-7bab-4b26-b6e4-459c340b8256" containerName="util" Nov 04 04:51:49.319905 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:51:49.319787 4509 cpu_manager.go:414] "containerMap: removing container" podUID="a922ad13-7bab-4b26-b6e4-459c340b8256" containerName="extract" Nov 04 04:51:49.319905 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:51:49.319792 4509 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a922ad13-7bab-4b26-b6e4-459c340b8256" containerName="extract" Nov 04 04:51:49.319905 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:51:49.319833 4509 memory_manager.go:365] "RemoveStaleState removing state" podUID="a922ad13-7bab-4b26-b6e4-459c340b8256" containerName="extract" Nov 04 04:51:49.322318 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:51:49.322303 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84f7664b48-r9wgb" Nov 04 04:51:49.375533 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:51:49.375502 4509 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Nov 04 04:51:49.375778 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:51:49.375745 4509 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Nov 04 04:51:49.375942 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:51:49.375927 4509 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-nj9j7\"" Nov 04 04:51:49.375984 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:51:49.375956 4509 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Nov 04 04:51:49.379111 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:51:49.379092 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fa7ec6ae-73ac-4a28-b26f-98c3c446428e-webhook-cert\") pod \"opendatahub-operator-controller-manager-84f7664b48-r9wgb\" (UID: \"fa7ec6ae-73ac-4a28-b26f-98c3c446428e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84f7664b48-r9wgb" Nov 04 04:51:49.379148 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:51:49.379132 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ddjs\" (UniqueName: \"kubernetes.io/projected/fa7ec6ae-73ac-4a28-b26f-98c3c446428e-kube-api-access-9ddjs\") pod \"opendatahub-operator-controller-manager-84f7664b48-r9wgb\" (UID: \"fa7ec6ae-73ac-4a28-b26f-98c3c446428e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84f7664b48-r9wgb" Nov 04 04:51:49.379192 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:51:49.379163 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fa7ec6ae-73ac-4a28-b26f-98c3c446428e-tmp\") pod \"opendatahub-operator-controller-manager-84f7664b48-r9wgb\" (UID: \"fa7ec6ae-73ac-4a28-b26f-98c3c446428e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84f7664b48-r9wgb" Nov 04 04:51:49.379263 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:51:49.379246 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/fa7ec6ae-73ac-4a28-b26f-98c3c446428e-metrics-cert\") pod \"opendatahub-operator-controller-manager-84f7664b48-r9wgb\" (UID: \"fa7ec6ae-73ac-4a28-b26f-98c3c446428e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84f7664b48-r9wgb" Nov 04 04:51:49.379299 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:51:49.379278 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/fa7ec6ae-73ac-4a28-b26f-98c3c446428e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-84f7664b48-r9wgb\" (UID: \"fa7ec6ae-73ac-4a28-b26f-98c3c446428e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84f7664b48-r9wgb" Nov 04 04:51:49.379358 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:51:49.379299 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/fa7ec6ae-73ac-4a28-b26f-98c3c446428e-manifests\") pod \"opendatahub-operator-controller-manager-84f7664b48-r9wgb\" (UID: \"fa7ec6ae-73ac-4a28-b26f-98c3c446428e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84f7664b48-r9wgb" Nov 04 04:51:49.427904 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:51:49.427877 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-84f7664b48-r9wgb"] Nov 04 04:51:49.437001 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:51:49.436982 4509 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Nov 04 04:51:49.437098 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:51:49.437017 4509 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Nov 04 04:51:49.480080 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:51:49.480049 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fa7ec6ae-73ac-4a28-b26f-98c3c446428e-webhook-cert\") pod \"opendatahub-operator-controller-manager-84f7664b48-r9wgb\" (UID: \"fa7ec6ae-73ac-4a28-b26f-98c3c446428e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84f7664b48-r9wgb" Nov 04 04:51:49.480187 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:51:49.480107 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ddjs\" (UniqueName: \"kubernetes.io/projected/fa7ec6ae-73ac-4a28-b26f-98c3c446428e-kube-api-access-9ddjs\") pod \"opendatahub-operator-controller-manager-84f7664b48-r9wgb\" (UID: \"fa7ec6ae-73ac-4a28-b26f-98c3c446428e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84f7664b48-r9wgb" Nov 04 04:51:49.480187 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:51:49.480146 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fa7ec6ae-73ac-4a28-b26f-98c3c446428e-tmp\") pod \"opendatahub-operator-controller-manager-84f7664b48-r9wgb\" (UID: \"fa7ec6ae-73ac-4a28-b26f-98c3c446428e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84f7664b48-r9wgb" Nov 04 04:51:49.480187 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:51:49.480168 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/fa7ec6ae-73ac-4a28-b26f-98c3c446428e-metrics-cert\") pod \"opendatahub-operator-controller-manager-84f7664b48-r9wgb\" (UID: \"fa7ec6ae-73ac-4a28-b26f-98c3c446428e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84f7664b48-r9wgb" Nov 04 04:51:49.480298 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:51:49.480271 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/fa7ec6ae-73ac-4a28-b26f-98c3c446428e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-84f7664b48-r9wgb\" (UID: \"fa7ec6ae-73ac-4a28-b26f-98c3c446428e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84f7664b48-r9wgb" Nov 04 04:51:49.480368 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:51:49.480317 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/fa7ec6ae-73ac-4a28-b26f-98c3c446428e-manifests\") pod \"opendatahub-operator-controller-manager-84f7664b48-r9wgb\" (UID: \"fa7ec6ae-73ac-4a28-b26f-98c3c446428e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84f7664b48-r9wgb" Nov 04 04:51:49.480723 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:51:49.480672 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/fa7ec6ae-73ac-4a28-b26f-98c3c446428e-manifests\") pod \"opendatahub-operator-controller-manager-84f7664b48-r9wgb\" (UID: \"fa7ec6ae-73ac-4a28-b26f-98c3c446428e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84f7664b48-r9wgb" Nov 04 04:51:49.482476 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:51:49.482454 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fa7ec6ae-73ac-4a28-b26f-98c3c446428e-tmp\") pod \"opendatahub-operator-controller-manager-84f7664b48-r9wgb\" (UID: \"fa7ec6ae-73ac-4a28-b26f-98c3c446428e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84f7664b48-r9wgb" Nov 04 04:51:49.482742 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:51:49.482723 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/fa7ec6ae-73ac-4a28-b26f-98c3c446428e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-84f7664b48-r9wgb\" (UID: \"fa7ec6ae-73ac-4a28-b26f-98c3c446428e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84f7664b48-r9wgb" Nov 04 04:51:49.482782 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:51:49.482759 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fa7ec6ae-73ac-4a28-b26f-98c3c446428e-webhook-cert\") pod \"opendatahub-operator-controller-manager-84f7664b48-r9wgb\" (UID: \"fa7ec6ae-73ac-4a28-b26f-98c3c446428e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84f7664b48-r9wgb" Nov 04 04:51:49.482828 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:51:49.482759 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/fa7ec6ae-73ac-4a28-b26f-98c3c446428e-metrics-cert\") pod \"opendatahub-operator-controller-manager-84f7664b48-r9wgb\" (UID: \"fa7ec6ae-73ac-4a28-b26f-98c3c446428e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84f7664b48-r9wgb" Nov 04 04:51:49.564797 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:51:49.564769 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ddjs\" (UniqueName: \"kubernetes.io/projected/fa7ec6ae-73ac-4a28-b26f-98c3c446428e-kube-api-access-9ddjs\") pod \"opendatahub-operator-controller-manager-84f7664b48-r9wgb\" (UID: \"fa7ec6ae-73ac-4a28-b26f-98c3c446428e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84f7664b48-r9wgb" Nov 04 04:51:49.634448 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:51:49.634363 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84f7664b48-r9wgb" Nov 04 04:51:50.189488 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:51:50.189466 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-84f7664b48-r9wgb"] Nov 04 04:51:50.191584 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:51:50.191558 4509 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa7ec6ae_73ac_4a28_b26f_98c3c446428e.slice/crio-a9cce4625d2335af5943ced4be2d69b21960f627b4a2d52cf183c14034f7f740 WatchSource:0}: Error finding container a9cce4625d2335af5943ced4be2d69b21960f627b4a2d52cf183c14034f7f740: Status 404 returned error can't find the container with id a9cce4625d2335af5943ced4be2d69b21960f627b4a2d52cf183c14034f7f740 Nov 04 04:51:50.273306 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:51:50.273276 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84f7664b48-r9wgb" event={"ID":"fa7ec6ae-73ac-4a28-b26f-98c3c446428e","Type":"ContainerStarted","Data":"a9cce4625d2335af5943ced4be2d69b21960f627b4a2d52cf183c14034f7f740"} Nov 04 04:51:54.290697 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:51:54.290664 4509 generic.go:356] "Generic (PLEG): container finished" podID="fa7ec6ae-73ac-4a28-b26f-98c3c446428e" containerID="1fb6ec5142251d04c381de75151bcd9a32cb6c721531cb918c6e4415e1bec0d2" exitCode=0 Nov 04 04:51:54.291071 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:51:54.290743 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84f7664b48-r9wgb" event={"ID":"fa7ec6ae-73ac-4a28-b26f-98c3c446428e","Type":"ContainerDied","Data":"1fb6ec5142251d04c381de75151bcd9a32cb6c721531cb918c6e4415e1bec0d2"} Nov 04 04:51:55.296334 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:51:55.296285 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84f7664b48-r9wgb" event={"ID":"fa7ec6ae-73ac-4a28-b26f-98c3c446428e","Type":"ContainerStarted","Data":"8087578144d972a7caba63793a76bd4e4d1c2d3661c510a936c2d6dfabeabf9f"} Nov 04 04:51:55.296783 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:51:55.296435 4509 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84f7664b48-r9wgb" Nov 04 04:51:55.529011 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:51:55.528962 4509 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84f7664b48-r9wgb" podStartSLOduration=2.028379844 podStartE2EDuration="6.528942026s" totalImagesPullingTime="4.067483294s" totalInitContainerRuntime="433.078888ms" isStatefulPod=false podCreationTimestamp="2026-11-04 04:51:49 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:51:55.478537661 +0000 UTC m=+279.356261182" watchObservedRunningTime="2026-11-04 04:51:55.528942026 +0000 UTC m=+279.406665549" Nov 04 04:52:06.301677 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:52:06.301646 4509 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84f7664b48-r9wgb" Nov 04 04:52:14.454378 ip-10-0-133-180 kubenswrapper[4509]: E1104 04:52:14.454333 4509 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmdhl" containerName="extractor" Nov 04 04:52:20.442556 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:52:20.442520 4509 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-546d78d5c-vbwrt"] Nov 04 04:52:20.448135 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:52:20.448116 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-546d78d5c-vbwrt" Nov 04 04:52:20.502875 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:52:20.502844 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-546d78d5c-vbwrt"] Nov 04 04:52:20.517779 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:52:20.517754 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddx9l\" (UniqueName: \"kubernetes.io/projected/542ce577-5ec5-400e-a82b-d5d1f3fe8c97-kube-api-access-ddx9l\") pod \"kube-auth-proxy-546d78d5c-vbwrt\" (UID: \"542ce577-5ec5-400e-a82b-d5d1f3fe8c97\") " pod="openshift-ingress/kube-auth-proxy-546d78d5c-vbwrt" Nov 04 04:52:20.517949 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:52:20.517789 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/542ce577-5ec5-400e-a82b-d5d1f3fe8c97-tls-certs\") pod \"kube-auth-proxy-546d78d5c-vbwrt\" (UID: \"542ce577-5ec5-400e-a82b-d5d1f3fe8c97\") " pod="openshift-ingress/kube-auth-proxy-546d78d5c-vbwrt" Nov 04 04:52:20.517949 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:52:20.517835 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/542ce577-5ec5-400e-a82b-d5d1f3fe8c97-tmp\") pod \"kube-auth-proxy-546d78d5c-vbwrt\" (UID: \"542ce577-5ec5-400e-a82b-d5d1f3fe8c97\") " pod="openshift-ingress/kube-auth-proxy-546d78d5c-vbwrt" Nov 04 04:52:20.551028 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:52:20.551003 4509 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Nov 04 04:52:20.551028 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:52:20.551034 4509 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-w6k6c\"" Nov 04 04:52:20.551238 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:52:20.551112 4509 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Nov 04 04:52:20.618388 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:52:20.618356 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddx9l\" (UniqueName: \"kubernetes.io/projected/542ce577-5ec5-400e-a82b-d5d1f3fe8c97-kube-api-access-ddx9l\") pod \"kube-auth-proxy-546d78d5c-vbwrt\" (UID: \"542ce577-5ec5-400e-a82b-d5d1f3fe8c97\") " pod="openshift-ingress/kube-auth-proxy-546d78d5c-vbwrt" Nov 04 04:52:20.618558 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:52:20.618397 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/542ce577-5ec5-400e-a82b-d5d1f3fe8c97-tls-certs\") pod \"kube-auth-proxy-546d78d5c-vbwrt\" (UID: \"542ce577-5ec5-400e-a82b-d5d1f3fe8c97\") " pod="openshift-ingress/kube-auth-proxy-546d78d5c-vbwrt" Nov 04 04:52:20.618558 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:52:20.618433 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/542ce577-5ec5-400e-a82b-d5d1f3fe8c97-tmp\") pod \"kube-auth-proxy-546d78d5c-vbwrt\" (UID: \"542ce577-5ec5-400e-a82b-d5d1f3fe8c97\") " pod="openshift-ingress/kube-auth-proxy-546d78d5c-vbwrt" Nov 04 04:52:20.620762 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:52:20.620738 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/542ce577-5ec5-400e-a82b-d5d1f3fe8c97-tmp\") pod \"kube-auth-proxy-546d78d5c-vbwrt\" (UID: \"542ce577-5ec5-400e-a82b-d5d1f3fe8c97\") " pod="openshift-ingress/kube-auth-proxy-546d78d5c-vbwrt" Nov 04 04:52:20.620978 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:52:20.620960 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/542ce577-5ec5-400e-a82b-d5d1f3fe8c97-tls-certs\") pod \"kube-auth-proxy-546d78d5c-vbwrt\" (UID: \"542ce577-5ec5-400e-a82b-d5d1f3fe8c97\") " pod="openshift-ingress/kube-auth-proxy-546d78d5c-vbwrt" Nov 04 04:52:20.675048 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:52:20.675019 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddx9l\" (UniqueName: \"kubernetes.io/projected/542ce577-5ec5-400e-a82b-d5d1f3fe8c97-kube-api-access-ddx9l\") pod \"kube-auth-proxy-546d78d5c-vbwrt\" (UID: \"542ce577-5ec5-400e-a82b-d5d1f3fe8c97\") " pod="openshift-ingress/kube-auth-proxy-546d78d5c-vbwrt" Nov 04 04:52:20.761541 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:52:20.761509 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-546d78d5c-vbwrt" Nov 04 04:52:21.136112 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:52:21.136031 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-546d78d5c-vbwrt"] Nov 04 04:52:21.139822 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:52:21.139793 4509 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod542ce577_5ec5_400e_a82b_d5d1f3fe8c97.slice/crio-4614ddfbd409dbeda3ec4d8a5f2176bb8d0f53639020a2118deb768931dc86df WatchSource:0}: Error finding container 4614ddfbd409dbeda3ec4d8a5f2176bb8d0f53639020a2118deb768931dc86df: Status 404 returned error can't find the container with id 4614ddfbd409dbeda3ec4d8a5f2176bb8d0f53639020a2118deb768931dc86df Nov 04 04:52:21.141931 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:52:21.141910 4509 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 04 04:52:21.388701 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:52:21.388619 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-546d78d5c-vbwrt" event={"ID":"542ce577-5ec5-400e-a82b-d5d1f3fe8c97","Type":"ContainerStarted","Data":"4614ddfbd409dbeda3ec4d8a5f2176bb8d0f53639020a2118deb768931dc86df"} Nov 04 04:52:25.406293 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:52:25.406251 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-546d78d5c-vbwrt" event={"ID":"542ce577-5ec5-400e-a82b-d5d1f3fe8c97","Type":"ContainerStarted","Data":"4f92481d4726ddd1c5da11d8f27b546e25505046cbeaf1a38d9713ba2f3fc0dd"} Nov 04 04:52:25.520114 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:52:25.520059 4509 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-546d78d5c-vbwrt" podStartSLOduration=2.281029735 podStartE2EDuration="5.520044022s" totalImagesPullingTime="3.239014287s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 04:52:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:52:25.518897515 +0000 UTC m=+309.396621029" watchObservedRunningTime="2026-11-04 04:52:25.520044022 +0000 UTC m=+309.397767580" Nov 04 04:52:40.514579 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:52:40.514482 4509 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-p4rkn"] Nov 04 04:52:40.518661 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:52:40.518633 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-p4rkn" Nov 04 04:52:40.572199 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:52:40.572126 4509 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-zfb5x\"" Nov 04 04:52:40.573578 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:52:40.573550 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-p4rkn"] Nov 04 04:52:40.621971 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:52:40.621931 4509 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 04 04:52:40.621971 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:52:40.621961 4509 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 04 04:52:40.688022 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:52:40.687993 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjxw2\" (UniqueName: \"kubernetes.io/projected/df6aa8aa-5fee-42cb-a41c-1934516cccc1-kube-api-access-tjxw2\") pod \"authorino-57f7f8659f-p4rkn\" (UID: \"df6aa8aa-5fee-42cb-a41c-1934516cccc1\") " pod="kuadrant-system/authorino-57f7f8659f-p4rkn" Nov 04 04:52:40.788683 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:52:40.788611 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjxw2\" (UniqueName: \"kubernetes.io/projected/df6aa8aa-5fee-42cb-a41c-1934516cccc1-kube-api-access-tjxw2\") pod \"authorino-57f7f8659f-p4rkn\" (UID: \"df6aa8aa-5fee-42cb-a41c-1934516cccc1\") " pod="kuadrant-system/authorino-57f7f8659f-p4rkn" Nov 04 04:52:40.847295 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:52:40.847260 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjxw2\" (UniqueName: \"kubernetes.io/projected/df6aa8aa-5fee-42cb-a41c-1934516cccc1-kube-api-access-tjxw2\") pod \"authorino-57f7f8659f-p4rkn\" (UID: \"df6aa8aa-5fee-42cb-a41c-1934516cccc1\") " pod="kuadrant-system/authorino-57f7f8659f-p4rkn" Nov 04 04:52:41.145859 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:52:41.145774 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-p4rkn" Nov 04 04:52:41.511951 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:52:41.511799 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-p4rkn"] Nov 04 04:52:41.514584 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:52:41.514556 4509 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddf6aa8aa_5fee_42cb_a41c_1934516cccc1.slice/crio-7c788b123767edaee12fa10ae5e97b881b91daf5d9b3edf421cbe5d1faefe64c WatchSource:0}: Error finding container 7c788b123767edaee12fa10ae5e97b881b91daf5d9b3edf421cbe5d1faefe64c: Status 404 returned error can't find the container with id 7c788b123767edaee12fa10ae5e97b881b91daf5d9b3edf421cbe5d1faefe64c Nov 04 04:52:42.471486 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:52:42.471448 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-p4rkn" event={"ID":"df6aa8aa-5fee-42cb-a41c-1934516cccc1","Type":"ContainerStarted","Data":"7c788b123767edaee12fa10ae5e97b881b91daf5d9b3edf421cbe5d1faefe64c"} Nov 04 04:52:43.540152 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:52:43.538877 4509 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-42dnm"] Nov 04 04:52:43.542384 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:52:43.542362 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-42dnm" Nov 04 04:52:43.594783 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:52:43.594753 4509 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-m7cqj\"" Nov 04 04:52:43.644442 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:52:43.644399 4509 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Nov 04 04:52:43.645963 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:52:43.645937 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-42dnm"] Nov 04 04:52:43.711913 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:52:43.711876 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e0ffdeeb-4a8f-437b-80ab-c4c09c6e14bc-config-file\") pod \"limitador-limitador-7c9b87d574-42dnm\" (UID: \"e0ffdeeb-4a8f-437b-80ab-c4c09c6e14bc\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-42dnm" Nov 04 04:52:43.712084 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:52:43.711950 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjbnd\" (UniqueName: \"kubernetes.io/projected/e0ffdeeb-4a8f-437b-80ab-c4c09c6e14bc-kube-api-access-jjbnd\") pod \"limitador-limitador-7c9b87d574-42dnm\" (UID: \"e0ffdeeb-4a8f-437b-80ab-c4c09c6e14bc\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-42dnm" Nov 04 04:52:43.813309 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:52:43.813219 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e0ffdeeb-4a8f-437b-80ab-c4c09c6e14bc-config-file\") pod \"limitador-limitador-7c9b87d574-42dnm\" (UID: \"e0ffdeeb-4a8f-437b-80ab-c4c09c6e14bc\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-42dnm" Nov 04 04:52:43.813309 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:52:43.813289 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjbnd\" (UniqueName: \"kubernetes.io/projected/e0ffdeeb-4a8f-437b-80ab-c4c09c6e14bc-kube-api-access-jjbnd\") pod \"limitador-limitador-7c9b87d574-42dnm\" (UID: \"e0ffdeeb-4a8f-437b-80ab-c4c09c6e14bc\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-42dnm" Nov 04 04:52:43.814023 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:52:43.814001 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e0ffdeeb-4a8f-437b-80ab-c4c09c6e14bc-config-file\") pod \"limitador-limitador-7c9b87d574-42dnm\" (UID: \"e0ffdeeb-4a8f-437b-80ab-c4c09c6e14bc\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-42dnm" Nov 04 04:52:43.855106 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:52:43.855071 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-42dnm"] Nov 04 04:52:43.871462 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:52:43.871433 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjbnd\" (UniqueName: \"kubernetes.io/projected/e0ffdeeb-4a8f-437b-80ab-c4c09c6e14bc-kube-api-access-jjbnd\") pod \"limitador-limitador-7c9b87d574-42dnm\" (UID: \"e0ffdeeb-4a8f-437b-80ab-c4c09c6e14bc\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-42dnm" Nov 04 04:52:44.158723 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:52:44.158639 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-42dnm" Nov 04 04:52:44.460685 ip-10-0-133-180 kubenswrapper[4509]: E1104 04:52:44.460650 4509 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmdhl" containerName="extractor" Nov 04 04:52:44.543891 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:52:44.543861 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-42dnm"] Nov 04 04:52:44.546955 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:52:44.546929 4509 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0ffdeeb_4a8f_437b_80ab_c4c09c6e14bc.slice/crio-90a0758e5dcbcd1c798e449fad42ddb794c92291b515cdb217bd1d0bb40376e4 WatchSource:0}: Error finding container 90a0758e5dcbcd1c798e449fad42ddb794c92291b515cdb217bd1d0bb40376e4: Status 404 returned error can't find the container with id 90a0758e5dcbcd1c798e449fad42ddb794c92291b515cdb217bd1d0bb40376e4 Nov 04 04:52:45.487615 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:52:45.487578 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-42dnm" event={"ID":"e0ffdeeb-4a8f-437b-80ab-c4c09c6e14bc","Type":"ContainerStarted","Data":"90a0758e5dcbcd1c798e449fad42ddb794c92291b515cdb217bd1d0bb40376e4"} Nov 04 04:52:45.488918 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:52:45.488883 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-p4rkn" event={"ID":"df6aa8aa-5fee-42cb-a41c-1934516cccc1","Type":"ContainerStarted","Data":"5a20a346bf4da437d716c40247858fb251b2c0ea6c1d7c45b972353ba6bc78a3"} Nov 04 04:52:45.602657 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:52:45.602601 4509 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-p4rkn" podStartSLOduration=2.089430973 podStartE2EDuration="5.602586505s" totalImagesPullingTime="3.513155532s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 04:52:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:52:45.60126663 +0000 UTC m=+329.478990162" watchObservedRunningTime="2026-11-04 04:52:45.602586505 +0000 UTC m=+329.480310025" Nov 04 04:52:46.440481 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:52:46.440402 4509 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-glcvz"] Nov 04 04:52:46.443800 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:52:46.443780 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-glcvz" Nov 04 04:52:46.494140 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:52:46.494082 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-42dnm" event={"ID":"e0ffdeeb-4a8f-437b-80ab-c4c09c6e14bc","Type":"ContainerStarted","Data":"41b0c6c974a25ed9d5cb3586fa7c0e5abfff631da0e3a3276eef60a661d91831"} Nov 04 04:52:46.494355 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:52:46.494163 4509 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-42dnm" Nov 04 04:52:46.496258 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:52:46.496236 4509 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 04 04:52:46.496384 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:52:46.496272 4509 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 04 04:52:46.496384 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:52:46.496339 4509 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-cmsxk\"" Nov 04 04:52:46.538280 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:52:46.538243 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b26b2ef7-b5a7-4933-8393-f830db7d1387-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-glcvz\" (UID: \"b26b2ef7-b5a7-4933-8393-f830db7d1387\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-glcvz" Nov 04 04:52:46.538459 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:52:46.538297 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b26b2ef7-b5a7-4933-8393-f830db7d1387-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-glcvz\" (UID: \"b26b2ef7-b5a7-4933-8393-f830db7d1387\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-glcvz" Nov 04 04:52:46.538459 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:52:46.538333 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b26b2ef7-b5a7-4933-8393-f830db7d1387-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-glcvz\" (UID: \"b26b2ef7-b5a7-4933-8393-f830db7d1387\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-glcvz" Nov 04 04:52:46.538459 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:52:46.538443 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ksgbg\" (UniqueName: \"kubernetes.io/projected/b26b2ef7-b5a7-4933-8393-f830db7d1387-kube-api-access-ksgbg\") pod \"kserve-module-controller-manager-7566cf5fc8-glcvz\" (UID: \"b26b2ef7-b5a7-4933-8393-f830db7d1387\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-glcvz" Nov 04 04:52:46.639553 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:52:46.639511 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b26b2ef7-b5a7-4933-8393-f830db7d1387-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-glcvz\" (UID: \"b26b2ef7-b5a7-4933-8393-f830db7d1387\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-glcvz" Nov 04 04:52:46.640046 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:52:46.639575 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b26b2ef7-b5a7-4933-8393-f830db7d1387-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-glcvz\" (UID: \"b26b2ef7-b5a7-4933-8393-f830db7d1387\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-glcvz" Nov 04 04:52:46.640046 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:52:46.639600 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b26b2ef7-b5a7-4933-8393-f830db7d1387-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-glcvz\" (UID: \"b26b2ef7-b5a7-4933-8393-f830db7d1387\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-glcvz" Nov 04 04:52:46.640046 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:52:46.639653 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ksgbg\" (UniqueName: \"kubernetes.io/projected/b26b2ef7-b5a7-4933-8393-f830db7d1387-kube-api-access-ksgbg\") pod \"kserve-module-controller-manager-7566cf5fc8-glcvz\" (UID: \"b26b2ef7-b5a7-4933-8393-f830db7d1387\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-glcvz" Nov 04 04:52:46.640046 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:52:46.640015 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b26b2ef7-b5a7-4933-8393-f830db7d1387-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-glcvz\" (UID: \"b26b2ef7-b5a7-4933-8393-f830db7d1387\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-glcvz" Nov 04 04:52:46.640192 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:52:46.640074 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b26b2ef7-b5a7-4933-8393-f830db7d1387-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-glcvz\" (UID: \"b26b2ef7-b5a7-4933-8393-f830db7d1387\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-glcvz" Nov 04 04:52:46.646950 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:52:46.646910 4509 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\"" Nov 04 04:52:46.648177 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:52:46.648156 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-glcvz"] Nov 04 04:52:46.650803 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:52:46.650782 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b26b2ef7-b5a7-4933-8393-f830db7d1387-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-glcvz\" (UID: \"b26b2ef7-b5a7-4933-8393-f830db7d1387\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-glcvz" Nov 04 04:52:46.700211 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:52:46.700134 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ksgbg\" (UniqueName: \"kubernetes.io/projected/b26b2ef7-b5a7-4933-8393-f830db7d1387-kube-api-access-ksgbg\") pod \"kserve-module-controller-manager-7566cf5fc8-glcvz\" (UID: \"b26b2ef7-b5a7-4933-8393-f830db7d1387\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-glcvz" Nov 04 04:52:46.756072 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:52:46.756034 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-glcvz" Nov 04 04:52:46.762474 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:52:46.762428 4509 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-42dnm" podStartSLOduration=2.140842336 podStartE2EDuration="3.762413935s" totalImagesPullingTime="1.621571599s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 04:52:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:52:46.760976604 +0000 UTC m=+330.638700126" watchObservedRunningTime="2026-11-04 04:52:46.762413935 +0000 UTC m=+330.640137455" Nov 04 04:52:47.134765 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:52:47.134735 4509 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-p4rkn"] Nov 04 04:52:47.180561 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:52:47.180371 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-glcvz"] Nov 04 04:52:47.182923 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:52:47.182895 4509 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb26b2ef7_b5a7_4933_8393_f830db7d1387.slice/crio-bac4eb9b1ade89999cd0050f10604ae5896adaae843025ead9dbd10915565c5c WatchSource:0}: Error finding container bac4eb9b1ade89999cd0050f10604ae5896adaae843025ead9dbd10915565c5c: Status 404 returned error can't find the container with id bac4eb9b1ade89999cd0050f10604ae5896adaae843025ead9dbd10915565c5c Nov 04 04:52:47.499165 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:52:47.499123 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-glcvz" event={"ID":"b26b2ef7-b5a7-4933-8393-f830db7d1387","Type":"ContainerStarted","Data":"bac4eb9b1ade89999cd0050f10604ae5896adaae843025ead9dbd10915565c5c"} Nov 04 04:52:47.499366 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:52:47.499183 4509 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-p4rkn" podUID="df6aa8aa-5fee-42cb-a41c-1934516cccc1" containerName="authorino" containerID="cri-o://5a20a346bf4da437d716c40247858fb251b2c0ea6c1d7c45b972353ba6bc78a3" gracePeriod=30 Nov 04 04:52:47.737174 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:52:47.737148 4509 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-p4rkn" Nov 04 04:52:47.850184 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:52:47.850109 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df6aa8aa-5fee-42cb-a41c-1934516cccc1-kube-api-access-tjxw2\" (UniqueName: \"kubernetes.io/projected/df6aa8aa-5fee-42cb-a41c-1934516cccc1-kube-api-access-tjxw2\") pod \"df6aa8aa-5fee-42cb-a41c-1934516cccc1\" (UID: \"df6aa8aa-5fee-42cb-a41c-1934516cccc1\") " Nov 04 04:52:47.852651 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:52:47.852616 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df6aa8aa-5fee-42cb-a41c-1934516cccc1-kube-api-access-tjxw2" pod "df6aa8aa-5fee-42cb-a41c-1934516cccc1" (UID: "df6aa8aa-5fee-42cb-a41c-1934516cccc1"). InnerVolumeSpecName "kube-api-access-tjxw2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 04:52:47.950917 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:52:47.950875 4509 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjxw2\" (UniqueName: \"kubernetes.io/projected/df6aa8aa-5fee-42cb-a41c-1934516cccc1-kube-api-access-tjxw2\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 04:52:48.505167 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:52:48.505131 4509 generic.go:356] "Generic (PLEG): container finished" podID="df6aa8aa-5fee-42cb-a41c-1934516cccc1" containerID="5a20a346bf4da437d716c40247858fb251b2c0ea6c1d7c45b972353ba6bc78a3" exitCode=0 Nov 04 04:52:48.505392 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:52:48.505174 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-p4rkn" event={"ID":"df6aa8aa-5fee-42cb-a41c-1934516cccc1","Type":"ContainerDied","Data":"5a20a346bf4da437d716c40247858fb251b2c0ea6c1d7c45b972353ba6bc78a3"} Nov 04 04:52:48.505392 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:52:48.505207 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-p4rkn" event={"ID":"df6aa8aa-5fee-42cb-a41c-1934516cccc1","Type":"ContainerDied","Data":"7c788b123767edaee12fa10ae5e97b881b91daf5d9b3edf421cbe5d1faefe64c"} Nov 04 04:52:48.505392 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:52:48.505230 4509 scope.go:122] "RemoveContainer" containerID="5a20a346bf4da437d716c40247858fb251b2c0ea6c1d7c45b972353ba6bc78a3" Nov 04 04:52:48.505392 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:52:48.505381 4509 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-p4rkn" Nov 04 04:52:48.514684 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:52:48.514663 4509 scope.go:122] "RemoveContainer" containerID="5a20a346bf4da437d716c40247858fb251b2c0ea6c1d7c45b972353ba6bc78a3" Nov 04 04:52:48.514987 ip-10-0-133-180 kubenswrapper[4509]: E1104 04:52:48.514966 4509 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a20a346bf4da437d716c40247858fb251b2c0ea6c1d7c45b972353ba6bc78a3\": container with ID starting with 5a20a346bf4da437d716c40247858fb251b2c0ea6c1d7c45b972353ba6bc78a3 not found: ID does not exist" containerID="5a20a346bf4da437d716c40247858fb251b2c0ea6c1d7c45b972353ba6bc78a3" Nov 04 04:52:48.515056 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:52:48.514998 4509 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a20a346bf4da437d716c40247858fb251b2c0ea6c1d7c45b972353ba6bc78a3"} err="failed to get container status \"5a20a346bf4da437d716c40247858fb251b2c0ea6c1d7c45b972353ba6bc78a3\": rpc error: code = NotFound desc = could not find container \"5a20a346bf4da437d716c40247858fb251b2c0ea6c1d7c45b972353ba6bc78a3\": container with ID starting with 5a20a346bf4da437d716c40247858fb251b2c0ea6c1d7c45b972353ba6bc78a3 not found: ID does not exist" Nov 04 04:52:48.669407 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:52:48.669312 4509 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-p4rkn"] Nov 04 04:52:48.719710 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:52:48.719679 4509 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-p4rkn"] Nov 04 04:52:49.511646 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:52:49.511607 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-glcvz" event={"ID":"b26b2ef7-b5a7-4933-8393-f830db7d1387","Type":"ContainerStarted","Data":"8181954d4faa397e84b24007244d901c4eb4ebc3fa20f5809f4ad87c1cb9aaa8"} Nov 04 04:52:49.512051 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:52:49.511740 4509 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-glcvz" Nov 04 04:52:49.627855 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:52:49.627797 4509 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-glcvz" podStartSLOduration=1.862951112 podStartE2EDuration="3.627783799s" totalImagesPullingTime="1.764832687s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 04:52:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:52:49.626956899 +0000 UTC m=+333.504680421" watchObservedRunningTime="2026-11-04 04:52:49.627783799 +0000 UTC m=+333.505507322" Nov 04 04:52:50.261117 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:52:50.261081 4509 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df6aa8aa-5fee-42cb-a41c-1934516cccc1" path="/var/lib/kubelet/pods/df6aa8aa-5fee-42cb-a41c-1934516cccc1/volumes" Nov 04 04:52:57.500480 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:52:57.500454 4509 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-42dnm" Nov 04 04:53:10.517990 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:53:10.517960 4509 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-glcvz" Nov 04 04:53:14.454537 ip-10-0-133-180 kubenswrapper[4509]: E1104 04:53:14.454501 4509 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmdhl" containerName="extractor" Nov 04 04:53:44.455138 ip-10-0-133-180 kubenswrapper[4509]: E1104 04:53:44.455101 4509 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmdhl" containerName="extractor" Nov 04 04:53:45.082084 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:53:45.082050 4509 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-f447m"] Nov 04 04:53:45.082378 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:53:45.082365 4509 cpu_manager.go:414] "containerMap: removing container" podUID="df6aa8aa-5fee-42cb-a41c-1934516cccc1" containerName="authorino" Nov 04 04:53:45.082428 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:53:45.082380 4509 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df6aa8aa-5fee-42cb-a41c-1934516cccc1" containerName="authorino" Nov 04 04:53:45.082465 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:53:45.082441 4509 memory_manager.go:365] "RemoveStaleState removing state" podUID="df6aa8aa-5fee-42cb-a41c-1934516cccc1" containerName="authorino" Nov 04 04:53:45.089394 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:53:45.089373 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-f447m" Nov 04 04:53:45.145668 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:53:45.145629 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-f447m"] Nov 04 04:53:45.228596 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:53:45.228564 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a4d34feb-7fd9-4b77-a628-eb9d84adb93c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-f447m\" (UID: \"a4d34feb-7fd9-4b77-a628-eb9d84adb93c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-f447m" Nov 04 04:53:45.228762 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:53:45.228659 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4d34feb-7fd9-4b77-a628-eb9d84adb93c-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-f447m\" (UID: \"a4d34feb-7fd9-4b77-a628-eb9d84adb93c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-f447m" Nov 04 04:53:45.228762 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:53:45.228684 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a4d34feb-7fd9-4b77-a628-eb9d84adb93c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-f447m\" (UID: \"a4d34feb-7fd9-4b77-a628-eb9d84adb93c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-f447m" Nov 04 04:53:45.228762 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:53:45.228749 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a4d34feb-7fd9-4b77-a628-eb9d84adb93c-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-f447m\" (UID: \"a4d34feb-7fd9-4b77-a628-eb9d84adb93c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-f447m" Nov 04 04:53:45.228865 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:53:45.228777 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a4d34feb-7fd9-4b77-a628-eb9d84adb93c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-f447m\" (UID: \"a4d34feb-7fd9-4b77-a628-eb9d84adb93c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-f447m" Nov 04 04:53:45.228865 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:53:45.228801 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4d34feb-7fd9-4b77-a628-eb9d84adb93c-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-f447m\" (UID: \"a4d34feb-7fd9-4b77-a628-eb9d84adb93c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-f447m" Nov 04 04:53:45.228865 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:53:45.228827 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-865hm\" (UniqueName: \"kubernetes.io/projected/a4d34feb-7fd9-4b77-a628-eb9d84adb93c-kube-api-access-865hm\") pod \"istiod-openshift-gateway-7ddd9cfdc8-f447m\" (UID: \"a4d34feb-7fd9-4b77-a628-eb9d84adb93c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-f447m" Nov 04 04:53:45.330002 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:53:45.329968 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4d34feb-7fd9-4b77-a628-eb9d84adb93c-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-f447m\" (UID: \"a4d34feb-7fd9-4b77-a628-eb9d84adb93c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-f447m" Nov 04 04:53:45.330002 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:53:45.330001 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a4d34feb-7fd9-4b77-a628-eb9d84adb93c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-f447m\" (UID: \"a4d34feb-7fd9-4b77-a628-eb9d84adb93c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-f447m" Nov 04 04:53:45.330205 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:53:45.330028 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a4d34feb-7fd9-4b77-a628-eb9d84adb93c-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-f447m\" (UID: \"a4d34feb-7fd9-4b77-a628-eb9d84adb93c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-f447m" Nov 04 04:53:45.330205 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:53:45.330055 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a4d34feb-7fd9-4b77-a628-eb9d84adb93c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-f447m\" (UID: \"a4d34feb-7fd9-4b77-a628-eb9d84adb93c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-f447m" Nov 04 04:53:45.330205 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:53:45.330089 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4d34feb-7fd9-4b77-a628-eb9d84adb93c-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-f447m\" (UID: \"a4d34feb-7fd9-4b77-a628-eb9d84adb93c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-f447m" Nov 04 04:53:45.330205 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:53:45.330117 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-865hm\" (UniqueName: \"kubernetes.io/projected/a4d34feb-7fd9-4b77-a628-eb9d84adb93c-kube-api-access-865hm\") pod \"istiod-openshift-gateway-7ddd9cfdc8-f447m\" (UID: \"a4d34feb-7fd9-4b77-a628-eb9d84adb93c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-f447m" Nov 04 04:53:45.330413 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:53:45.330306 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a4d34feb-7fd9-4b77-a628-eb9d84adb93c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-f447m\" (UID: \"a4d34feb-7fd9-4b77-a628-eb9d84adb93c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-f447m" Nov 04 04:53:45.331042 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:53:45.331016 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a4d34feb-7fd9-4b77-a628-eb9d84adb93c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-f447m\" (UID: \"a4d34feb-7fd9-4b77-a628-eb9d84adb93c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-f447m" Nov 04 04:53:45.332650 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:53:45.332593 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4d34feb-7fd9-4b77-a628-eb9d84adb93c-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-f447m\" (UID: \"a4d34feb-7fd9-4b77-a628-eb9d84adb93c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-f447m" Nov 04 04:53:45.332650 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:53:45.332622 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a4d34feb-7fd9-4b77-a628-eb9d84adb93c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-f447m\" (UID: \"a4d34feb-7fd9-4b77-a628-eb9d84adb93c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-f447m" Nov 04 04:53:45.332772 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:53:45.332645 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a4d34feb-7fd9-4b77-a628-eb9d84adb93c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-f447m\" (UID: \"a4d34feb-7fd9-4b77-a628-eb9d84adb93c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-f447m" Nov 04 04:53:45.332772 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:53:45.332740 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a4d34feb-7fd9-4b77-a628-eb9d84adb93c-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-f447m\" (UID: \"a4d34feb-7fd9-4b77-a628-eb9d84adb93c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-f447m" Nov 04 04:53:45.387142 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:53:45.387110 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4d34feb-7fd9-4b77-a628-eb9d84adb93c-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-f447m\" (UID: \"a4d34feb-7fd9-4b77-a628-eb9d84adb93c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-f447m" Nov 04 04:53:45.387453 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:53:45.387431 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-865hm\" (UniqueName: \"kubernetes.io/projected/a4d34feb-7fd9-4b77-a628-eb9d84adb93c-kube-api-access-865hm\") pod \"istiod-openshift-gateway-7ddd9cfdc8-f447m\" (UID: \"a4d34feb-7fd9-4b77-a628-eb9d84adb93c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-f447m" Nov 04 04:53:45.400726 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:53:45.400704 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-f447m" Nov 04 04:53:45.823442 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:53:45.823333 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-f447m"] Nov 04 04:53:45.826092 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:53:45.826066 4509 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4d34feb_7fd9_4b77_a628_eb9d84adb93c.slice/crio-ed8890ba19c93c6a8fa03abb57f59f406e479e1cac139d5194386a57fabb56e4 WatchSource:0}: Error finding container ed8890ba19c93c6a8fa03abb57f59f406e479e1cac139d5194386a57fabb56e4: Status 404 returned error can't find the container with id ed8890ba19c93c6a8fa03abb57f59f406e479e1cac139d5194386a57fabb56e4 Nov 04 04:53:45.828078 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:53:45.828048 4509 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 04 04:53:45.828133 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:53:45.828109 4509 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 04 04:53:46.740068 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:53:46.740026 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-f447m" event={"ID":"a4d34feb-7fd9-4b77-a628-eb9d84adb93c","Type":"ContainerStarted","Data":"33e91a32ede27c25291ae26ea2f1bf4bd83fbff7317abf4da6e9da819f432313"} Nov 04 04:53:46.740358 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:53:46.740088 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-f447m" event={"ID":"a4d34feb-7fd9-4b77-a628-eb9d84adb93c","Type":"ContainerStarted","Data":"ed8890ba19c93c6a8fa03abb57f59f406e479e1cac139d5194386a57fabb56e4"} Nov 04 04:53:46.857462 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:53:46.857407 4509 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-f447m" podStartSLOduration=1.857393485 podStartE2EDuration="1.857393485s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 04:53:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:53:46.855414776 +0000 UTC m=+390.733138297" watchObservedRunningTime="2026-11-04 04:53:46.857393485 +0000 UTC m=+390.735117005" Nov 04 04:53:47.743630 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:53:47.743599 4509 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-f447m" Nov 04 04:53:47.745504 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:53:47.745481 4509 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-f447m" Nov 04 04:53:47.962960 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:53:47.962917 4509 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-g8rd9"] Nov 04 04:53:47.963451 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:53:47.963133 4509 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-g8rd9" podUID="ce33eec0-7da6-4daa-a122-3b14aab546bb" containerName="discovery" containerID="cri-o://1451d99d41659d2a36809a9936e812aec4d65f72bd8f62ee5ec10850eb039bd1" gracePeriod=30 Nov 04 04:53:48.199227 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:53:48.199202 4509 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-g8rd9" Nov 04 04:53:48.357144 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:53:48.357067 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce33eec0-7da6-4daa-a122-3b14aab546bb-kube-api-access-98ckb\" (UniqueName: \"kubernetes.io/projected/ce33eec0-7da6-4daa-a122-3b14aab546bb-kube-api-access-98ckb\") pod \"ce33eec0-7da6-4daa-a122-3b14aab546bb\" (UID: \"ce33eec0-7da6-4daa-a122-3b14aab546bb\") " Nov 04 04:53:48.357144 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:53:48.357120 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce33eec0-7da6-4daa-a122-3b14aab546bb-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ce33eec0-7da6-4daa-a122-3b14aab546bb-istio-csr-dns-cert\") pod \"ce33eec0-7da6-4daa-a122-3b14aab546bb\" (UID: \"ce33eec0-7da6-4daa-a122-3b14aab546bb\") " Nov 04 04:53:48.357395 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:53:48.357160 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce33eec0-7da6-4daa-a122-3b14aab546bb-istio-token\" (UniqueName: \"kubernetes.io/projected/ce33eec0-7da6-4daa-a122-3b14aab546bb-istio-token\") pod \"ce33eec0-7da6-4daa-a122-3b14aab546bb\" (UID: \"ce33eec0-7da6-4daa-a122-3b14aab546bb\") " Nov 04 04:53:48.357395 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:53:48.357178 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce33eec0-7da6-4daa-a122-3b14aab546bb-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ce33eec0-7da6-4daa-a122-3b14aab546bb-istio-csr-ca-configmap\") pod \"ce33eec0-7da6-4daa-a122-3b14aab546bb\" (UID: \"ce33eec0-7da6-4daa-a122-3b14aab546bb\") " Nov 04 04:53:48.357395 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:53:48.357213 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce33eec0-7da6-4daa-a122-3b14aab546bb-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce33eec0-7da6-4daa-a122-3b14aab546bb-local-certs\") pod \"ce33eec0-7da6-4daa-a122-3b14aab546bb\" (UID: \"ce33eec0-7da6-4daa-a122-3b14aab546bb\") " Nov 04 04:53:48.357395 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:53:48.357251 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce33eec0-7da6-4daa-a122-3b14aab546bb-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ce33eec0-7da6-4daa-a122-3b14aab546bb-istio-kubeconfig\") pod \"ce33eec0-7da6-4daa-a122-3b14aab546bb\" (UID: \"ce33eec0-7da6-4daa-a122-3b14aab546bb\") " Nov 04 04:53:48.357395 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:53:48.357281 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce33eec0-7da6-4daa-a122-3b14aab546bb-cacerts\" (UniqueName: \"kubernetes.io/secret/ce33eec0-7da6-4daa-a122-3b14aab546bb-cacerts\") pod \"ce33eec0-7da6-4daa-a122-3b14aab546bb\" (UID: \"ce33eec0-7da6-4daa-a122-3b14aab546bb\") " Nov 04 04:53:48.357735 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:53:48.357688 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce33eec0-7da6-4daa-a122-3b14aab546bb-istio-csr-ca-configmap" pod "ce33eec0-7da6-4daa-a122-3b14aab546bb" (UID: "ce33eec0-7da6-4daa-a122-3b14aab546bb"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 04:53:48.360264 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:53:48.360238 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce33eec0-7da6-4daa-a122-3b14aab546bb-cacerts" pod "ce33eec0-7da6-4daa-a122-3b14aab546bb" (UID: "ce33eec0-7da6-4daa-a122-3b14aab546bb"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 04:53:48.360378 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:53:48.360251 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce33eec0-7da6-4daa-a122-3b14aab546bb-local-certs" pod "ce33eec0-7da6-4daa-a122-3b14aab546bb" (UID: "ce33eec0-7da6-4daa-a122-3b14aab546bb"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 04:53:48.360378 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:53:48.360270 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce33eec0-7da6-4daa-a122-3b14aab546bb-istio-csr-dns-cert" pod "ce33eec0-7da6-4daa-a122-3b14aab546bb" (UID: "ce33eec0-7da6-4daa-a122-3b14aab546bb"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 04:53:48.360378 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:53:48.360275 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce33eec0-7da6-4daa-a122-3b14aab546bb-kube-api-access-98ckb" pod "ce33eec0-7da6-4daa-a122-3b14aab546bb" (UID: "ce33eec0-7da6-4daa-a122-3b14aab546bb"). InnerVolumeSpecName "kube-api-access-98ckb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 04:53:48.360378 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:53:48.360288 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce33eec0-7da6-4daa-a122-3b14aab546bb-istio-token" pod "ce33eec0-7da6-4daa-a122-3b14aab546bb" (UID: "ce33eec0-7da6-4daa-a122-3b14aab546bb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 04:53:48.360378 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:53:48.360297 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce33eec0-7da6-4daa-a122-3b14aab546bb-istio-kubeconfig" pod "ce33eec0-7da6-4daa-a122-3b14aab546bb" (UID: "ce33eec0-7da6-4daa-a122-3b14aab546bb"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 04:53:48.457919 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:53:48.457881 4509 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ce33eec0-7da6-4daa-a122-3b14aab546bb-istio-csr-dns-cert\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 04:53:48.457919 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:53:48.457910 4509 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce33eec0-7da6-4daa-a122-3b14aab546bb-istio-token\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 04:53:48.457919 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:53:48.457919 4509 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ce33eec0-7da6-4daa-a122-3b14aab546bb-istio-csr-ca-configmap\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 04:53:48.458140 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:53:48.457930 4509 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce33eec0-7da6-4daa-a122-3b14aab546bb-local-certs\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 04:53:48.458140 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:53:48.457939 4509 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ce33eec0-7da6-4daa-a122-3b14aab546bb-istio-kubeconfig\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 04:53:48.458140 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:53:48.457948 4509 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ce33eec0-7da6-4daa-a122-3b14aab546bb-cacerts\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 04:53:48.458140 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:53:48.457956 4509 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98ckb\" (UniqueName: \"kubernetes.io/projected/ce33eec0-7da6-4daa-a122-3b14aab546bb-kube-api-access-98ckb\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 04:53:48.748919 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:53:48.748883 4509 generic.go:356] "Generic (PLEG): container finished" podID="ce33eec0-7da6-4daa-a122-3b14aab546bb" containerID="1451d99d41659d2a36809a9936e812aec4d65f72bd8f62ee5ec10850eb039bd1" exitCode=0 Nov 04 04:53:48.749103 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:53:48.748942 4509 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-g8rd9" Nov 04 04:53:48.749103 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:53:48.748961 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-g8rd9" event={"ID":"ce33eec0-7da6-4daa-a122-3b14aab546bb","Type":"ContainerDied","Data":"1451d99d41659d2a36809a9936e812aec4d65f72bd8f62ee5ec10850eb039bd1"} Nov 04 04:53:48.749103 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:53:48.749017 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-g8rd9" event={"ID":"ce33eec0-7da6-4daa-a122-3b14aab546bb","Type":"ContainerDied","Data":"2500b21e560f4ffbe0b8e2425204365fb5e1f7ca774e1f3fb07d1196a97a16e3"} Nov 04 04:53:48.749103 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:53:48.749037 4509 scope.go:122] "RemoveContainer" containerID="1451d99d41659d2a36809a9936e812aec4d65f72bd8f62ee5ec10850eb039bd1" Nov 04 04:53:48.757422 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:53:48.757404 4509 scope.go:122] "RemoveContainer" containerID="1451d99d41659d2a36809a9936e812aec4d65f72bd8f62ee5ec10850eb039bd1" Nov 04 04:53:48.757686 ip-10-0-133-180 kubenswrapper[4509]: E1104 04:53:48.757667 4509 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1451d99d41659d2a36809a9936e812aec4d65f72bd8f62ee5ec10850eb039bd1\": container with ID starting with 1451d99d41659d2a36809a9936e812aec4d65f72bd8f62ee5ec10850eb039bd1 not found: ID does not exist" containerID="1451d99d41659d2a36809a9936e812aec4d65f72bd8f62ee5ec10850eb039bd1" Nov 04 04:53:48.757734 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:53:48.757696 4509 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1451d99d41659d2a36809a9936e812aec4d65f72bd8f62ee5ec10850eb039bd1"} err="failed to get container status \"1451d99d41659d2a36809a9936e812aec4d65f72bd8f62ee5ec10850eb039bd1\": rpc error: code = NotFound desc = could not find container \"1451d99d41659d2a36809a9936e812aec4d65f72bd8f62ee5ec10850eb039bd1\": container with ID starting with 1451d99d41659d2a36809a9936e812aec4d65f72bd8f62ee5ec10850eb039bd1 not found: ID does not exist" Nov 04 04:53:48.912091 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:53:48.912063 4509 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-g8rd9"] Nov 04 04:53:48.961855 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:53:48.961821 4509 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-g8rd9"] Nov 04 04:53:50.260913 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:53:50.260882 4509 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce33eec0-7da6-4daa-a122-3b14aab546bb" path="/var/lib/kubelet/pods/ce33eec0-7da6-4daa-a122-3b14aab546bb/volumes" Nov 04 04:53:51.236471 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:53:51.236439 4509 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-7pj6b"] Nov 04 04:53:51.236786 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:53:51.236773 4509 cpu_manager.go:414] "containerMap: removing container" podUID="ce33eec0-7da6-4daa-a122-3b14aab546bb" containerName="discovery" Nov 04 04:53:51.236833 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:53:51.236789 4509 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce33eec0-7da6-4daa-a122-3b14aab546bb" containerName="discovery" Nov 04 04:53:51.236866 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:53:51.236848 4509 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce33eec0-7da6-4daa-a122-3b14aab546bb" containerName="discovery" Nov 04 04:53:51.240675 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:53:51.240658 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-7pj6b" Nov 04 04:53:51.344970 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:53:51.344935 4509 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-cll8n\"" Nov 04 04:53:51.345464 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:53:51.345210 4509 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Nov 04 04:53:51.346374 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:53:51.346352 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-7pj6b"] Nov 04 04:53:51.382106 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:53:51.382076 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4f68592-75e2-4d18-9b72-f987c072af88-tls-certs\") pod \"model-serving-api-75bd89fdfc-7pj6b\" (UID: \"b4f68592-75e2-4d18-9b72-f987c072af88\") " pod="opendatahub/model-serving-api-75bd89fdfc-7pj6b" Nov 04 04:53:51.382260 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:53:51.382240 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvx6f\" (UniqueName: \"kubernetes.io/projected/b4f68592-75e2-4d18-9b72-f987c072af88-kube-api-access-mvx6f\") pod \"model-serving-api-75bd89fdfc-7pj6b\" (UID: \"b4f68592-75e2-4d18-9b72-f987c072af88\") " pod="opendatahub/model-serving-api-75bd89fdfc-7pj6b" Nov 04 04:53:51.483300 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:53:51.483266 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4f68592-75e2-4d18-9b72-f987c072af88-tls-certs\") pod \"model-serving-api-75bd89fdfc-7pj6b\" (UID: \"b4f68592-75e2-4d18-9b72-f987c072af88\") " pod="opendatahub/model-serving-api-75bd89fdfc-7pj6b" Nov 04 04:53:51.483476 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:53:51.483357 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvx6f\" (UniqueName: \"kubernetes.io/projected/b4f68592-75e2-4d18-9b72-f987c072af88-kube-api-access-mvx6f\") pod \"model-serving-api-75bd89fdfc-7pj6b\" (UID: \"b4f68592-75e2-4d18-9b72-f987c072af88\") " pod="opendatahub/model-serving-api-75bd89fdfc-7pj6b" Nov 04 04:53:51.485844 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:53:51.485816 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4f68592-75e2-4d18-9b72-f987c072af88-tls-certs\") pod \"model-serving-api-75bd89fdfc-7pj6b\" (UID: \"b4f68592-75e2-4d18-9b72-f987c072af88\") " pod="opendatahub/model-serving-api-75bd89fdfc-7pj6b" Nov 04 04:53:51.540481 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:53:51.540410 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvx6f\" (UniqueName: \"kubernetes.io/projected/b4f68592-75e2-4d18-9b72-f987c072af88-kube-api-access-mvx6f\") pod \"model-serving-api-75bd89fdfc-7pj6b\" (UID: \"b4f68592-75e2-4d18-9b72-f987c072af88\") " pod="opendatahub/model-serving-api-75bd89fdfc-7pj6b" Nov 04 04:53:51.553038 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:53:51.553009 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-7pj6b" Nov 04 04:53:51.920922 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:53:51.920845 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-7pj6b"] Nov 04 04:53:51.923883 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:53:51.923841 4509 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4f68592_75e2_4d18_9b72_f987c072af88.slice/crio-8e07a171c6e7e6b2f2a8751e8bda99ed724c5a1704b6cbccbbcc76ac05b3e10c WatchSource:0}: Error finding container 8e07a171c6e7e6b2f2a8751e8bda99ed724c5a1704b6cbccbbcc76ac05b3e10c: Status 404 returned error can't find the container with id 8e07a171c6e7e6b2f2a8751e8bda99ed724c5a1704b6cbccbbcc76ac05b3e10c Nov 04 04:53:52.768535 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:53:52.768492 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-7pj6b" event={"ID":"b4f68592-75e2-4d18-9b72-f987c072af88","Type":"ContainerStarted","Data":"8e07a171c6e7e6b2f2a8751e8bda99ed724c5a1704b6cbccbbcc76ac05b3e10c"} Nov 04 04:53:54.778924 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:53:54.778886 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-7pj6b" event={"ID":"b4f68592-75e2-4d18-9b72-f987c072af88","Type":"ContainerStarted","Data":"e990d9f7e72a67cdd94845ab44d465061e1efd31ef7f3ed13829f49a36ea17a6"} Nov 04 04:53:54.779430 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:53:54.778980 4509 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-7pj6b" Nov 04 04:53:54.893810 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:53:54.893755 4509 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-7pj6b" podStartSLOduration=1.759498623 podStartE2EDuration="3.893735223s" totalImagesPullingTime="2.1342366s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 04:53:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:53:54.89280225 +0000 UTC m=+398.770525772" watchObservedRunningTime="2026-11-04 04:53:54.893735223 +0000 UTC m=+398.771458746" Nov 04 04:54:05.786797 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:54:05.786765 4509 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-7pj6b" Nov 04 04:54:09.925240 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:54:09.925208 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-42dnm"] Nov 04 04:54:14.453216 ip-10-0-133-180 kubenswrapper[4509]: E1104 04:54:14.453181 4509 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmdhl" containerName="extractor" Nov 04 04:54:17.003434 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:54:17.003396 4509 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-584d2"] Nov 04 04:54:17.007904 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:54:17.007886 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-584d2" Nov 04 04:54:17.063803 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:54:17.063769 4509 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-zfb5x\"" Nov 04 04:54:17.064908 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:54:17.064886 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-584d2"] Nov 04 04:54:17.085292 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:54:17.085261 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssc6r\" (UniqueName: \"kubernetes.io/projected/782e96b5-1df2-47c1-bed0-816096e364fc-kube-api-access-ssc6r\") pod \"authorino-6f6944bcdd-584d2\" (UID: \"782e96b5-1df2-47c1-bed0-816096e364fc\") " pod="kuadrant-system/authorino-6f6944bcdd-584d2" Nov 04 04:54:17.186047 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:54:17.186009 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssc6r\" (UniqueName: \"kubernetes.io/projected/782e96b5-1df2-47c1-bed0-816096e364fc-kube-api-access-ssc6r\") pod \"authorino-6f6944bcdd-584d2\" (UID: \"782e96b5-1df2-47c1-bed0-816096e364fc\") " pod="kuadrant-system/authorino-6f6944bcdd-584d2" Nov 04 04:54:17.243877 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:54:17.243850 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssc6r\" (UniqueName: \"kubernetes.io/projected/782e96b5-1df2-47c1-bed0-816096e364fc-kube-api-access-ssc6r\") pod \"authorino-6f6944bcdd-584d2\" (UID: \"782e96b5-1df2-47c1-bed0-816096e364fc\") " pod="kuadrant-system/authorino-6f6944bcdd-584d2" Nov 04 04:54:17.328896 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:54:17.328811 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-584d2" Nov 04 04:54:17.345829 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:54:17.345803 4509 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-584d2"] Nov 04 04:54:17.367686 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:54:17.367656 4509 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-d5789b98-2pds2"] Nov 04 04:54:17.371367 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:54:17.371345 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-d5789b98-2pds2" Nov 04 04:54:17.427076 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:54:17.427046 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-d5789b98-2pds2"] Nov 04 04:54:17.489106 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:54:17.489076 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfrwk\" (UniqueName: \"kubernetes.io/projected/89f2cb82-96df-4bae-aa4b-75948be10e22-kube-api-access-cfrwk\") pod \"authorino-d5789b98-2pds2\" (UID: \"89f2cb82-96df-4bae-aa4b-75948be10e22\") " pod="kuadrant-system/authorino-d5789b98-2pds2" Nov 04 04:54:17.590371 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:54:17.590286 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfrwk\" (UniqueName: \"kubernetes.io/projected/89f2cb82-96df-4bae-aa4b-75948be10e22-kube-api-access-cfrwk\") pod \"authorino-d5789b98-2pds2\" (UID: \"89f2cb82-96df-4bae-aa4b-75948be10e22\") " pod="kuadrant-system/authorino-d5789b98-2pds2" Nov 04 04:54:17.647139 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:54:17.647108 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfrwk\" (UniqueName: \"kubernetes.io/projected/89f2cb82-96df-4bae-aa4b-75948be10e22-kube-api-access-cfrwk\") pod \"authorino-d5789b98-2pds2\" (UID: \"89f2cb82-96df-4bae-aa4b-75948be10e22\") " pod="kuadrant-system/authorino-d5789b98-2pds2" Nov 04 04:54:17.687924 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:54:17.687893 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-d5789b98-2pds2" Nov 04 04:54:17.699085 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:54:17.699061 4509 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-584d2"] Nov 04 04:54:17.700778 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:54:17.700756 4509 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod782e96b5_1df2_47c1_bed0_816096e364fc.slice/crio-426f73ef2473889d26c9fbc72c918a2cd296aebadacccb1f92c0002e43405fdc WatchSource:0}: Error finding container 426f73ef2473889d26c9fbc72c918a2cd296aebadacccb1f92c0002e43405fdc: Status 404 returned error can't find the container with id 426f73ef2473889d26c9fbc72c918a2cd296aebadacccb1f92c0002e43405fdc Nov 04 04:54:17.810164 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:54:17.810129 4509 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-d5789b98-2pds2"] Nov 04 04:54:17.867164 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:54:17.867082 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-584d2" event={"ID":"782e96b5-1df2-47c1-bed0-816096e364fc","Type":"ContainerStarted","Data":"426f73ef2473889d26c9fbc72c918a2cd296aebadacccb1f92c0002e43405fdc"} Nov 04 04:54:18.054363 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:54:18.054309 4509 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-d5789b98-2pds2"] Nov 04 04:54:18.057032 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:54:18.057000 4509 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod89f2cb82_96df_4bae_aa4b_75948be10e22.slice/crio-af0b397a2d7b8a4437fce67ab65ccd7119adfd79dce6cc33ecb33838e797b6ed WatchSource:0}: Error finding container af0b397a2d7b8a4437fce67ab65ccd7119adfd79dce6cc33ecb33838e797b6ed: Status 404 returned error can't find the container with id af0b397a2d7b8a4437fce67ab65ccd7119adfd79dce6cc33ecb33838e797b6ed Nov 04 04:54:18.871867 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:54:18.871830 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-d5789b98-2pds2" event={"ID":"89f2cb82-96df-4bae-aa4b-75948be10e22","Type":"ContainerStarted","Data":"af0b397a2d7b8a4437fce67ab65ccd7119adfd79dce6cc33ecb33838e797b6ed"} Nov 04 04:54:19.877838 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:54:19.877752 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-d5789b98-2pds2" event={"ID":"89f2cb82-96df-4bae-aa4b-75948be10e22","Type":"ContainerStarted","Data":"67fbc14ae4a34c7e683692ad5289fccba76df2e699e803bff6d14bf171642791"} Nov 04 04:54:19.878247 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:54:19.877855 4509 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-d5789b98-2pds2" podUID="89f2cb82-96df-4bae-aa4b-75948be10e22" containerName="authorino" containerID="cri-o://67fbc14ae4a34c7e683692ad5289fccba76df2e699e803bff6d14bf171642791" gracePeriod=30 Nov 04 04:54:19.993093 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:54:19.993036 4509 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-d5789b98-2pds2" podStartSLOduration=1.532325589 podStartE2EDuration="2.993021939s" totalImagesPullingTime="1.46069635s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 04:54:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:54:19.991803112 +0000 UTC m=+423.869526633" watchObservedRunningTime="2026-11-04 04:54:19.993021939 +0000 UTC m=+423.870745460" Nov 04 04:54:20.191464 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:54:20.191438 4509 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-d5789b98-2pds2" Nov 04 04:54:20.313218 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:54:20.313186 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89f2cb82-96df-4bae-aa4b-75948be10e22-kube-api-access-cfrwk\" (UniqueName: \"kubernetes.io/projected/89f2cb82-96df-4bae-aa4b-75948be10e22-kube-api-access-cfrwk\") pod \"89f2cb82-96df-4bae-aa4b-75948be10e22\" (UID: \"89f2cb82-96df-4bae-aa4b-75948be10e22\") " Nov 04 04:54:20.315405 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:54:20.315381 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89f2cb82-96df-4bae-aa4b-75948be10e22-kube-api-access-cfrwk" pod "89f2cb82-96df-4bae-aa4b-75948be10e22" (UID: "89f2cb82-96df-4bae-aa4b-75948be10e22"). InnerVolumeSpecName "kube-api-access-cfrwk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 04:54:20.414800 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:54:20.414749 4509 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfrwk\" (UniqueName: \"kubernetes.io/projected/89f2cb82-96df-4bae-aa4b-75948be10e22-kube-api-access-cfrwk\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 04:54:20.883290 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:54:20.883255 4509 generic.go:356] "Generic (PLEG): container finished" podID="89f2cb82-96df-4bae-aa4b-75948be10e22" containerID="67fbc14ae4a34c7e683692ad5289fccba76df2e699e803bff6d14bf171642791" exitCode=0 Nov 04 04:54:20.883786 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:54:20.883301 4509 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-d5789b98-2pds2" Nov 04 04:54:20.883786 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:54:20.883357 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-d5789b98-2pds2" event={"ID":"89f2cb82-96df-4bae-aa4b-75948be10e22","Type":"ContainerDied","Data":"67fbc14ae4a34c7e683692ad5289fccba76df2e699e803bff6d14bf171642791"} Nov 04 04:54:20.883786 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:54:20.883411 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-d5789b98-2pds2" event={"ID":"89f2cb82-96df-4bae-aa4b-75948be10e22","Type":"ContainerDied","Data":"af0b397a2d7b8a4437fce67ab65ccd7119adfd79dce6cc33ecb33838e797b6ed"} Nov 04 04:54:20.883786 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:54:20.883434 4509 scope.go:122] "RemoveContainer" containerID="67fbc14ae4a34c7e683692ad5289fccba76df2e699e803bff6d14bf171642791" Nov 04 04:54:20.885213 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:54:20.885166 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-584d2" event={"ID":"782e96b5-1df2-47c1-bed0-816096e364fc","Type":"ContainerStarted","Data":"27caf1340e7c235cf087d80235bb49c26442c1660bb3ae2f34444ca987b6da81"} Nov 04 04:54:20.885365 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:54:20.885244 4509 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6f6944bcdd-584d2" podUID="782e96b5-1df2-47c1-bed0-816096e364fc" containerName="authorino" containerID="cri-o://27caf1340e7c235cf087d80235bb49c26442c1660bb3ae2f34444ca987b6da81" gracePeriod=30 Nov 04 04:54:20.894869 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:54:20.894850 4509 scope.go:122] "RemoveContainer" containerID="67fbc14ae4a34c7e683692ad5289fccba76df2e699e803bff6d14bf171642791" Nov 04 04:54:20.895170 ip-10-0-133-180 kubenswrapper[4509]: E1104 04:54:20.895151 4509 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67fbc14ae4a34c7e683692ad5289fccba76df2e699e803bff6d14bf171642791\": container with ID starting with 67fbc14ae4a34c7e683692ad5289fccba76df2e699e803bff6d14bf171642791 not found: ID does not exist" containerID="67fbc14ae4a34c7e683692ad5289fccba76df2e699e803bff6d14bf171642791" Nov 04 04:54:20.895209 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:54:20.895179 4509 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67fbc14ae4a34c7e683692ad5289fccba76df2e699e803bff6d14bf171642791"} err="failed to get container status \"67fbc14ae4a34c7e683692ad5289fccba76df2e699e803bff6d14bf171642791\": rpc error: code = NotFound desc = could not find container \"67fbc14ae4a34c7e683692ad5289fccba76df2e699e803bff6d14bf171642791\": container with ID starting with 67fbc14ae4a34c7e683692ad5289fccba76df2e699e803bff6d14bf171642791 not found: ID does not exist" Nov 04 04:54:21.046557 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:54:21.046527 4509 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-d5789b98-2pds2"] Nov 04 04:54:21.048663 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:54:21.048639 4509 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-d5789b98-2pds2"] Nov 04 04:54:21.131602 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:54:21.131578 4509 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-584d2" Nov 04 04:54:21.159002 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:54:21.158918 4509 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6f6944bcdd-584d2" podStartSLOduration=2.7661917320000002 podStartE2EDuration="5.158903899s" totalImagesPullingTime="2.392712167s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 04:54:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:54:21.158514103 +0000 UTC m=+425.036237625" watchObservedRunningTime="2026-11-04 04:54:21.158903899 +0000 UTC m=+425.036627467" Nov 04 04:54:21.221314 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:54:21.221285 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/782e96b5-1df2-47c1-bed0-816096e364fc-kube-api-access-ssc6r\" (UniqueName: \"kubernetes.io/projected/782e96b5-1df2-47c1-bed0-816096e364fc-kube-api-access-ssc6r\") pod \"782e96b5-1df2-47c1-bed0-816096e364fc\" (UID: \"782e96b5-1df2-47c1-bed0-816096e364fc\") " Nov 04 04:54:21.223458 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:54:21.223433 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/782e96b5-1df2-47c1-bed0-816096e364fc-kube-api-access-ssc6r" pod "782e96b5-1df2-47c1-bed0-816096e364fc" (UID: "782e96b5-1df2-47c1-bed0-816096e364fc"). InnerVolumeSpecName "kube-api-access-ssc6r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 04:54:21.323100 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:54:21.323055 4509 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ssc6r\" (UniqueName: \"kubernetes.io/projected/782e96b5-1df2-47c1-bed0-816096e364fc-kube-api-access-ssc6r\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 04:54:21.891252 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:54:21.891217 4509 generic.go:356] "Generic (PLEG): container finished" podID="782e96b5-1df2-47c1-bed0-816096e364fc" containerID="27caf1340e7c235cf087d80235bb49c26442c1660bb3ae2f34444ca987b6da81" exitCode=0 Nov 04 04:54:21.891684 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:54:21.891266 4509 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-584d2" Nov 04 04:54:21.891684 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:54:21.891305 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-584d2" event={"ID":"782e96b5-1df2-47c1-bed0-816096e364fc","Type":"ContainerDied","Data":"27caf1340e7c235cf087d80235bb49c26442c1660bb3ae2f34444ca987b6da81"} Nov 04 04:54:21.891684 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:54:21.891372 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-584d2" event={"ID":"782e96b5-1df2-47c1-bed0-816096e364fc","Type":"ContainerDied","Data":"426f73ef2473889d26c9fbc72c918a2cd296aebadacccb1f92c0002e43405fdc"} Nov 04 04:54:21.891684 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:54:21.891396 4509 scope.go:122] "RemoveContainer" containerID="27caf1340e7c235cf087d80235bb49c26442c1660bb3ae2f34444ca987b6da81" Nov 04 04:54:21.899269 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:54:21.899254 4509 scope.go:122] "RemoveContainer" containerID="27caf1340e7c235cf087d80235bb49c26442c1660bb3ae2f34444ca987b6da81" Nov 04 04:54:21.899560 ip-10-0-133-180 kubenswrapper[4509]: E1104 04:54:21.899540 4509 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27caf1340e7c235cf087d80235bb49c26442c1660bb3ae2f34444ca987b6da81\": container with ID starting with 27caf1340e7c235cf087d80235bb49c26442c1660bb3ae2f34444ca987b6da81 not found: ID does not exist" containerID="27caf1340e7c235cf087d80235bb49c26442c1660bb3ae2f34444ca987b6da81" Nov 04 04:54:21.899632 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:54:21.899571 4509 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27caf1340e7c235cf087d80235bb49c26442c1660bb3ae2f34444ca987b6da81"} err="failed to get container status \"27caf1340e7c235cf087d80235bb49c26442c1660bb3ae2f34444ca987b6da81\": rpc error: code = NotFound desc = could not find container \"27caf1340e7c235cf087d80235bb49c26442c1660bb3ae2f34444ca987b6da81\": container with ID starting with 27caf1340e7c235cf087d80235bb49c26442c1660bb3ae2f34444ca987b6da81 not found: ID does not exist" Nov 04 04:54:22.057357 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:54:22.057306 4509 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-584d2"] Nov 04 04:54:22.061801 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:54:22.061776 4509 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-584d2"] Nov 04 04:54:22.260126 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:54:22.260094 4509 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="782e96b5-1df2-47c1-bed0-816096e364fc" path="/var/lib/kubelet/pods/782e96b5-1df2-47c1-bed0-816096e364fc/volumes" Nov 04 04:54:22.260420 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:54:22.260407 4509 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89f2cb82-96df-4bae-aa4b-75948be10e22" path="/var/lib/kubelet/pods/89f2cb82-96df-4bae-aa4b-75948be10e22/volumes" Nov 04 04:54:44.453277 ip-10-0-133-180 kubenswrapper[4509]: E1104 04:54:44.453244 4509 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmdhl" containerName="extractor" Nov 04 04:55:14.453624 ip-10-0-133-180 kubenswrapper[4509]: E1104 04:55:14.453588 4509 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmdhl" containerName="extractor" Nov 04 04:55:44.453930 ip-10-0-133-180 kubenswrapper[4509]: E1104 04:55:44.453837 4509 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmdhl" containerName="extractor" Nov 04 04:55:57.559916 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:55:57.559882 4509 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-vkcnr"] Nov 04 04:55:57.560266 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:55:57.560162 4509 cpu_manager.go:414] "containerMap: removing container" podUID="89f2cb82-96df-4bae-aa4b-75948be10e22" containerName="authorino" Nov 04 04:55:57.560266 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:55:57.560172 4509 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89f2cb82-96df-4bae-aa4b-75948be10e22" containerName="authorino" Nov 04 04:55:57.560266 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:55:57.560184 4509 cpu_manager.go:414] "containerMap: removing container" podUID="782e96b5-1df2-47c1-bed0-816096e364fc" containerName="authorino" Nov 04 04:55:57.560266 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:55:57.560189 4509 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="782e96b5-1df2-47c1-bed0-816096e364fc" containerName="authorino" Nov 04 04:55:57.560266 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:55:57.560236 4509 memory_manager.go:365] "RemoveStaleState removing state" podUID="89f2cb82-96df-4bae-aa4b-75948be10e22" containerName="authorino" Nov 04 04:55:57.560266 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:55:57.560244 4509 memory_manager.go:365] "RemoveStaleState removing state" podUID="782e96b5-1df2-47c1-bed0-816096e364fc" containerName="authorino" Nov 04 04:55:57.563114 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:55:57.563100 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vkcnr" Nov 04 04:55:57.615750 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:55:57.615726 4509 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Nov 04 04:55:57.665526 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:55:57.665505 4509 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-wx27q\"" Nov 04 04:55:57.665667 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:55:57.665522 4509 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Nov 04 04:55:57.665667 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:55:57.665652 4509 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Nov 04 04:55:57.666860 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:55:57.666843 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-vkcnr"] Nov 04 04:55:57.668960 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:55:57.668945 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03dca812-45a7-42e0-81ce-9dad9861776c-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-vkcnr\" (UID: \"03dca812-45a7-42e0-81ce-9dad9861776c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vkcnr" Nov 04 04:55:57.669008 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:55:57.668971 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbllm\" (UniqueName: \"kubernetes.io/projected/03dca812-45a7-42e0-81ce-9dad9861776c-kube-api-access-fbllm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-vkcnr\" (UID: \"03dca812-45a7-42e0-81ce-9dad9861776c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vkcnr" Nov 04 04:55:57.669008 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:55:57.668992 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03dca812-45a7-42e0-81ce-9dad9861776c-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-vkcnr\" (UID: \"03dca812-45a7-42e0-81ce-9dad9861776c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vkcnr" Nov 04 04:55:57.669102 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:55:57.669017 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03dca812-45a7-42e0-81ce-9dad9861776c-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-vkcnr\" (UID: \"03dca812-45a7-42e0-81ce-9dad9861776c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vkcnr" Nov 04 04:55:57.669178 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:55:57.669162 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03dca812-45a7-42e0-81ce-9dad9861776c-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-vkcnr\" (UID: \"03dca812-45a7-42e0-81ce-9dad9861776c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vkcnr" Nov 04 04:55:57.669233 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:55:57.669206 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03dca812-45a7-42e0-81ce-9dad9861776c-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-vkcnr\" (UID: \"03dca812-45a7-42e0-81ce-9dad9861776c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vkcnr" Nov 04 04:55:57.769908 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:55:57.769881 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03dca812-45a7-42e0-81ce-9dad9861776c-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-vkcnr\" (UID: \"03dca812-45a7-42e0-81ce-9dad9861776c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vkcnr" Nov 04 04:55:57.770036 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:55:57.769927 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03dca812-45a7-42e0-81ce-9dad9861776c-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-vkcnr\" (UID: \"03dca812-45a7-42e0-81ce-9dad9861776c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vkcnr" Nov 04 04:55:57.770036 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:55:57.769952 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbllm\" (UniqueName: \"kubernetes.io/projected/03dca812-45a7-42e0-81ce-9dad9861776c-kube-api-access-fbllm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-vkcnr\" (UID: \"03dca812-45a7-42e0-81ce-9dad9861776c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vkcnr" Nov 04 04:55:57.770036 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:55:57.769980 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03dca812-45a7-42e0-81ce-9dad9861776c-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-vkcnr\" (UID: \"03dca812-45a7-42e0-81ce-9dad9861776c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vkcnr" Nov 04 04:55:57.770036 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:55:57.770020 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03dca812-45a7-42e0-81ce-9dad9861776c-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-vkcnr\" (UID: \"03dca812-45a7-42e0-81ce-9dad9861776c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vkcnr" Nov 04 04:55:57.770233 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:55:57.770086 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03dca812-45a7-42e0-81ce-9dad9861776c-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-vkcnr\" (UID: \"03dca812-45a7-42e0-81ce-9dad9861776c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vkcnr" Nov 04 04:55:57.770368 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:55:57.770308 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03dca812-45a7-42e0-81ce-9dad9861776c-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-vkcnr\" (UID: \"03dca812-45a7-42e0-81ce-9dad9861776c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vkcnr" Nov 04 04:55:57.770441 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:55:57.770315 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03dca812-45a7-42e0-81ce-9dad9861776c-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-vkcnr\" (UID: \"03dca812-45a7-42e0-81ce-9dad9861776c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vkcnr" Nov 04 04:55:57.770441 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:55:57.770383 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03dca812-45a7-42e0-81ce-9dad9861776c-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-vkcnr\" (UID: \"03dca812-45a7-42e0-81ce-9dad9861776c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vkcnr" Nov 04 04:55:57.772373 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:55:57.772345 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03dca812-45a7-42e0-81ce-9dad9861776c-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-vkcnr\" (UID: \"03dca812-45a7-42e0-81ce-9dad9861776c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vkcnr" Nov 04 04:55:57.772626 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:55:57.772606 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03dca812-45a7-42e0-81ce-9dad9861776c-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-vkcnr\" (UID: \"03dca812-45a7-42e0-81ce-9dad9861776c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vkcnr" Nov 04 04:55:57.826967 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:55:57.826912 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbllm\" (UniqueName: \"kubernetes.io/projected/03dca812-45a7-42e0-81ce-9dad9861776c-kube-api-access-fbllm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-vkcnr\" (UID: \"03dca812-45a7-42e0-81ce-9dad9861776c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vkcnr" Nov 04 04:55:57.874877 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:55:57.874852 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vkcnr" Nov 04 04:55:58.243093 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:55:58.243067 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-vkcnr"] Nov 04 04:55:58.246289 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:55:58.246264 4509 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03dca812_45a7_42e0_81ce_9dad9861776c.slice/crio-fa74d2ba662bffa3cff7aca5b798c7ce48b816836c0200fcb63e6e2d79a6a0f7 WatchSource:0}: Error finding container fa74d2ba662bffa3cff7aca5b798c7ce48b816836c0200fcb63e6e2d79a6a0f7: Status 404 returned error can't find the container with id fa74d2ba662bffa3cff7aca5b798c7ce48b816836c0200fcb63e6e2d79a6a0f7 Nov 04 04:55:58.271564 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:55:58.271532 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vkcnr" event={"ID":"03dca812-45a7-42e0-81ce-9dad9861776c","Type":"ContainerStarted","Data":"fa74d2ba662bffa3cff7aca5b798c7ce48b816836c0200fcb63e6e2d79a6a0f7"} Nov 04 04:56:01.068690 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:01.068656 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-42dnm"] Nov 04 04:56:01.285272 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:01.285242 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vkcnr" event={"ID":"03dca812-45a7-42e0-81ce-9dad9861776c","Type":"ContainerStarted","Data":"ce5952065b7124cff726e0b90119b86b5d3b1cb0dee9fa0e8a95a5b5dbcbbf1f"} Nov 04 04:56:01.400720 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:01.400643 4509 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vkcnr" podStartSLOduration=2.184431483 podStartE2EDuration="4.400631699s" totalImagesPullingTime="2.216200216s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 04:55:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:56:01.399861842 +0000 UTC m=+525.277585363" watchObservedRunningTime="2026-11-04 04:56:01.400631699 +0000 UTC m=+525.278355219" Nov 04 04:56:01.875101 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:01.875077 4509 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vkcnr" Nov 04 04:56:07.727228 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:07.727201 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-42dnm"] Nov 04 04:56:07.875899 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:07.875868 4509 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vkcnr" Nov 04 04:56:07.888075 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:07.888053 4509 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vkcnr" Nov 04 04:56:08.324206 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:08.324173 4509 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vkcnr" Nov 04 04:56:08.441825 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:08.441786 4509 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-qrvq5"] Nov 04 04:56:08.447369 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:08.447344 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-qrvq5" Nov 04 04:56:08.499696 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:08.499675 4509 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Nov 04 04:56:08.545300 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:08.545274 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-qrvq5"] Nov 04 04:56:08.547797 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:08.547771 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5162291-7c0d-4a1d-93bb-35b19fe7582b-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-qrvq5\" (UID: \"d5162291-7c0d-4a1d-93bb-35b19fe7582b\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-qrvq5" Nov 04 04:56:08.547934 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:08.547840 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hg247\" (UniqueName: \"kubernetes.io/projected/d5162291-7c0d-4a1d-93bb-35b19fe7582b-kube-api-access-hg247\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-qrvq5\" (UID: \"d5162291-7c0d-4a1d-93bb-35b19fe7582b\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-qrvq5" Nov 04 04:56:08.547934 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:08.547905 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5162291-7c0d-4a1d-93bb-35b19fe7582b-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-qrvq5\" (UID: \"d5162291-7c0d-4a1d-93bb-35b19fe7582b\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-qrvq5" Nov 04 04:56:08.547934 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:08.547927 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5162291-7c0d-4a1d-93bb-35b19fe7582b-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-qrvq5\" (UID: \"d5162291-7c0d-4a1d-93bb-35b19fe7582b\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-qrvq5" Nov 04 04:56:08.548101 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:08.547943 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5162291-7c0d-4a1d-93bb-35b19fe7582b-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-qrvq5\" (UID: \"d5162291-7c0d-4a1d-93bb-35b19fe7582b\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-qrvq5" Nov 04 04:56:08.548101 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:08.547976 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5162291-7c0d-4a1d-93bb-35b19fe7582b-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-qrvq5\" (UID: \"d5162291-7c0d-4a1d-93bb-35b19fe7582b\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-qrvq5" Nov 04 04:56:08.649093 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:08.649032 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5162291-7c0d-4a1d-93bb-35b19fe7582b-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-qrvq5\" (UID: \"d5162291-7c0d-4a1d-93bb-35b19fe7582b\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-qrvq5" Nov 04 04:56:08.649093 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:08.649068 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5162291-7c0d-4a1d-93bb-35b19fe7582b-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-qrvq5\" (UID: \"d5162291-7c0d-4a1d-93bb-35b19fe7582b\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-qrvq5" Nov 04 04:56:08.649233 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:08.649114 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hg247\" (UniqueName: \"kubernetes.io/projected/d5162291-7c0d-4a1d-93bb-35b19fe7582b-kube-api-access-hg247\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-qrvq5\" (UID: \"d5162291-7c0d-4a1d-93bb-35b19fe7582b\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-qrvq5" Nov 04 04:56:08.649233 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:08.649150 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5162291-7c0d-4a1d-93bb-35b19fe7582b-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-qrvq5\" (UID: \"d5162291-7c0d-4a1d-93bb-35b19fe7582b\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-qrvq5" Nov 04 04:56:08.649233 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:08.649176 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5162291-7c0d-4a1d-93bb-35b19fe7582b-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-qrvq5\" (UID: \"d5162291-7c0d-4a1d-93bb-35b19fe7582b\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-qrvq5" Nov 04 04:56:08.649233 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:08.649199 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5162291-7c0d-4a1d-93bb-35b19fe7582b-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-qrvq5\" (UID: \"d5162291-7c0d-4a1d-93bb-35b19fe7582b\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-qrvq5" Nov 04 04:56:08.649584 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:08.649563 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5162291-7c0d-4a1d-93bb-35b19fe7582b-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-qrvq5\" (UID: \"d5162291-7c0d-4a1d-93bb-35b19fe7582b\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-qrvq5" Nov 04 04:56:08.649584 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:08.649573 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5162291-7c0d-4a1d-93bb-35b19fe7582b-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-qrvq5\" (UID: \"d5162291-7c0d-4a1d-93bb-35b19fe7582b\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-qrvq5" Nov 04 04:56:08.649707 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:08.649627 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5162291-7c0d-4a1d-93bb-35b19fe7582b-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-qrvq5\" (UID: \"d5162291-7c0d-4a1d-93bb-35b19fe7582b\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-qrvq5" Nov 04 04:56:08.651415 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:08.651397 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5162291-7c0d-4a1d-93bb-35b19fe7582b-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-qrvq5\" (UID: \"d5162291-7c0d-4a1d-93bb-35b19fe7582b\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-qrvq5" Nov 04 04:56:08.651618 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:08.651601 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5162291-7c0d-4a1d-93bb-35b19fe7582b-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-qrvq5\" (UID: \"d5162291-7c0d-4a1d-93bb-35b19fe7582b\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-qrvq5" Nov 04 04:56:08.709575 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:08.709553 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hg247\" (UniqueName: \"kubernetes.io/projected/d5162291-7c0d-4a1d-93bb-35b19fe7582b-kube-api-access-hg247\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-qrvq5\" (UID: \"d5162291-7c0d-4a1d-93bb-35b19fe7582b\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-qrvq5" Nov 04 04:56:08.759837 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:08.759816 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-qrvq5" Nov 04 04:56:09.127981 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:09.127953 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-qrvq5"] Nov 04 04:56:09.130770 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:56:09.130742 4509 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5162291_7c0d_4a1d_93bb_35b19fe7582b.slice/crio-c4e8880138e14f124434eb745f372a5a95faa3d43ef9d606a2a845ec039a8c1c WatchSource:0}: Error finding container c4e8880138e14f124434eb745f372a5a95faa3d43ef9d606a2a845ec039a8c1c: Status 404 returned error can't find the container with id c4e8880138e14f124434eb745f372a5a95faa3d43ef9d606a2a845ec039a8c1c Nov 04 04:56:09.315347 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:09.315309 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-qrvq5" event={"ID":"d5162291-7c0d-4a1d-93bb-35b19fe7582b","Type":"ContainerStarted","Data":"c4e8880138e14f124434eb745f372a5a95faa3d43ef9d606a2a845ec039a8c1c"} Nov 04 04:56:10.319823 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:10.319786 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-qrvq5" event={"ID":"d5162291-7c0d-4a1d-93bb-35b19fe7582b","Type":"ContainerStarted","Data":"7ea179fa3ab25456b6eb302ffa8c4d385cd0cb828b40ee3a5f1bd03397324c8b"} Nov 04 04:56:10.433097 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:10.433059 4509 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-qrvq5" podStartSLOduration=1.842722018 podStartE2EDuration="2.433048209s" totalImagesPullingTime="590.326191ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 04:56:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:56:10.432530544 +0000 UTC m=+534.310254060" watchObservedRunningTime="2026-11-04 04:56:10.433048209 +0000 UTC m=+534.310771729" Nov 04 04:56:10.760637 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:10.760616 4509 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-qrvq5" Nov 04 04:56:12.427633 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:12.427601 4509 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qsr5h"] Nov 04 04:56:12.430747 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:12.430728 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qsr5h" Nov 04 04:56:12.489957 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:12.489932 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qsr5h"] Nov 04 04:56:12.534717 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:12.534696 4509 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Nov 04 04:56:12.580393 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:12.580367 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b59b3e50-b834-4624-b100-46a7df2e159f-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-qsr5h\" (UID: \"b59b3e50-b834-4624-b100-46a7df2e159f\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qsr5h" Nov 04 04:56:12.580501 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:12.580398 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b59b3e50-b834-4624-b100-46a7df2e159f-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-qsr5h\" (UID: \"b59b3e50-b834-4624-b100-46a7df2e159f\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qsr5h" Nov 04 04:56:12.580501 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:12.580415 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kx4b\" (UniqueName: \"kubernetes.io/projected/b59b3e50-b834-4624-b100-46a7df2e159f-kube-api-access-4kx4b\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-qsr5h\" (UID: \"b59b3e50-b834-4624-b100-46a7df2e159f\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qsr5h" Nov 04 04:56:12.580501 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:12.580437 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b59b3e50-b834-4624-b100-46a7df2e159f-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-qsr5h\" (UID: \"b59b3e50-b834-4624-b100-46a7df2e159f\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qsr5h" Nov 04 04:56:12.580615 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:12.580547 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b59b3e50-b834-4624-b100-46a7df2e159f-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-qsr5h\" (UID: \"b59b3e50-b834-4624-b100-46a7df2e159f\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qsr5h" Nov 04 04:56:12.580615 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:12.580606 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b59b3e50-b834-4624-b100-46a7df2e159f-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-qsr5h\" (UID: \"b59b3e50-b834-4624-b100-46a7df2e159f\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qsr5h" Nov 04 04:56:12.681634 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:12.681579 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b59b3e50-b834-4624-b100-46a7df2e159f-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-qsr5h\" (UID: \"b59b3e50-b834-4624-b100-46a7df2e159f\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qsr5h" Nov 04 04:56:12.681634 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:12.681606 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b59b3e50-b834-4624-b100-46a7df2e159f-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-qsr5h\" (UID: \"b59b3e50-b834-4624-b100-46a7df2e159f\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qsr5h" Nov 04 04:56:12.681634 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:12.681625 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kx4b\" (UniqueName: \"kubernetes.io/projected/b59b3e50-b834-4624-b100-46a7df2e159f-kube-api-access-4kx4b\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-qsr5h\" (UID: \"b59b3e50-b834-4624-b100-46a7df2e159f\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qsr5h" Nov 04 04:56:12.681814 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:12.681646 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b59b3e50-b834-4624-b100-46a7df2e159f-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-qsr5h\" (UID: \"b59b3e50-b834-4624-b100-46a7df2e159f\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qsr5h" Nov 04 04:56:12.681814 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:12.681673 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b59b3e50-b834-4624-b100-46a7df2e159f-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-qsr5h\" (UID: \"b59b3e50-b834-4624-b100-46a7df2e159f\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qsr5h" Nov 04 04:56:12.681814 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:12.681720 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b59b3e50-b834-4624-b100-46a7df2e159f-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-qsr5h\" (UID: \"b59b3e50-b834-4624-b100-46a7df2e159f\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qsr5h" Nov 04 04:56:12.682089 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:12.682063 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b59b3e50-b834-4624-b100-46a7df2e159f-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-qsr5h\" (UID: \"b59b3e50-b834-4624-b100-46a7df2e159f\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qsr5h" Nov 04 04:56:12.682179 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:12.682104 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b59b3e50-b834-4624-b100-46a7df2e159f-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-qsr5h\" (UID: \"b59b3e50-b834-4624-b100-46a7df2e159f\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qsr5h" Nov 04 04:56:12.682179 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:12.682120 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b59b3e50-b834-4624-b100-46a7df2e159f-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-qsr5h\" (UID: \"b59b3e50-b834-4624-b100-46a7df2e159f\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qsr5h" Nov 04 04:56:12.683949 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:12.683918 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b59b3e50-b834-4624-b100-46a7df2e159f-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-qsr5h\" (UID: \"b59b3e50-b834-4624-b100-46a7df2e159f\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qsr5h" Nov 04 04:56:12.684191 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:12.684172 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b59b3e50-b834-4624-b100-46a7df2e159f-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-qsr5h\" (UID: \"b59b3e50-b834-4624-b100-46a7df2e159f\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qsr5h" Nov 04 04:56:12.738145 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:12.738120 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kx4b\" (UniqueName: \"kubernetes.io/projected/b59b3e50-b834-4624-b100-46a7df2e159f-kube-api-access-4kx4b\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-qsr5h\" (UID: \"b59b3e50-b834-4624-b100-46a7df2e159f\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qsr5h" Nov 04 04:56:12.751023 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:12.751002 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qsr5h" Nov 04 04:56:13.118930 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:13.118898 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qsr5h"] Nov 04 04:56:13.122800 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:56:13.122774 4509 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb59b3e50_b834_4624_b100_46a7df2e159f.slice/crio-a4f4b3fcbaf11a4c8c62b7ca3169821218b230d3ddccf9e6ad5685104496112d WatchSource:0}: Error finding container a4f4b3fcbaf11a4c8c62b7ca3169821218b230d3ddccf9e6ad5685104496112d: Status 404 returned error can't find the container with id a4f4b3fcbaf11a4c8c62b7ca3169821218b230d3ddccf9e6ad5685104496112d Nov 04 04:56:13.331635 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:13.331605 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qsr5h" event={"ID":"b59b3e50-b834-4624-b100-46a7df2e159f","Type":"ContainerStarted","Data":"a4f4b3fcbaf11a4c8c62b7ca3169821218b230d3ddccf9e6ad5685104496112d"} Nov 04 04:56:14.337278 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:14.337244 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qsr5h" event={"ID":"b59b3e50-b834-4624-b100-46a7df2e159f","Type":"ContainerStarted","Data":"d9ac26cbdf759f9dc8ae83c6865bcedb2e6ff3be834c644ece16d332ae649248"} Nov 04 04:56:14.450834 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:14.450786 4509 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qsr5h" podStartSLOduration=1.868424385 podStartE2EDuration="2.450769953s" totalImagesPullingTime="582.345568ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 04:56:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:56:14.450035738 +0000 UTC m=+538.327759262" watchObservedRunningTime="2026-11-04 04:56:14.450769953 +0000 UTC m=+538.328493474" Nov 04 04:56:14.458764 ip-10-0-133-180 kubenswrapper[4509]: E1104 04:56:14.458736 4509 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmdhl" containerName="extractor" Nov 04 04:56:14.751675 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:14.751652 4509 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qsr5h" Nov 04 04:56:18.760989 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:18.760955 4509 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-qrvq5" Nov 04 04:56:18.772749 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:18.772722 4509 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-qrvq5" Nov 04 04:56:18.966155 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:18.966126 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-42dnm"] Nov 04 04:56:19.368145 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:19.368123 4509 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-qrvq5" Nov 04 04:56:22.751464 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:22.751433 4509 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qsr5h" Nov 04 04:56:22.763189 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:22.763171 4509 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qsr5h" Nov 04 04:56:22.949305 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:22.949279 4509 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-wc5pz"] Nov 04 04:56:22.953636 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:22.953619 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wc5pz" Nov 04 04:56:22.955755 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:22.955731 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5697f521-fbf5-4d8d-9e09-cfb64fb7827f-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-wc5pz\" (UID: \"5697f521-fbf5-4d8d-9e09-cfb64fb7827f\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wc5pz" Nov 04 04:56:22.955885 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:22.955785 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5697f521-fbf5-4d8d-9e09-cfb64fb7827f-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-wc5pz\" (UID: \"5697f521-fbf5-4d8d-9e09-cfb64fb7827f\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wc5pz" Nov 04 04:56:22.955952 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:22.955883 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5697f521-fbf5-4d8d-9e09-cfb64fb7827f-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-wc5pz\" (UID: \"5697f521-fbf5-4d8d-9e09-cfb64fb7827f\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wc5pz" Nov 04 04:56:22.955952 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:22.955912 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cm8kp\" (UniqueName: \"kubernetes.io/projected/5697f521-fbf5-4d8d-9e09-cfb64fb7827f-kube-api-access-cm8kp\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-wc5pz\" (UID: \"5697f521-fbf5-4d8d-9e09-cfb64fb7827f\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wc5pz" Nov 04 04:56:22.956027 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:22.956006 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5697f521-fbf5-4d8d-9e09-cfb64fb7827f-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-wc5pz\" (UID: \"5697f521-fbf5-4d8d-9e09-cfb64fb7827f\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wc5pz" Nov 04 04:56:22.956061 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:22.956052 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5697f521-fbf5-4d8d-9e09-cfb64fb7827f-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-wc5pz\" (UID: \"5697f521-fbf5-4d8d-9e09-cfb64fb7827f\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wc5pz" Nov 04 04:56:23.009551 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:23.009489 4509 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\"" Nov 04 04:56:23.011350 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:23.011307 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-wc5pz"] Nov 04 04:56:23.056823 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:23.056802 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5697f521-fbf5-4d8d-9e09-cfb64fb7827f-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-wc5pz\" (UID: \"5697f521-fbf5-4d8d-9e09-cfb64fb7827f\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wc5pz" Nov 04 04:56:23.056919 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:23.056841 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5697f521-fbf5-4d8d-9e09-cfb64fb7827f-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-wc5pz\" (UID: \"5697f521-fbf5-4d8d-9e09-cfb64fb7827f\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wc5pz" Nov 04 04:56:23.056919 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:23.056865 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cm8kp\" (UniqueName: \"kubernetes.io/projected/5697f521-fbf5-4d8d-9e09-cfb64fb7827f-kube-api-access-cm8kp\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-wc5pz\" (UID: \"5697f521-fbf5-4d8d-9e09-cfb64fb7827f\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wc5pz" Nov 04 04:56:23.056919 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:23.056905 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5697f521-fbf5-4d8d-9e09-cfb64fb7827f-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-wc5pz\" (UID: \"5697f521-fbf5-4d8d-9e09-cfb64fb7827f\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wc5pz" Nov 04 04:56:23.057082 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:23.057057 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5697f521-fbf5-4d8d-9e09-cfb64fb7827f-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-wc5pz\" (UID: \"5697f521-fbf5-4d8d-9e09-cfb64fb7827f\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wc5pz" Nov 04 04:56:23.057135 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:23.057109 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5697f521-fbf5-4d8d-9e09-cfb64fb7827f-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-wc5pz\" (UID: \"5697f521-fbf5-4d8d-9e09-cfb64fb7827f\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wc5pz" Nov 04 04:56:23.057377 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:23.057349 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5697f521-fbf5-4d8d-9e09-cfb64fb7827f-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-wc5pz\" (UID: \"5697f521-fbf5-4d8d-9e09-cfb64fb7827f\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wc5pz" Nov 04 04:56:23.057495 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:23.057383 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5697f521-fbf5-4d8d-9e09-cfb64fb7827f-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-wc5pz\" (UID: \"5697f521-fbf5-4d8d-9e09-cfb64fb7827f\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wc5pz" Nov 04 04:56:23.057495 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:23.057456 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5697f521-fbf5-4d8d-9e09-cfb64fb7827f-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-wc5pz\" (UID: \"5697f521-fbf5-4d8d-9e09-cfb64fb7827f\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wc5pz" Nov 04 04:56:23.059039 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:23.059019 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5697f521-fbf5-4d8d-9e09-cfb64fb7827f-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-wc5pz\" (UID: \"5697f521-fbf5-4d8d-9e09-cfb64fb7827f\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wc5pz" Nov 04 04:56:23.059396 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:23.059381 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5697f521-fbf5-4d8d-9e09-cfb64fb7827f-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-wc5pz\" (UID: \"5697f521-fbf5-4d8d-9e09-cfb64fb7827f\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wc5pz" Nov 04 04:56:23.113449 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:23.113426 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cm8kp\" (UniqueName: \"kubernetes.io/projected/5697f521-fbf5-4d8d-9e09-cfb64fb7827f-kube-api-access-cm8kp\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-wc5pz\" (UID: \"5697f521-fbf5-4d8d-9e09-cfb64fb7827f\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wc5pz" Nov 04 04:56:23.266119 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:23.266068 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wc5pz" Nov 04 04:56:23.384397 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:23.384366 4509 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qsr5h" Nov 04 04:56:23.631939 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:23.631913 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-wc5pz"] Nov 04 04:56:23.634199 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:56:23.634172 4509 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5697f521_fbf5_4d8d_9e09_cfb64fb7827f.slice/crio-7457bd49cddb46c96e4a03b0be30c6f6bedafc810d12870a84a7eef475b0abcc WatchSource:0}: Error finding container 7457bd49cddb46c96e4a03b0be30c6f6bedafc810d12870a84a7eef475b0abcc: Status 404 returned error can't find the container with id 7457bd49cddb46c96e4a03b0be30c6f6bedafc810d12870a84a7eef475b0abcc Nov 04 04:56:24.378370 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:24.378339 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wc5pz" event={"ID":"5697f521-fbf5-4d8d-9e09-cfb64fb7827f","Type":"ContainerStarted","Data":"7457bd49cddb46c96e4a03b0be30c6f6bedafc810d12870a84a7eef475b0abcc"} Nov 04 04:56:25.383730 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:25.383702 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wc5pz" event={"ID":"5697f521-fbf5-4d8d-9e09-cfb64fb7827f","Type":"ContainerStarted","Data":"04474270f65773c6123ca02814687227f0b9e0dda32dc4465d617c97f3787098"} Nov 04 04:56:25.501552 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:25.501504 4509 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wc5pz" podStartSLOduration=2.497831775 podStartE2EDuration="3.5014929s" totalImagesPullingTime="1.003661125s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 04:56:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:56:25.499694687 +0000 UTC m=+549.377418209" watchObservedRunningTime="2026-11-04 04:56:25.5014929 +0000 UTC m=+549.379216442" Nov 04 04:56:26.266495 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:26.266469 4509 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wc5pz" Nov 04 04:56:30.394315 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:30.394276 4509 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-j8ns4"] Nov 04 04:56:30.397381 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:30.397366 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-j8ns4" Nov 04 04:56:30.409783 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:30.409756 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3914303c-ffca-4fa0-bf97-d15430976da6-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-j8ns4\" (UID: \"3914303c-ffca-4fa0-bf97-d15430976da6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-j8ns4" Nov 04 04:56:30.409893 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:30.409792 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3914303c-ffca-4fa0-bf97-d15430976da6-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-j8ns4\" (UID: \"3914303c-ffca-4fa0-bf97-d15430976da6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-j8ns4" Nov 04 04:56:30.409893 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:30.409822 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3914303c-ffca-4fa0-bf97-d15430976da6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-j8ns4\" (UID: \"3914303c-ffca-4fa0-bf97-d15430976da6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-j8ns4" Nov 04 04:56:30.409893 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:30.409879 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3914303c-ffca-4fa0-bf97-d15430976da6-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-j8ns4\" (UID: \"3914303c-ffca-4fa0-bf97-d15430976da6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-j8ns4" Nov 04 04:56:30.409990 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:30.409916 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3914303c-ffca-4fa0-bf97-d15430976da6-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-j8ns4\" (UID: \"3914303c-ffca-4fa0-bf97-d15430976da6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-j8ns4" Nov 04 04:56:30.409990 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:30.409931 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3914303c-ffca-4fa0-bf97-d15430976da6-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-j8ns4\" (UID: \"3914303c-ffca-4fa0-bf97-d15430976da6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-j8ns4" Nov 04 04:56:30.409990 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:30.409958 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3914303c-ffca-4fa0-bf97-d15430976da6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-j8ns4\" (UID: \"3914303c-ffca-4fa0-bf97-d15430976da6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-j8ns4" Nov 04 04:56:30.410078 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:30.410049 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3914303c-ffca-4fa0-bf97-d15430976da6-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-j8ns4\" (UID: \"3914303c-ffca-4fa0-bf97-d15430976da6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-j8ns4" Nov 04 04:56:30.410110 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:30.410089 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkgkw\" (UniqueName: \"kubernetes.io/projected/3914303c-ffca-4fa0-bf97-d15430976da6-kube-api-access-lkgkw\") pod \"maas-default-gateway-openshift-default-6886b96f9-j8ns4\" (UID: \"3914303c-ffca-4fa0-bf97-d15430976da6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-j8ns4" Nov 04 04:56:30.444307 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:30.444286 4509 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4sj9h"] Nov 04 04:56:30.447250 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:30.447237 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4sj9h" Nov 04 04:56:30.455905 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:30.455888 4509 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-p9hzm\"" Nov 04 04:56:30.499767 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:30.499745 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-j8ns4"] Nov 04 04:56:30.511123 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:30.511103 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkgkw\" (UniqueName: \"kubernetes.io/projected/3914303c-ffca-4fa0-bf97-d15430976da6-kube-api-access-lkgkw\") pod \"maas-default-gateway-openshift-default-6886b96f9-j8ns4\" (UID: \"3914303c-ffca-4fa0-bf97-d15430976da6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-j8ns4" Nov 04 04:56:30.511221 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:30.511138 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e057215d-2b26-47c6-8c61-3d1413ef6aec-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-4sj9h\" (UID: \"e057215d-2b26-47c6-8c61-3d1413ef6aec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4sj9h" Nov 04 04:56:30.511221 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:30.511156 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e057215d-2b26-47c6-8c61-3d1413ef6aec-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-4sj9h\" (UID: \"e057215d-2b26-47c6-8c61-3d1413ef6aec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4sj9h" Nov 04 04:56:30.511221 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:30.511176 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e057215d-2b26-47c6-8c61-3d1413ef6aec-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-4sj9h\" (UID: \"e057215d-2b26-47c6-8c61-3d1413ef6aec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4sj9h" Nov 04 04:56:30.511398 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:30.511238 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3914303c-ffca-4fa0-bf97-d15430976da6-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-j8ns4\" (UID: \"3914303c-ffca-4fa0-bf97-d15430976da6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-j8ns4" Nov 04 04:56:30.511398 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:30.511318 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3914303c-ffca-4fa0-bf97-d15430976da6-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-j8ns4\" (UID: \"3914303c-ffca-4fa0-bf97-d15430976da6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-j8ns4" Nov 04 04:56:30.511398 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:30.511368 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3914303c-ffca-4fa0-bf97-d15430976da6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-j8ns4\" (UID: \"3914303c-ffca-4fa0-bf97-d15430976da6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-j8ns4" Nov 04 04:56:30.511542 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:30.511496 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3914303c-ffca-4fa0-bf97-d15430976da6-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-j8ns4\" (UID: \"3914303c-ffca-4fa0-bf97-d15430976da6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-j8ns4" Nov 04 04:56:30.511542 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:30.511531 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e057215d-2b26-47c6-8c61-3d1413ef6aec-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-4sj9h\" (UID: \"e057215d-2b26-47c6-8c61-3d1413ef6aec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4sj9h" Nov 04 04:56:30.511622 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:30.511562 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e057215d-2b26-47c6-8c61-3d1413ef6aec-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-4sj9h\" (UID: \"e057215d-2b26-47c6-8c61-3d1413ef6aec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4sj9h" Nov 04 04:56:30.511622 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:30.511587 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e057215d-2b26-47c6-8c61-3d1413ef6aec-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-4sj9h\" (UID: \"e057215d-2b26-47c6-8c61-3d1413ef6aec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4sj9h" Nov 04 04:56:30.511622 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:30.511612 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5tts\" (UniqueName: \"kubernetes.io/projected/e057215d-2b26-47c6-8c61-3d1413ef6aec-kube-api-access-l5tts\") pod \"maas-default-gateway-openshift-default-6886b96f9-4sj9h\" (UID: \"e057215d-2b26-47c6-8c61-3d1413ef6aec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4sj9h" Nov 04 04:56:30.511759 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:30.511646 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e057215d-2b26-47c6-8c61-3d1413ef6aec-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-4sj9h\" (UID: \"e057215d-2b26-47c6-8c61-3d1413ef6aec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4sj9h" Nov 04 04:56:30.511759 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:30.511680 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3914303c-ffca-4fa0-bf97-d15430976da6-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-j8ns4\" (UID: \"3914303c-ffca-4fa0-bf97-d15430976da6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-j8ns4" Nov 04 04:56:30.511759 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:30.511702 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3914303c-ffca-4fa0-bf97-d15430976da6-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-j8ns4\" (UID: \"3914303c-ffca-4fa0-bf97-d15430976da6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-j8ns4" Nov 04 04:56:30.511759 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:30.511710 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3914303c-ffca-4fa0-bf97-d15430976da6-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-j8ns4\" (UID: \"3914303c-ffca-4fa0-bf97-d15430976da6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-j8ns4" Nov 04 04:56:30.511759 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:30.511728 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3914303c-ffca-4fa0-bf97-d15430976da6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-j8ns4\" (UID: \"3914303c-ffca-4fa0-bf97-d15430976da6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-j8ns4" Nov 04 04:56:30.511940 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:30.511858 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3914303c-ffca-4fa0-bf97-d15430976da6-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-j8ns4\" (UID: \"3914303c-ffca-4fa0-bf97-d15430976da6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-j8ns4" Nov 04 04:56:30.512138 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:30.512094 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e057215d-2b26-47c6-8c61-3d1413ef6aec-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-4sj9h\" (UID: \"e057215d-2b26-47c6-8c61-3d1413ef6aec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4sj9h" Nov 04 04:56:30.512239 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:30.512131 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3914303c-ffca-4fa0-bf97-d15430976da6-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-j8ns4\" (UID: \"3914303c-ffca-4fa0-bf97-d15430976da6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-j8ns4" Nov 04 04:56:30.512239 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:30.512214 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3914303c-ffca-4fa0-bf97-d15430976da6-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-j8ns4\" (UID: \"3914303c-ffca-4fa0-bf97-d15430976da6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-j8ns4" Nov 04 04:56:30.512361 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:30.512217 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3914303c-ffca-4fa0-bf97-d15430976da6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-j8ns4\" (UID: \"3914303c-ffca-4fa0-bf97-d15430976da6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-j8ns4" Nov 04 04:56:30.513531 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:30.513497 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3914303c-ffca-4fa0-bf97-d15430976da6-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-j8ns4\" (UID: \"3914303c-ffca-4fa0-bf97-d15430976da6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-j8ns4" Nov 04 04:56:30.514386 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:30.514362 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3914303c-ffca-4fa0-bf97-d15430976da6-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-j8ns4\" (UID: \"3914303c-ffca-4fa0-bf97-d15430976da6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-j8ns4" Nov 04 04:56:30.514530 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:30.514513 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3914303c-ffca-4fa0-bf97-d15430976da6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-j8ns4\" (UID: \"3914303c-ffca-4fa0-bf97-d15430976da6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-j8ns4" Nov 04 04:56:30.549384 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:30.549361 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4sj9h"] Nov 04 04:56:30.569001 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:30.568973 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkgkw\" (UniqueName: \"kubernetes.io/projected/3914303c-ffca-4fa0-bf97-d15430976da6-kube-api-access-lkgkw\") pod \"maas-default-gateway-openshift-default-6886b96f9-j8ns4\" (UID: \"3914303c-ffca-4fa0-bf97-d15430976da6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-j8ns4" Nov 04 04:56:30.569078 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:30.569011 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3914303c-ffca-4fa0-bf97-d15430976da6-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-j8ns4\" (UID: \"3914303c-ffca-4fa0-bf97-d15430976da6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-j8ns4" Nov 04 04:56:30.612552 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:30.612534 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e057215d-2b26-47c6-8c61-3d1413ef6aec-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-4sj9h\" (UID: \"e057215d-2b26-47c6-8c61-3d1413ef6aec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4sj9h" Nov 04 04:56:30.612641 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:30.612560 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e057215d-2b26-47c6-8c61-3d1413ef6aec-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-4sj9h\" (UID: \"e057215d-2b26-47c6-8c61-3d1413ef6aec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4sj9h" Nov 04 04:56:30.612641 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:30.612575 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e057215d-2b26-47c6-8c61-3d1413ef6aec-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-4sj9h\" (UID: \"e057215d-2b26-47c6-8c61-3d1413ef6aec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4sj9h" Nov 04 04:56:30.612641 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:30.612591 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5tts\" (UniqueName: \"kubernetes.io/projected/e057215d-2b26-47c6-8c61-3d1413ef6aec-kube-api-access-l5tts\") pod \"maas-default-gateway-openshift-default-6886b96f9-4sj9h\" (UID: \"e057215d-2b26-47c6-8c61-3d1413ef6aec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4sj9h" Nov 04 04:56:30.612641 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:30.612622 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e057215d-2b26-47c6-8c61-3d1413ef6aec-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-4sj9h\" (UID: \"e057215d-2b26-47c6-8c61-3d1413ef6aec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4sj9h" Nov 04 04:56:30.612821 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:30.612672 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e057215d-2b26-47c6-8c61-3d1413ef6aec-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-4sj9h\" (UID: \"e057215d-2b26-47c6-8c61-3d1413ef6aec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4sj9h" Nov 04 04:56:30.612821 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:30.612738 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e057215d-2b26-47c6-8c61-3d1413ef6aec-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-4sj9h\" (UID: \"e057215d-2b26-47c6-8c61-3d1413ef6aec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4sj9h" Nov 04 04:56:30.612922 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:30.612861 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e057215d-2b26-47c6-8c61-3d1413ef6aec-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-4sj9h\" (UID: \"e057215d-2b26-47c6-8c61-3d1413ef6aec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4sj9h" Nov 04 04:56:30.612922 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:30.612902 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e057215d-2b26-47c6-8c61-3d1413ef6aec-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-4sj9h\" (UID: \"e057215d-2b26-47c6-8c61-3d1413ef6aec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4sj9h" Nov 04 04:56:30.613021 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:30.612922 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e057215d-2b26-47c6-8c61-3d1413ef6aec-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-4sj9h\" (UID: \"e057215d-2b26-47c6-8c61-3d1413ef6aec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4sj9h" Nov 04 04:56:30.613075 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:30.613033 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e057215d-2b26-47c6-8c61-3d1413ef6aec-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-4sj9h\" (UID: \"e057215d-2b26-47c6-8c61-3d1413ef6aec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4sj9h" Nov 04 04:56:30.613161 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:30.613136 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e057215d-2b26-47c6-8c61-3d1413ef6aec-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-4sj9h\" (UID: \"e057215d-2b26-47c6-8c61-3d1413ef6aec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4sj9h" Nov 04 04:56:30.613254 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:30.613242 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e057215d-2b26-47c6-8c61-3d1413ef6aec-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-4sj9h\" (UID: \"e057215d-2b26-47c6-8c61-3d1413ef6aec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4sj9h" Nov 04 04:56:30.613305 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:30.613279 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e057215d-2b26-47c6-8c61-3d1413ef6aec-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-4sj9h\" (UID: \"e057215d-2b26-47c6-8c61-3d1413ef6aec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4sj9h" Nov 04 04:56:30.615265 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:30.615245 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e057215d-2b26-47c6-8c61-3d1413ef6aec-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-4sj9h\" (UID: \"e057215d-2b26-47c6-8c61-3d1413ef6aec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4sj9h" Nov 04 04:56:30.615417 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:30.615400 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e057215d-2b26-47c6-8c61-3d1413ef6aec-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-4sj9h\" (UID: \"e057215d-2b26-47c6-8c61-3d1413ef6aec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4sj9h" Nov 04 04:56:30.669734 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:30.669680 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e057215d-2b26-47c6-8c61-3d1413ef6aec-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-4sj9h\" (UID: \"e057215d-2b26-47c6-8c61-3d1413ef6aec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4sj9h" Nov 04 04:56:30.669839 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:30.669798 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5tts\" (UniqueName: \"kubernetes.io/projected/e057215d-2b26-47c6-8c61-3d1413ef6aec-kube-api-access-l5tts\") pod \"maas-default-gateway-openshift-default-6886b96f9-4sj9h\" (UID: \"e057215d-2b26-47c6-8c61-3d1413ef6aec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4sj9h" Nov 04 04:56:30.709401 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:30.709368 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-j8ns4" Nov 04 04:56:30.760449 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:30.758855 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4sj9h" Nov 04 04:56:31.130768 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:31.130740 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-j8ns4"] Nov 04 04:56:31.172032 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:56:31.172005 4509 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3914303c_ffca_4fa0_bf97_d15430976da6.slice/crio-df1c7a88a578f8018c5d2028bd196f2134449580388eb6e9a7198e8baae5cce3 WatchSource:0}: Error finding container df1c7a88a578f8018c5d2028bd196f2134449580388eb6e9a7198e8baae5cce3: Status 404 returned error can't find the container with id df1c7a88a578f8018c5d2028bd196f2134449580388eb6e9a7198e8baae5cce3 Nov 04 04:56:31.190180 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:31.185597 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4sj9h"] Nov 04 04:56:31.407763 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:31.407701 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-j8ns4" event={"ID":"3914303c-ffca-4fa0-bf97-d15430976da6","Type":"ContainerStarted","Data":"df1c7a88a578f8018c5d2028bd196f2134449580388eb6e9a7198e8baae5cce3"} Nov 04 04:56:31.408807 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:31.408786 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4sj9h" event={"ID":"e057215d-2b26-47c6-8c61-3d1413ef6aec","Type":"ContainerStarted","Data":"4eebb20a722089077c6d6d0b52894298c75d95bad6f3ffbe5b72413b8e022982"} Nov 04 04:56:31.450473 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:31.450446 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-42dnm"] Nov 04 04:56:33.267302 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:33.267269 4509 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wc5pz" Nov 04 04:56:33.283784 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:33.283760 4509 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wc5pz" Nov 04 04:56:33.431159 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:33.431096 4509 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wc5pz" Nov 04 04:56:35.158005 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:35.157974 4509 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 04 04:56:35.158287 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:35.158041 4509 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 04 04:56:35.158287 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:35.158069 4509 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 04 04:56:35.165749 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:35.165717 4509 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 04 04:56:35.165847 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:35.165791 4509 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 04 04:56:35.165847 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:35.165827 4509 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 04 04:56:35.430561 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:35.430449 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-j8ns4" event={"ID":"3914303c-ffca-4fa0-bf97-d15430976da6","Type":"ContainerStarted","Data":"e85fa7471bbfc80c1eafbaace18a80c16a30420142fe26f0bd8d48ce9842caae"} Nov 04 04:56:35.432583 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:35.432544 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4sj9h" event={"ID":"e057215d-2b26-47c6-8c61-3d1413ef6aec","Type":"ContainerStarted","Data":"673bd1903b0a40d633c225927d842cf9cd6b453d71b98ecc7498c4fcf1e85f40"} Nov 04 04:56:35.650921 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:35.649845 4509 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-j8ns4" podStartSLOduration=1.6666130049999999 podStartE2EDuration="5.649826105s" totalImagesPullingTime="3.9832131s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 04:56:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:56:35.597479909 +0000 UTC m=+559.475203431" watchObservedRunningTime="2026-11-04 04:56:35.649826105 +0000 UTC m=+559.527549627" Nov 04 04:56:35.714568 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:35.714536 4509 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-j8ns4" Nov 04 04:56:35.716652 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:35.716597 4509 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4sj9h" podStartSLOduration=1.742549789 podStartE2EDuration="5.716583585s" totalImagesPullingTime="3.974033796s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 04:56:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:56:35.71496138 +0000 UTC m=+559.592684902" watchObservedRunningTime="2026-11-04 04:56:35.716583585 +0000 UTC m=+559.594307104" Nov 04 04:56:35.770346 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:35.765415 4509 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4sj9h" Nov 04 04:56:36.712498 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:36.712461 4509 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-j8ns4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body= Nov 04 04:56:36.712995 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:36.712509 4509 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-j8ns4" podUID="3914303c-ffca-4fa0-bf97-d15430976da6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" Nov 04 04:56:36.780373 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:36.780305 4509 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-4sj9h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Nov 04 04:56:36.780620 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:36.780596 4509 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4sj9h" podUID="e057215d-2b26-47c6-8c61-3d1413ef6aec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Nov 04 04:56:37.714102 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:37.714069 4509 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-j8ns4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body= Nov 04 04:56:37.714581 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:37.714132 4509 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-j8ns4" podUID="3914303c-ffca-4fa0-bf97-d15430976da6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" Nov 04 04:56:37.766351 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:37.763646 4509 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-4sj9h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Nov 04 04:56:37.766351 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:37.763703 4509 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4sj9h" podUID="e057215d-2b26-47c6-8c61-3d1413ef6aec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Nov 04 04:56:38.711736 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:38.711702 4509 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-j8ns4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body= Nov 04 04:56:38.711890 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:38.711749 4509 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-j8ns4" podUID="3914303c-ffca-4fa0-bf97-d15430976da6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" Nov 04 04:56:38.759422 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:38.759394 4509 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-4sj9h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Nov 04 04:56:38.759938 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:38.759909 4509 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4sj9h" podUID="e057215d-2b26-47c6-8c61-3d1413ef6aec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Nov 04 04:56:39.710742 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:39.710712 4509 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-j8ns4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body= Nov 04 04:56:39.710960 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:39.710941 4509 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-j8ns4" podUID="3914303c-ffca-4fa0-bf97-d15430976da6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" Nov 04 04:56:39.768347 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:39.759469 4509 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-4sj9h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Nov 04 04:56:39.768347 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:39.759522 4509 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4sj9h" podUID="e057215d-2b26-47c6-8c61-3d1413ef6aec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Nov 04 04:56:40.710461 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:40.710430 4509 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-j8ns4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body= Nov 04 04:56:40.710708 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:40.710673 4509 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-j8ns4" podUID="3914303c-ffca-4fa0-bf97-d15430976da6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" Nov 04 04:56:40.763635 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:40.763604 4509 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-4sj9h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Nov 04 04:56:40.763863 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:40.763838 4509 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4sj9h" podUID="e057215d-2b26-47c6-8c61-3d1413ef6aec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Nov 04 04:56:41.710487 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:41.710451 4509 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-j8ns4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body= Nov 04 04:56:41.710855 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:41.710511 4509 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-j8ns4" podUID="3914303c-ffca-4fa0-bf97-d15430976da6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" Nov 04 04:56:41.760465 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:41.760432 4509 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-4sj9h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Nov 04 04:56:41.760567 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:41.760487 4509 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4sj9h" podUID="e057215d-2b26-47c6-8c61-3d1413ef6aec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Nov 04 04:56:43.557101 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:43.557072 4509 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4sj9h" Nov 04 04:56:43.558398 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:43.557744 4509 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4sj9h" Nov 04 04:56:43.558956 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:43.558937 4509 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4sj9h" Nov 04 04:56:43.571577 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:43.571557 4509 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-j8ns4" Nov 04 04:56:43.571822 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:43.571802 4509 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-j8ns4" Nov 04 04:56:43.572598 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:43.572583 4509 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-j8ns4" Nov 04 04:56:43.934839 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:43.934770 4509 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-775668dcd7-z2z29"] Nov 04 04:56:43.939255 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:43.939238 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-775668dcd7-z2z29" Nov 04 04:56:43.992375 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:43.992349 4509 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-zfb5x\"" Nov 04 04:56:43.993011 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:43.992989 4509 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Nov 04 04:56:43.994049 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:43.994029 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-775668dcd7-z2z29"] Nov 04 04:56:44.074309 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:44.074282 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76cd9\" (UniqueName: \"kubernetes.io/projected/d90081c2-08dc-4d6d-b1eb-876a5e27584a-kube-api-access-76cd9\") pod \"authorino-775668dcd7-z2z29\" (UID: \"d90081c2-08dc-4d6d-b1eb-876a5e27584a\") " pod="kuadrant-system/authorino-775668dcd7-z2z29" Nov 04 04:56:44.074478 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:44.074356 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d90081c2-08dc-4d6d-b1eb-876a5e27584a-tls-cert\") pod \"authorino-775668dcd7-z2z29\" (UID: \"d90081c2-08dc-4d6d-b1eb-876a5e27584a\") " pod="kuadrant-system/authorino-775668dcd7-z2z29" Nov 04 04:56:44.175570 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:44.175541 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76cd9\" (UniqueName: \"kubernetes.io/projected/d90081c2-08dc-4d6d-b1eb-876a5e27584a-kube-api-access-76cd9\") pod \"authorino-775668dcd7-z2z29\" (UID: \"d90081c2-08dc-4d6d-b1eb-876a5e27584a\") " pod="kuadrant-system/authorino-775668dcd7-z2z29" Nov 04 04:56:44.175570 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:44.175575 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d90081c2-08dc-4d6d-b1eb-876a5e27584a-tls-cert\") pod \"authorino-775668dcd7-z2z29\" (UID: \"d90081c2-08dc-4d6d-b1eb-876a5e27584a\") " pod="kuadrant-system/authorino-775668dcd7-z2z29" Nov 04 04:56:44.177979 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:44.177947 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d90081c2-08dc-4d6d-b1eb-876a5e27584a-tls-cert\") pod \"authorino-775668dcd7-z2z29\" (UID: \"d90081c2-08dc-4d6d-b1eb-876a5e27584a\") " pod="kuadrant-system/authorino-775668dcd7-z2z29" Nov 04 04:56:44.236217 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:44.236187 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76cd9\" (UniqueName: \"kubernetes.io/projected/d90081c2-08dc-4d6d-b1eb-876a5e27584a-kube-api-access-76cd9\") pod \"authorino-775668dcd7-z2z29\" (UID: \"d90081c2-08dc-4d6d-b1eb-876a5e27584a\") " pod="kuadrant-system/authorino-775668dcd7-z2z29" Nov 04 04:56:44.250753 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:44.250734 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-775668dcd7-z2z29" Nov 04 04:56:44.452838 ip-10-0-133-180 kubenswrapper[4509]: E1104 04:56:44.452804 4509 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmdhl" containerName="extractor" Nov 04 04:56:44.620598 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:44.618698 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-775668dcd7-z2z29"] Nov 04 04:56:44.621361 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:56:44.621318 4509 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd90081c2_08dc_4d6d_b1eb_876a5e27584a.slice/crio-93c2e90568c353772126aa0a20e57cc1c6883f022efdb3322a5bc14e513a00cc WatchSource:0}: Error finding container 93c2e90568c353772126aa0a20e57cc1c6883f022efdb3322a5bc14e513a00cc: Status 404 returned error can't find the container with id 93c2e90568c353772126aa0a20e57cc1c6883f022efdb3322a5bc14e513a00cc Nov 04 04:56:45.477065 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:45.477029 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-775668dcd7-z2z29" event={"ID":"d90081c2-08dc-4d6d-b1eb-876a5e27584a","Type":"ContainerStarted","Data":"93c2e90568c353772126aa0a20e57cc1c6883f022efdb3322a5bc14e513a00cc"} Nov 04 04:56:46.482231 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:46.482203 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-775668dcd7-z2z29" event={"ID":"d90081c2-08dc-4d6d-b1eb-876a5e27584a","Type":"ContainerStarted","Data":"3ea7414e5098afa75c8dc34cd813ddb91b91553be8763d02375070f397419528"} Nov 04 04:56:55.686965 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:55.686907 4509 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-775668dcd7-z2z29" podStartSLOduration=11.117570538 podStartE2EDuration="12.686889211s" totalImagesPullingTime="1.569318673s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 04:56:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:56:46.595412639 +0000 UTC m=+570.473136164" watchObservedRunningTime="2026-11-04 04:56:55.686889211 +0000 UTC m=+579.564612731" Nov 04 04:56:55.687803 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:55.687785 4509 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-n9sxt"] Nov 04 04:56:55.693995 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:55.693974 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-n9sxt" Nov 04 04:56:55.757889 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:55.757856 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-n9sxt"] Nov 04 04:56:55.805576 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:55.805543 4509 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Nov 04 04:56:55.805731 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:55.805606 4509 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-ctwfx\"" Nov 04 04:56:55.809947 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:55.809927 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbe3a747-d280-4042-8cb9-62a148236fd2-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-n9sxt\" (UID: \"dbe3a747-d280-4042-8cb9-62a148236fd2\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-n9sxt" Nov 04 04:56:55.810042 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:55.809954 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjz2q\" (UniqueName: \"kubernetes.io/projected/dbe3a747-d280-4042-8cb9-62a148236fd2-kube-api-access-mjz2q\") pod \"payload-pre-processing-7b98bb678c-n9sxt\" (UID: \"dbe3a747-d280-4042-8cb9-62a148236fd2\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-n9sxt" Nov 04 04:56:55.910849 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:55.910812 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbe3a747-d280-4042-8cb9-62a148236fd2-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-n9sxt\" (UID: \"dbe3a747-d280-4042-8cb9-62a148236fd2\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-n9sxt" Nov 04 04:56:55.911040 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:55.910863 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjz2q\" (UniqueName: \"kubernetes.io/projected/dbe3a747-d280-4042-8cb9-62a148236fd2-kube-api-access-mjz2q\") pod \"payload-pre-processing-7b98bb678c-n9sxt\" (UID: \"dbe3a747-d280-4042-8cb9-62a148236fd2\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-n9sxt" Nov 04 04:56:55.911781 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:55.911756 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbe3a747-d280-4042-8cb9-62a148236fd2-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-n9sxt\" (UID: \"dbe3a747-d280-4042-8cb9-62a148236fd2\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-n9sxt" Nov 04 04:56:55.968753 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:55.968729 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjz2q\" (UniqueName: \"kubernetes.io/projected/dbe3a747-d280-4042-8cb9-62a148236fd2-kube-api-access-mjz2q\") pod \"payload-pre-processing-7b98bb678c-n9sxt\" (UID: \"dbe3a747-d280-4042-8cb9-62a148236fd2\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-n9sxt" Nov 04 04:56:56.006484 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:56.006465 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-n9sxt" Nov 04 04:56:56.384368 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:56.384264 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-n9sxt"] Nov 04 04:56:56.388899 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:56:56.388859 4509 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbe3a747_d280_4042_8cb9_62a148236fd2.slice/crio-7bf90a6e110737d6d4491ebe2b501f8a8e6915f83c621a545c33e78121ca73be WatchSource:0}: Error finding container 7bf90a6e110737d6d4491ebe2b501f8a8e6915f83c621a545c33e78121ca73be: Status 404 returned error can't find the container with id 7bf90a6e110737d6d4491ebe2b501f8a8e6915f83c621a545c33e78121ca73be Nov 04 04:56:56.527881 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:56.527847 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-n9sxt" event={"ID":"dbe3a747-d280-4042-8cb9-62a148236fd2","Type":"ContainerStarted","Data":"7bf90a6e110737d6d4491ebe2b501f8a8e6915f83c621a545c33e78121ca73be"} Nov 04 04:56:59.543628 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:59.543597 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-n9sxt" event={"ID":"dbe3a747-d280-4042-8cb9-62a148236fd2","Type":"ContainerStarted","Data":"77cdfd4ea605190b0d52eafbdf65647d733ac3247e85166f7d78b9643d6e17d7"} Nov 04 04:56:59.544028 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:59.543737 4509 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-n9sxt" Nov 04 04:56:59.659952 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:56:59.659856 4509 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-n9sxt" podStartSLOduration=1.67150388 podStartE2EDuration="4.65984158s" totalImagesPullingTime="2.9883377s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 04:56:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:56:59.659303215 +0000 UTC m=+583.537026737" watchObservedRunningTime="2026-11-04 04:56:59.65984158 +0000 UTC m=+583.537565101" Nov 04 04:57:03.277338 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:57:03.277286 4509 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-64466f8c8d-bg6dc"] Nov 04 04:57:03.284544 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:57:03.284521 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-64466f8c8d-bg6dc" Nov 04 04:57:03.337011 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:57:03.336980 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-64466f8c8d-bg6dc"] Nov 04 04:57:03.380781 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:57:03.380752 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/242c7b88-7028-4b0f-a9e5-2e9fd16bc446-metrics-certs\") pod \"ai-gateway-controller-64466f8c8d-bg6dc\" (UID: \"242c7b88-7028-4b0f-a9e5-2e9fd16bc446\") " pod="opendatahub/ai-gateway-controller-64466f8c8d-bg6dc" Nov 04 04:57:03.380935 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:57:03.380888 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dv4vm\" (UniqueName: \"kubernetes.io/projected/242c7b88-7028-4b0f-a9e5-2e9fd16bc446-kube-api-access-dv4vm\") pod \"ai-gateway-controller-64466f8c8d-bg6dc\" (UID: \"242c7b88-7028-4b0f-a9e5-2e9fd16bc446\") " pod="opendatahub/ai-gateway-controller-64466f8c8d-bg6dc" Nov 04 04:57:03.385056 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:57:03.385038 4509 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\"" Nov 04 04:57:03.385156 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:57:03.385075 4509 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-g8rvz\"" Nov 04 04:57:03.386492 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:57:03.386477 4509 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\"" Nov 04 04:57:03.482093 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:57:03.482061 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/242c7b88-7028-4b0f-a9e5-2e9fd16bc446-metrics-certs\") pod \"ai-gateway-controller-64466f8c8d-bg6dc\" (UID: \"242c7b88-7028-4b0f-a9e5-2e9fd16bc446\") " pod="opendatahub/ai-gateway-controller-64466f8c8d-bg6dc" Nov 04 04:57:03.482265 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:57:03.482160 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dv4vm\" (UniqueName: \"kubernetes.io/projected/242c7b88-7028-4b0f-a9e5-2e9fd16bc446-kube-api-access-dv4vm\") pod \"ai-gateway-controller-64466f8c8d-bg6dc\" (UID: \"242c7b88-7028-4b0f-a9e5-2e9fd16bc446\") " pod="opendatahub/ai-gateway-controller-64466f8c8d-bg6dc" Nov 04 04:57:03.484550 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:57:03.484534 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/242c7b88-7028-4b0f-a9e5-2e9fd16bc446-metrics-certs\") pod \"ai-gateway-controller-64466f8c8d-bg6dc\" (UID: \"242c7b88-7028-4b0f-a9e5-2e9fd16bc446\") " pod="opendatahub/ai-gateway-controller-64466f8c8d-bg6dc" Nov 04 04:57:03.539773 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:57:03.539697 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dv4vm\" (UniqueName: \"kubernetes.io/projected/242c7b88-7028-4b0f-a9e5-2e9fd16bc446-kube-api-access-dv4vm\") pod \"ai-gateway-controller-64466f8c8d-bg6dc\" (UID: \"242c7b88-7028-4b0f-a9e5-2e9fd16bc446\") " pod="opendatahub/ai-gateway-controller-64466f8c8d-bg6dc" Nov 04 04:57:03.596481 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:57:03.596455 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-64466f8c8d-bg6dc" Nov 04 04:57:03.976984 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:57:03.976958 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-64466f8c8d-bg6dc"] Nov 04 04:57:03.983931 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:57:03.983902 4509 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod242c7b88_7028_4b0f_a9e5_2e9fd16bc446.slice/crio-270e81255326d35022788f57b9c4368e3e68f5c223d593d3dfb6103531cfb153 WatchSource:0}: Error finding container 270e81255326d35022788f57b9c4368e3e68f5c223d593d3dfb6103531cfb153: Status 404 returned error can't find the container with id 270e81255326d35022788f57b9c4368e3e68f5c223d593d3dfb6103531cfb153 Nov 04 04:57:04.564728 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:57:04.564688 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-64466f8c8d-bg6dc" event={"ID":"242c7b88-7028-4b0f-a9e5-2e9fd16bc446","Type":"ContainerStarted","Data":"270e81255326d35022788f57b9c4368e3e68f5c223d593d3dfb6103531cfb153"} Nov 04 04:57:07.587935 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:57:07.587901 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-64466f8c8d-bg6dc" event={"ID":"242c7b88-7028-4b0f-a9e5-2e9fd16bc446","Type":"ContainerStarted","Data":"5f3464170a8ae9cb5c07a43bd64a03820d0db6d461d04df57826bc6e5f840a9e"} Nov 04 04:57:07.588318 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:57:07.588036 4509 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-64466f8c8d-bg6dc" Nov 04 04:57:07.703582 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:57:07.703533 4509 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-64466f8c8d-bg6dc" podStartSLOduration=2.00354225 podStartE2EDuration="4.703519941s" totalImagesPullingTime="2.699977691s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 04:57:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:57:07.701921996 +0000 UTC m=+591.579645517" watchObservedRunningTime="2026-11-04 04:57:07.703519941 +0000 UTC m=+591.581243461" Nov 04 04:57:10.551531 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:57:10.551499 4509 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-n9sxt" Nov 04 04:57:14.454537 ip-10-0-133-180 kubenswrapper[4509]: E1104 04:57:14.454497 4509 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmdhl" containerName="extractor" Nov 04 04:57:18.594143 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:57:18.594115 4509 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-64466f8c8d-bg6dc" Nov 04 04:57:30.437399 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:57:30.437365 4509 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-d8rj2"] Nov 04 04:57:30.440831 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:57:30.440804 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-d8rj2" Nov 04 04:57:30.505688 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:57:30.505655 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a89237f-e122-4578-98bc-8a0658f3998d-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-d8rj2\" (UID: \"5a89237f-e122-4578-98bc-8a0658f3998d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-d8rj2" Nov 04 04:57:30.505800 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:57:30.505699 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a89237f-e122-4578-98bc-8a0658f3998d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-d8rj2\" (UID: \"5a89237f-e122-4578-98bc-8a0658f3998d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-d8rj2" Nov 04 04:57:30.505800 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:57:30.505732 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a89237f-e122-4578-98bc-8a0658f3998d-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-d8rj2\" (UID: \"5a89237f-e122-4578-98bc-8a0658f3998d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-d8rj2" Nov 04 04:57:30.505800 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:57:30.505791 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a89237f-e122-4578-98bc-8a0658f3998d-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-d8rj2\" (UID: \"5a89237f-e122-4578-98bc-8a0658f3998d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-d8rj2" Nov 04 04:57:30.505918 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:57:30.505841 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a89237f-e122-4578-98bc-8a0658f3998d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-d8rj2\" (UID: \"5a89237f-e122-4578-98bc-8a0658f3998d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-d8rj2" Nov 04 04:57:30.505953 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:57:30.505942 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a89237f-e122-4578-98bc-8a0658f3998d-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-d8rj2\" (UID: \"5a89237f-e122-4578-98bc-8a0658f3998d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-d8rj2" Nov 04 04:57:30.505997 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:57:30.505964 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a89237f-e122-4578-98bc-8a0658f3998d-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-d8rj2\" (UID: \"5a89237f-e122-4578-98bc-8a0658f3998d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-d8rj2" Nov 04 04:57:30.506058 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:57:30.506031 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpc79\" (UniqueName: \"kubernetes.io/projected/5a89237f-e122-4578-98bc-8a0658f3998d-kube-api-access-jpc79\") pod \"maas-default-gateway-openshift-default-6886b96f9-d8rj2\" (UID: \"5a89237f-e122-4578-98bc-8a0658f3998d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-d8rj2" Nov 04 04:57:30.506106 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:57:30.506086 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a89237f-e122-4578-98bc-8a0658f3998d-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-d8rj2\" (UID: \"5a89237f-e122-4578-98bc-8a0658f3998d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-d8rj2" Nov 04 04:57:30.607138 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:57:30.607111 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a89237f-e122-4578-98bc-8a0658f3998d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-d8rj2\" (UID: \"5a89237f-e122-4578-98bc-8a0658f3998d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-d8rj2" Nov 04 04:57:30.607268 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:57:30.607150 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a89237f-e122-4578-98bc-8a0658f3998d-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-d8rj2\" (UID: \"5a89237f-e122-4578-98bc-8a0658f3998d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-d8rj2" Nov 04 04:57:30.607268 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:57:30.607174 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a89237f-e122-4578-98bc-8a0658f3998d-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-d8rj2\" (UID: \"5a89237f-e122-4578-98bc-8a0658f3998d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-d8rj2" Nov 04 04:57:30.607268 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:57:30.607196 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a89237f-e122-4578-98bc-8a0658f3998d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-d8rj2\" (UID: \"5a89237f-e122-4578-98bc-8a0658f3998d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-d8rj2" Nov 04 04:57:30.607268 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:57:30.607262 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a89237f-e122-4578-98bc-8a0658f3998d-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-d8rj2\" (UID: \"5a89237f-e122-4578-98bc-8a0658f3998d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-d8rj2" Nov 04 04:57:30.607498 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:57:30.607400 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a89237f-e122-4578-98bc-8a0658f3998d-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-d8rj2\" (UID: \"5a89237f-e122-4578-98bc-8a0658f3998d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-d8rj2" Nov 04 04:57:30.607498 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:57:30.607435 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpc79\" (UniqueName: \"kubernetes.io/projected/5a89237f-e122-4578-98bc-8a0658f3998d-kube-api-access-jpc79\") pod \"maas-default-gateway-openshift-default-6886b96f9-d8rj2\" (UID: \"5a89237f-e122-4578-98bc-8a0658f3998d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-d8rj2" Nov 04 04:57:30.607498 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:57:30.607490 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a89237f-e122-4578-98bc-8a0658f3998d-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-d8rj2\" (UID: \"5a89237f-e122-4578-98bc-8a0658f3998d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-d8rj2" Nov 04 04:57:30.607652 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:57:30.607568 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a89237f-e122-4578-98bc-8a0658f3998d-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-d8rj2\" (UID: \"5a89237f-e122-4578-98bc-8a0658f3998d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-d8rj2" Nov 04 04:57:30.607652 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:57:30.607575 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a89237f-e122-4578-98bc-8a0658f3998d-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-d8rj2\" (UID: \"5a89237f-e122-4578-98bc-8a0658f3998d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-d8rj2" Nov 04 04:57:30.607652 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:57:30.607615 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a89237f-e122-4578-98bc-8a0658f3998d-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-d8rj2\" (UID: \"5a89237f-e122-4578-98bc-8a0658f3998d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-d8rj2" Nov 04 04:57:30.607917 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:57:30.607897 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a89237f-e122-4578-98bc-8a0658f3998d-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-d8rj2\" (UID: \"5a89237f-e122-4578-98bc-8a0658f3998d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-d8rj2" Nov 04 04:57:30.608108 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:57:30.607988 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a89237f-e122-4578-98bc-8a0658f3998d-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-d8rj2\" (UID: \"5a89237f-e122-4578-98bc-8a0658f3998d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-d8rj2" Nov 04 04:57:30.608108 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:57:30.608061 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a89237f-e122-4578-98bc-8a0658f3998d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-d8rj2\" (UID: \"5a89237f-e122-4578-98bc-8a0658f3998d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-d8rj2" Nov 04 04:57:30.609610 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:57:30.609591 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a89237f-e122-4578-98bc-8a0658f3998d-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-d8rj2\" (UID: \"5a89237f-e122-4578-98bc-8a0658f3998d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-d8rj2" Nov 04 04:57:30.609696 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:57:30.609660 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a89237f-e122-4578-98bc-8a0658f3998d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-d8rj2\" (UID: \"5a89237f-e122-4578-98bc-8a0658f3998d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-d8rj2" Nov 04 04:57:30.644548 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:57:30.644528 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-d8rj2"] Nov 04 04:57:30.664162 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:57:30.664129 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a89237f-e122-4578-98bc-8a0658f3998d-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-d8rj2\" (UID: \"5a89237f-e122-4578-98bc-8a0658f3998d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-d8rj2" Nov 04 04:57:30.664270 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:57:30.664175 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpc79\" (UniqueName: \"kubernetes.io/projected/5a89237f-e122-4578-98bc-8a0658f3998d-kube-api-access-jpc79\") pod \"maas-default-gateway-openshift-default-6886b96f9-d8rj2\" (UID: \"5a89237f-e122-4578-98bc-8a0658f3998d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-d8rj2" Nov 04 04:57:30.752423 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:57:30.752403 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-d8rj2" Nov 04 04:57:31.228231 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:57:31.228204 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-d8rj2"] Nov 04 04:57:31.230215 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:57:31.230186 4509 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a89237f_e122_4578_98bc_8a0658f3998d.slice/crio-e760ab8c22a9e31a471c87c86da89d1370462695406b530f8e4dcb7f18b5fd7a WatchSource:0}: Error finding container e760ab8c22a9e31a471c87c86da89d1370462695406b530f8e4dcb7f18b5fd7a: Status 404 returned error can't find the container with id e760ab8c22a9e31a471c87c86da89d1370462695406b530f8e4dcb7f18b5fd7a Nov 04 04:57:31.232179 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:57:31.232153 4509 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 04 04:57:31.232230 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:57:31.232213 4509 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 04 04:57:31.232266 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:57:31.232241 4509 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 04 04:57:31.688667 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:57:31.688569 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-d8rj2" event={"ID":"5a89237f-e122-4578-98bc-8a0658f3998d","Type":"ContainerStarted","Data":"720158b6fbc8846f05a48c54ae0ef358db2d53f40c79e97a3e1b3f708fa35935"} Nov 04 04:57:31.688667 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:57:31.688636 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-d8rj2" event={"ID":"5a89237f-e122-4578-98bc-8a0658f3998d","Type":"ContainerStarted","Data":"e760ab8c22a9e31a471c87c86da89d1370462695406b530f8e4dcb7f18b5fd7a"} Nov 04 04:57:31.753170 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:57:31.753130 4509 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-d8rj2" Nov 04 04:57:31.806367 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:57:31.806295 4509 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-d8rj2" podStartSLOduration=1.8062785190000001 podStartE2EDuration="1.806278519s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 04:57:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:57:31.804680386 +0000 UTC m=+615.682403908" watchObservedRunningTime="2026-11-04 04:57:31.806278519 +0000 UTC m=+615.684002038" Nov 04 04:57:32.754385 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:57:32.754353 4509 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-d8rj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Nov 04 04:57:32.754790 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:57:32.754405 4509 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-d8rj2" podUID="5a89237f-e122-4578-98bc-8a0658f3998d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Nov 04 04:57:33.753272 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:57:33.753239 4509 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-d8rj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Nov 04 04:57:33.753467 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:57:33.753291 4509 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-d8rj2" podUID="5a89237f-e122-4578-98bc-8a0658f3998d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Nov 04 04:57:34.753351 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:57:34.753292 4509 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-d8rj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Nov 04 04:57:34.753756 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:57:34.753374 4509 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-d8rj2" podUID="5a89237f-e122-4578-98bc-8a0658f3998d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Nov 04 04:57:35.753512 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:57:35.753483 4509 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-d8rj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Nov 04 04:57:35.753885 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:57:35.753532 4509 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-d8rj2" podUID="5a89237f-e122-4578-98bc-8a0658f3998d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Nov 04 04:57:36.753787 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:57:36.753754 4509 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-d8rj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Nov 04 04:57:36.754152 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:57:36.753808 4509 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-d8rj2" podUID="5a89237f-e122-4578-98bc-8a0658f3998d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Nov 04 04:57:37.753536 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:57:37.753497 4509 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-d8rj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Nov 04 04:57:37.753726 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:57:37.753568 4509 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-d8rj2" podUID="5a89237f-e122-4578-98bc-8a0658f3998d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Nov 04 04:57:38.753253 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:57:38.753216 4509 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-d8rj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Nov 04 04:57:38.753673 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:57:38.753277 4509 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-d8rj2" podUID="5a89237f-e122-4578-98bc-8a0658f3998d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Nov 04 04:57:39.756987 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:57:39.756961 4509 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-d8rj2" Nov 04 04:57:39.757362 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:57:39.757269 4509 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-d8rj2" Nov 04 04:57:39.758147 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:57:39.758124 4509 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-d8rj2" Nov 04 04:57:44.452629 ip-10-0-133-180 kubenswrapper[4509]: E1104 04:57:44.452598 4509 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmdhl" containerName="extractor" Nov 04 04:58:14.453289 ip-10-0-133-180 kubenswrapper[4509]: E1104 04:58:14.453252 4509 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmdhl" containerName="extractor" Nov 04 04:58:18.074379 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:18.074346 4509 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-znf89"] Nov 04 04:58:18.078072 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:18.078055 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-znf89" Nov 04 04:58:18.128123 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:18.128092 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67ec6f66-3a36-4d01-be83-4ea5d42ba52e-workload-certs\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-znf89\" (UID: \"67ec6f66-3a36-4d01-be83-4ea5d42ba52e\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-znf89" Nov 04 04:58:18.128235 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:18.128130 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67ec6f66-3a36-4d01-be83-4ea5d42ba52e-istio-podinfo\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-znf89\" (UID: \"67ec6f66-3a36-4d01-be83-4ea5d42ba52e\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-znf89" Nov 04 04:58:18.128235 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:18.128151 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67ec6f66-3a36-4d01-be83-4ea5d42ba52e-istio-token\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-znf89\" (UID: \"67ec6f66-3a36-4d01-be83-4ea5d42ba52e\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-znf89" Nov 04 04:58:18.128235 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:18.128207 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67ec6f66-3a36-4d01-be83-4ea5d42ba52e-workload-socket\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-znf89\" (UID: \"67ec6f66-3a36-4d01-be83-4ea5d42ba52e\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-znf89" Nov 04 04:58:18.128372 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:18.128306 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67ec6f66-3a36-4d01-be83-4ea5d42ba52e-istio-data\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-znf89\" (UID: \"67ec6f66-3a36-4d01-be83-4ea5d42ba52e\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-znf89" Nov 04 04:58:18.128372 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:18.128339 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gm95c\" (UniqueName: \"kubernetes.io/projected/67ec6f66-3a36-4d01-be83-4ea5d42ba52e-kube-api-access-gm95c\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-znf89\" (UID: \"67ec6f66-3a36-4d01-be83-4ea5d42ba52e\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-znf89" Nov 04 04:58:18.128508 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:18.128486 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67ec6f66-3a36-4d01-be83-4ea5d42ba52e-istio-envoy\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-znf89\" (UID: \"67ec6f66-3a36-4d01-be83-4ea5d42ba52e\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-znf89" Nov 04 04:58:18.128569 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:18.128532 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67ec6f66-3a36-4d01-be83-4ea5d42ba52e-credential-socket\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-znf89\" (UID: \"67ec6f66-3a36-4d01-be83-4ea5d42ba52e\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-znf89" Nov 04 04:58:18.128641 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:18.128625 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67ec6f66-3a36-4d01-be83-4ea5d42ba52e-istiod-ca-cert\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-znf89\" (UID: \"67ec6f66-3a36-4d01-be83-4ea5d42ba52e\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-znf89" Nov 04 04:58:18.130544 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:18.130527 4509 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-4cce33-openshift-default-dockercfg-qrlrc\"" Nov 04 04:58:18.182879 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:18.182852 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-znf89"] Nov 04 04:58:18.231659 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:18.231613 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67ec6f66-3a36-4d01-be83-4ea5d42ba52e-istio-data\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-znf89\" (UID: \"67ec6f66-3a36-4d01-be83-4ea5d42ba52e\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-znf89" Nov 04 04:58:18.231659 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:18.231665 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gm95c\" (UniqueName: \"kubernetes.io/projected/67ec6f66-3a36-4d01-be83-4ea5d42ba52e-kube-api-access-gm95c\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-znf89\" (UID: \"67ec6f66-3a36-4d01-be83-4ea5d42ba52e\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-znf89" Nov 04 04:58:18.231916 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:18.231709 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67ec6f66-3a36-4d01-be83-4ea5d42ba52e-istio-envoy\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-znf89\" (UID: \"67ec6f66-3a36-4d01-be83-4ea5d42ba52e\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-znf89" Nov 04 04:58:18.231916 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:18.231740 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67ec6f66-3a36-4d01-be83-4ea5d42ba52e-credential-socket\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-znf89\" (UID: \"67ec6f66-3a36-4d01-be83-4ea5d42ba52e\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-znf89" Nov 04 04:58:18.231916 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:18.231801 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67ec6f66-3a36-4d01-be83-4ea5d42ba52e-istiod-ca-cert\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-znf89\" (UID: \"67ec6f66-3a36-4d01-be83-4ea5d42ba52e\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-znf89" Nov 04 04:58:18.231916 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:18.231850 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67ec6f66-3a36-4d01-be83-4ea5d42ba52e-workload-certs\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-znf89\" (UID: \"67ec6f66-3a36-4d01-be83-4ea5d42ba52e\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-znf89" Nov 04 04:58:18.231916 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:18.231878 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67ec6f66-3a36-4d01-be83-4ea5d42ba52e-istio-podinfo\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-znf89\" (UID: \"67ec6f66-3a36-4d01-be83-4ea5d42ba52e\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-znf89" Nov 04 04:58:18.231916 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:18.231913 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67ec6f66-3a36-4d01-be83-4ea5d42ba52e-istio-token\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-znf89\" (UID: \"67ec6f66-3a36-4d01-be83-4ea5d42ba52e\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-znf89" Nov 04 04:58:18.232214 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:18.231946 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67ec6f66-3a36-4d01-be83-4ea5d42ba52e-workload-socket\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-znf89\" (UID: \"67ec6f66-3a36-4d01-be83-4ea5d42ba52e\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-znf89" Nov 04 04:58:18.232506 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:18.232478 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67ec6f66-3a36-4d01-be83-4ea5d42ba52e-workload-socket\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-znf89\" (UID: \"67ec6f66-3a36-4d01-be83-4ea5d42ba52e\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-znf89" Nov 04 04:58:18.232874 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:18.232850 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67ec6f66-3a36-4d01-be83-4ea5d42ba52e-workload-certs\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-znf89\" (UID: \"67ec6f66-3a36-4d01-be83-4ea5d42ba52e\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-znf89" Nov 04 04:58:18.233187 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:18.233159 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67ec6f66-3a36-4d01-be83-4ea5d42ba52e-istio-data\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-znf89\" (UID: \"67ec6f66-3a36-4d01-be83-4ea5d42ba52e\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-znf89" Nov 04 04:58:18.233370 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:18.233345 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67ec6f66-3a36-4d01-be83-4ea5d42ba52e-credential-socket\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-znf89\" (UID: \"67ec6f66-3a36-4d01-be83-4ea5d42ba52e\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-znf89" Nov 04 04:58:18.233908 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:18.233884 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67ec6f66-3a36-4d01-be83-4ea5d42ba52e-istiod-ca-cert\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-znf89\" (UID: \"67ec6f66-3a36-4d01-be83-4ea5d42ba52e\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-znf89" Nov 04 04:58:18.235773 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:18.235742 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67ec6f66-3a36-4d01-be83-4ea5d42ba52e-istio-envoy\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-znf89\" (UID: \"67ec6f66-3a36-4d01-be83-4ea5d42ba52e\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-znf89" Nov 04 04:58:18.235914 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:18.235892 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67ec6f66-3a36-4d01-be83-4ea5d42ba52e-istio-podinfo\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-znf89\" (UID: \"67ec6f66-3a36-4d01-be83-4ea5d42ba52e\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-znf89" Nov 04 04:58:18.292071 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:18.292039 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gm95c\" (UniqueName: \"kubernetes.io/projected/67ec6f66-3a36-4d01-be83-4ea5d42ba52e-kube-api-access-gm95c\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-znf89\" (UID: \"67ec6f66-3a36-4d01-be83-4ea5d42ba52e\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-znf89" Nov 04 04:58:18.292801 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:18.292774 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67ec6f66-3a36-4d01-be83-4ea5d42ba52e-istio-token\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-znf89\" (UID: \"67ec6f66-3a36-4d01-be83-4ea5d42ba52e\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-znf89" Nov 04 04:58:18.391535 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:18.391476 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-znf89" Nov 04 04:58:18.713721 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:18.713695 4509 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-m6qb5"] Nov 04 04:58:18.717528 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:18.717508 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-m6qb5" Nov 04 04:58:18.770273 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:18.770238 4509 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-cedfca-openshift-default-dockercfg-fdtt2\"" Nov 04 04:58:18.779991 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:18.779955 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-m6qb5"] Nov 04 04:58:18.818600 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:18.818574 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-znf89"] Nov 04 04:58:18.831807 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:58:18.831786 4509 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67ec6f66_3a36_4d01_be83_4ea5d42ba52e.slice/crio-ce995f29b532a90db65521bc5f44e3071c0c84984adbb40ffdab86f87ef089c0 WatchSource:0}: Error finding container ce995f29b532a90db65521bc5f44e3071c0c84984adbb40ffdab86f87ef089c0: Status 404 returned error can't find the container with id ce995f29b532a90db65521bc5f44e3071c0c84984adbb40ffdab86f87ef089c0 Nov 04 04:58:18.834209 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:18.834178 4509 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 04 04:58:18.834297 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:18.834243 4509 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 04 04:58:18.834297 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:18.834271 4509 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 04 04:58:18.846255 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:18.846234 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d751b557-4e07-4657-96ec-59fc0e70fd57-istio-podinfo\") pod \"e2e-worker-w6-cedfca-openshift-default-6b99d7696f-m6qb5\" (UID: \"d751b557-4e07-4657-96ec-59fc0e70fd57\") " pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-m6qb5" Nov 04 04:58:18.846313 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:18.846268 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d751b557-4e07-4657-96ec-59fc0e70fd57-workload-socket\") pod \"e2e-worker-w6-cedfca-openshift-default-6b99d7696f-m6qb5\" (UID: \"d751b557-4e07-4657-96ec-59fc0e70fd57\") " pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-m6qb5" Nov 04 04:58:18.846313 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:18.846286 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d751b557-4e07-4657-96ec-59fc0e70fd57-istio-envoy\") pod \"e2e-worker-w6-cedfca-openshift-default-6b99d7696f-m6qb5\" (UID: \"d751b557-4e07-4657-96ec-59fc0e70fd57\") " pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-m6qb5" Nov 04 04:58:18.846409 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:18.846394 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d751b557-4e07-4657-96ec-59fc0e70fd57-workload-certs\") pod \"e2e-worker-w6-cedfca-openshift-default-6b99d7696f-m6qb5\" (UID: \"d751b557-4e07-4657-96ec-59fc0e70fd57\") " pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-m6qb5" Nov 04 04:58:18.846503 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:18.846488 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d751b557-4e07-4657-96ec-59fc0e70fd57-istio-data\") pod \"e2e-worker-w6-cedfca-openshift-default-6b99d7696f-m6qb5\" (UID: \"d751b557-4e07-4657-96ec-59fc0e70fd57\") " pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-m6qb5" Nov 04 04:58:18.846542 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:18.846529 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d751b557-4e07-4657-96ec-59fc0e70fd57-credential-socket\") pod \"e2e-worker-w6-cedfca-openshift-default-6b99d7696f-m6qb5\" (UID: \"d751b557-4e07-4657-96ec-59fc0e70fd57\") " pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-m6qb5" Nov 04 04:58:18.846575 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:18.846568 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d751b557-4e07-4657-96ec-59fc0e70fd57-istiod-ca-cert\") pod \"e2e-worker-w6-cedfca-openshift-default-6b99d7696f-m6qb5\" (UID: \"d751b557-4e07-4657-96ec-59fc0e70fd57\") " pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-m6qb5" Nov 04 04:58:18.846608 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:18.846589 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d751b557-4e07-4657-96ec-59fc0e70fd57-istio-token\") pod \"e2e-worker-w6-cedfca-openshift-default-6b99d7696f-m6qb5\" (UID: \"d751b557-4e07-4657-96ec-59fc0e70fd57\") " pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-m6qb5" Nov 04 04:58:18.846692 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:18.846679 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5b5qb\" (UniqueName: \"kubernetes.io/projected/d751b557-4e07-4657-96ec-59fc0e70fd57-kube-api-access-5b5qb\") pod \"e2e-worker-w6-cedfca-openshift-default-6b99d7696f-m6qb5\" (UID: \"d751b557-4e07-4657-96ec-59fc0e70fd57\") " pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-m6qb5" Nov 04 04:58:18.889775 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:18.889751 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-znf89" event={"ID":"67ec6f66-3a36-4d01-be83-4ea5d42ba52e","Type":"ContainerStarted","Data":"ce995f29b532a90db65521bc5f44e3071c0c84984adbb40ffdab86f87ef089c0"} Nov 04 04:58:18.951299 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:18.949143 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d751b557-4e07-4657-96ec-59fc0e70fd57-istiod-ca-cert\") pod \"e2e-worker-w6-cedfca-openshift-default-6b99d7696f-m6qb5\" (UID: \"d751b557-4e07-4657-96ec-59fc0e70fd57\") " pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-m6qb5" Nov 04 04:58:18.951299 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:18.949202 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d751b557-4e07-4657-96ec-59fc0e70fd57-istio-token\") pod \"e2e-worker-w6-cedfca-openshift-default-6b99d7696f-m6qb5\" (UID: \"d751b557-4e07-4657-96ec-59fc0e70fd57\") " pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-m6qb5" Nov 04 04:58:18.951299 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:18.949257 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5b5qb\" (UniqueName: \"kubernetes.io/projected/d751b557-4e07-4657-96ec-59fc0e70fd57-kube-api-access-5b5qb\") pod \"e2e-worker-w6-cedfca-openshift-default-6b99d7696f-m6qb5\" (UID: \"d751b557-4e07-4657-96ec-59fc0e70fd57\") " pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-m6qb5" Nov 04 04:58:18.951299 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:18.949301 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d751b557-4e07-4657-96ec-59fc0e70fd57-istio-podinfo\") pod \"e2e-worker-w6-cedfca-openshift-default-6b99d7696f-m6qb5\" (UID: \"d751b557-4e07-4657-96ec-59fc0e70fd57\") " pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-m6qb5" Nov 04 04:58:18.951299 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:18.949338 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d751b557-4e07-4657-96ec-59fc0e70fd57-workload-socket\") pod \"e2e-worker-w6-cedfca-openshift-default-6b99d7696f-m6qb5\" (UID: \"d751b557-4e07-4657-96ec-59fc0e70fd57\") " pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-m6qb5" Nov 04 04:58:18.951299 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:18.949368 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d751b557-4e07-4657-96ec-59fc0e70fd57-istio-envoy\") pod \"e2e-worker-w6-cedfca-openshift-default-6b99d7696f-m6qb5\" (UID: \"d751b557-4e07-4657-96ec-59fc0e70fd57\") " pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-m6qb5" Nov 04 04:58:18.951299 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:18.949414 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d751b557-4e07-4657-96ec-59fc0e70fd57-workload-certs\") pod \"e2e-worker-w6-cedfca-openshift-default-6b99d7696f-m6qb5\" (UID: \"d751b557-4e07-4657-96ec-59fc0e70fd57\") " pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-m6qb5" Nov 04 04:58:18.951299 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:18.949459 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d751b557-4e07-4657-96ec-59fc0e70fd57-istio-data\") pod \"e2e-worker-w6-cedfca-openshift-default-6b99d7696f-m6qb5\" (UID: \"d751b557-4e07-4657-96ec-59fc0e70fd57\") " pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-m6qb5" Nov 04 04:58:18.951299 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:18.949493 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d751b557-4e07-4657-96ec-59fc0e70fd57-credential-socket\") pod \"e2e-worker-w6-cedfca-openshift-default-6b99d7696f-m6qb5\" (UID: \"d751b557-4e07-4657-96ec-59fc0e70fd57\") " pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-m6qb5" Nov 04 04:58:18.951299 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:18.950304 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d751b557-4e07-4657-96ec-59fc0e70fd57-workload-certs\") pod \"e2e-worker-w6-cedfca-openshift-default-6b99d7696f-m6qb5\" (UID: \"d751b557-4e07-4657-96ec-59fc0e70fd57\") " pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-m6qb5" Nov 04 04:58:18.951299 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:18.950638 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d751b557-4e07-4657-96ec-59fc0e70fd57-workload-socket\") pod \"e2e-worker-w6-cedfca-openshift-default-6b99d7696f-m6qb5\" (UID: \"d751b557-4e07-4657-96ec-59fc0e70fd57\") " pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-m6qb5" Nov 04 04:58:18.951299 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:18.950936 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d751b557-4e07-4657-96ec-59fc0e70fd57-istio-data\") pod \"e2e-worker-w6-cedfca-openshift-default-6b99d7696f-m6qb5\" (UID: \"d751b557-4e07-4657-96ec-59fc0e70fd57\") " pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-m6qb5" Nov 04 04:58:18.951299 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:18.951241 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d751b557-4e07-4657-96ec-59fc0e70fd57-istiod-ca-cert\") pod \"e2e-worker-w6-cedfca-openshift-default-6b99d7696f-m6qb5\" (UID: \"d751b557-4e07-4657-96ec-59fc0e70fd57\") " pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-m6qb5" Nov 04 04:58:18.954289 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:18.954263 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d751b557-4e07-4657-96ec-59fc0e70fd57-istio-podinfo\") pod \"e2e-worker-w6-cedfca-openshift-default-6b99d7696f-m6qb5\" (UID: \"d751b557-4e07-4657-96ec-59fc0e70fd57\") " pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-m6qb5" Nov 04 04:58:18.954485 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:18.954316 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d751b557-4e07-4657-96ec-59fc0e70fd57-credential-socket\") pod \"e2e-worker-w6-cedfca-openshift-default-6b99d7696f-m6qb5\" (UID: \"d751b557-4e07-4657-96ec-59fc0e70fd57\") " pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-m6qb5" Nov 04 04:58:18.955571 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:18.955542 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d751b557-4e07-4657-96ec-59fc0e70fd57-istio-envoy\") pod \"e2e-worker-w6-cedfca-openshift-default-6b99d7696f-m6qb5\" (UID: \"d751b557-4e07-4657-96ec-59fc0e70fd57\") " pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-m6qb5" Nov 04 04:58:19.015887 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:19.015816 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d751b557-4e07-4657-96ec-59fc0e70fd57-istio-token\") pod \"e2e-worker-w6-cedfca-openshift-default-6b99d7696f-m6qb5\" (UID: \"d751b557-4e07-4657-96ec-59fc0e70fd57\") " pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-m6qb5" Nov 04 04:58:19.016010 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:19.015986 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5b5qb\" (UniqueName: \"kubernetes.io/projected/d751b557-4e07-4657-96ec-59fc0e70fd57-kube-api-access-5b5qb\") pod \"e2e-worker-w6-cedfca-openshift-default-6b99d7696f-m6qb5\" (UID: \"d751b557-4e07-4657-96ec-59fc0e70fd57\") " pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-m6qb5" Nov 04 04:58:19.035845 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:19.035817 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-m6qb5" Nov 04 04:58:19.467534 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:19.467514 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-m6qb5"] Nov 04 04:58:19.469882 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:58:19.469856 4509 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd751b557_4e07_4657_96ec_59fc0e70fd57.slice/crio-1012908b2358a73c7b071a9072d9df084e9e7de2a8691f5cf931dc031bd15128 WatchSource:0}: Error finding container 1012908b2358a73c7b071a9072d9df084e9e7de2a8691f5cf931dc031bd15128: Status 404 returned error can't find the container with id 1012908b2358a73c7b071a9072d9df084e9e7de2a8691f5cf931dc031bd15128 Nov 04 04:58:19.472314 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:19.472284 4509 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 04 04:58:19.472411 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:19.472381 4509 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 04 04:58:19.472463 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:19.472412 4509 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 04 04:58:19.895817 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:19.895730 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-znf89" event={"ID":"67ec6f66-3a36-4d01-be83-4ea5d42ba52e","Type":"ContainerStarted","Data":"38fd3e3b00aad944d8c094dcef7e8eb6d1d7b4f72f2be1288d1c49e77bcd6097"} Nov 04 04:58:19.897214 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:19.897189 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-m6qb5" event={"ID":"d751b557-4e07-4657-96ec-59fc0e70fd57","Type":"ContainerStarted","Data":"90901cf35b3a5c5999f8076892b42910a049c313e90fe487980febe641e77b6b"} Nov 04 04:58:19.897357 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:19.897228 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-m6qb5" event={"ID":"d751b557-4e07-4657-96ec-59fc0e70fd57","Type":"ContainerStarted","Data":"1012908b2358a73c7b071a9072d9df084e9e7de2a8691f5cf931dc031bd15128"} Nov 04 04:58:20.013053 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:20.013009 4509 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-znf89" podStartSLOduration=2.012996324 podStartE2EDuration="2.012996324s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 04:58:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:58:20.011407882 +0000 UTC m=+663.889131427" watchObservedRunningTime="2026-11-04 04:58:20.012996324 +0000 UTC m=+663.890719844" Nov 04 04:58:20.036658 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:20.036632 4509 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-m6qb5" Nov 04 04:58:20.041258 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:20.041238 4509 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-m6qb5" Nov 04 04:58:20.144279 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:20.141851 4509 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-m6qb5" podStartSLOduration=2.141831713 podStartE2EDuration="2.141831713s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 04:58:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:58:20.137359971 +0000 UTC m=+664.015083496" watchObservedRunningTime="2026-11-04 04:58:20.141831713 +0000 UTC m=+664.019555234" Nov 04 04:58:20.391981 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:20.391954 4509 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-znf89" Nov 04 04:58:20.396481 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:20.396458 4509 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-znf89" Nov 04 04:58:20.903024 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:20.902993 4509 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-znf89" Nov 04 04:58:20.903442 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:20.903045 4509 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-m6qb5" Nov 04 04:58:20.903732 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:20.903713 4509 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-znf89" Nov 04 04:58:20.903834 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:20.903770 4509 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-m6qb5" Nov 04 04:58:26.982838 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:26.982801 4509 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-4cce33-56666d7678-r88jh"] Nov 04 04:58:26.986335 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:26.986308 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-4cce33-56666d7678-r88jh" Nov 04 04:58:27.042932 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:27.042910 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-4cce33-56666d7678-r88jh"] Nov 04 04:58:27.090456 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:27.090432 4509 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-4cce33\"" Nov 04 04:58:27.090575 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:27.090511 4509 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-4cce33-dockercfg-zgtwj\"" Nov 04 04:58:27.135615 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:27.135590 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a16804b-60cc-46b5-9879-163bda7bd069-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-4cce33-56666d7678-r88jh\" (UID: \"2a16804b-60cc-46b5-9879-163bda7bd069\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-4cce33-56666d7678-r88jh" Nov 04 04:58:27.135709 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:27.135626 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmpj2\" (UniqueName: \"kubernetes.io/projected/2a16804b-60cc-46b5-9879-163bda7bd069-kube-api-access-xmpj2\") pod \"payload-processing-e2e-worker-w5-4cce33-56666d7678-r88jh\" (UID: \"2a16804b-60cc-46b5-9879-163bda7bd069\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-4cce33-56666d7678-r88jh" Nov 04 04:58:27.237071 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:27.236998 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a16804b-60cc-46b5-9879-163bda7bd069-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-4cce33-56666d7678-r88jh\" (UID: \"2a16804b-60cc-46b5-9879-163bda7bd069\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-4cce33-56666d7678-r88jh" Nov 04 04:58:27.237071 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:27.237038 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmpj2\" (UniqueName: \"kubernetes.io/projected/2a16804b-60cc-46b5-9879-163bda7bd069-kube-api-access-xmpj2\") pod \"payload-processing-e2e-worker-w5-4cce33-56666d7678-r88jh\" (UID: \"2a16804b-60cc-46b5-9879-163bda7bd069\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-4cce33-56666d7678-r88jh" Nov 04 04:58:27.237745 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:27.237726 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a16804b-60cc-46b5-9879-163bda7bd069-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-4cce33-56666d7678-r88jh\" (UID: \"2a16804b-60cc-46b5-9879-163bda7bd069\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-4cce33-56666d7678-r88jh" Nov 04 04:58:27.296278 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:27.296251 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmpj2\" (UniqueName: \"kubernetes.io/projected/2a16804b-60cc-46b5-9879-163bda7bd069-kube-api-access-xmpj2\") pod \"payload-processing-e2e-worker-w5-4cce33-56666d7678-r88jh\" (UID: \"2a16804b-60cc-46b5-9879-163bda7bd069\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-4cce33-56666d7678-r88jh" Nov 04 04:58:27.300105 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:27.300081 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-4cce33-56666d7678-r88jh" Nov 04 04:58:27.671907 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:27.671879 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-4cce33-56666d7678-r88jh"] Nov 04 04:58:27.674344 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:58:27.674296 4509 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a16804b_60cc_46b5_9879_163bda7bd069.slice/crio-745892b1d1b7f5b91bc3577b66fd5916eb975246a1503ac41bbda14f67c2c717 WatchSource:0}: Error finding container 745892b1d1b7f5b91bc3577b66fd5916eb975246a1503ac41bbda14f67c2c717: Status 404 returned error can't find the container with id 745892b1d1b7f5b91bc3577b66fd5916eb975246a1503ac41bbda14f67c2c717 Nov 04 04:58:27.955313 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:27.955210 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-4cce33-56666d7678-r88jh" event={"ID":"2a16804b-60cc-46b5-9879-163bda7bd069","Type":"ContainerStarted","Data":"9f5a4844648e5c09869687b1f548d63b756b2a51a6ac37f646cf789559c04cbb"} Nov 04 04:58:27.955641 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:27.955622 4509 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-4cce33-56666d7678-r88jh" Nov 04 04:58:27.955807 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:27.955782 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-4cce33-56666d7678-r88jh" event={"ID":"2a16804b-60cc-46b5-9879-163bda7bd069","Type":"ContainerStarted","Data":"745892b1d1b7f5b91bc3577b66fd5916eb975246a1503ac41bbda14f67c2c717"} Nov 04 04:58:28.070481 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:28.070437 4509 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w5-4cce33-56666d7678-r88jh" podStartSLOduration=2.070426157 podStartE2EDuration="2.070426157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 04:58:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:58:28.069185668 +0000 UTC m=+671.946909371" watchObservedRunningTime="2026-11-04 04:58:28.070426157 +0000 UTC m=+671.948149677" Nov 04 04:58:31.569390 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:31.569349 4509 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-3eb31c-5bc57d7f87-xfpgv"] Nov 04 04:58:31.572898 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:31.572877 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3eb31c-5bc57d7f87-xfpgv" Nov 04 04:58:31.624985 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:31.624966 4509 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-3eb31c\"" Nov 04 04:58:31.674909 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:31.674879 4509 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-3eb31c-dockercfg-m6f5m\"" Nov 04 04:58:31.676874 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:31.676837 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-3eb31c-5bc57d7f87-xfpgv"] Nov 04 04:58:31.685127 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:31.685044 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c425a743-509e-4356-86a2-8a1779db79ae-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-3eb31c-5bc57d7f87-xfpgv\" (UID: \"c425a743-509e-4356-86a2-8a1779db79ae\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3eb31c-5bc57d7f87-xfpgv" Nov 04 04:58:31.685284 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:31.685257 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkfgk\" (UniqueName: \"kubernetes.io/projected/c425a743-509e-4356-86a2-8a1779db79ae-kube-api-access-lkfgk\") pod \"payload-pre-processing-e2e-worker-w4-3eb31c-5bc57d7f87-xfpgv\" (UID: \"c425a743-509e-4356-86a2-8a1779db79ae\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3eb31c-5bc57d7f87-xfpgv" Nov 04 04:58:31.726684 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:31.726653 4509 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-3eb31c-589ff8b4b8-v89hw"] Nov 04 04:58:31.731627 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:31.731598 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-3eb31c-589ff8b4b8-v89hw" Nov 04 04:58:31.785831 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:31.785809 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c425a743-509e-4356-86a2-8a1779db79ae-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-3eb31c-5bc57d7f87-xfpgv\" (UID: \"c425a743-509e-4356-86a2-8a1779db79ae\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3eb31c-5bc57d7f87-xfpgv" Nov 04 04:58:31.785956 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:31.785905 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkfgk\" (UniqueName: \"kubernetes.io/projected/c425a743-509e-4356-86a2-8a1779db79ae-kube-api-access-lkfgk\") pod \"payload-pre-processing-e2e-worker-w4-3eb31c-5bc57d7f87-xfpgv\" (UID: \"c425a743-509e-4356-86a2-8a1779db79ae\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3eb31c-5bc57d7f87-xfpgv" Nov 04 04:58:31.786487 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:31.786469 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c425a743-509e-4356-86a2-8a1779db79ae-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-3eb31c-5bc57d7f87-xfpgv\" (UID: \"c425a743-509e-4356-86a2-8a1779db79ae\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3eb31c-5bc57d7f87-xfpgv" Nov 04 04:58:31.798126 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:31.798092 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-3eb31c-589ff8b4b8-v89hw"] Nov 04 04:58:31.847185 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:31.847137 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkfgk\" (UniqueName: \"kubernetes.io/projected/c425a743-509e-4356-86a2-8a1779db79ae-kube-api-access-lkfgk\") pod \"payload-pre-processing-e2e-worker-w4-3eb31c-5bc57d7f87-xfpgv\" (UID: \"c425a743-509e-4356-86a2-8a1779db79ae\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3eb31c-5bc57d7f87-xfpgv" Nov 04 04:58:31.885081 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:31.885058 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3eb31c-5bc57d7f87-xfpgv" Nov 04 04:58:31.886335 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:31.886306 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfv9t\" (UniqueName: \"kubernetes.io/projected/1df0a5e9-fb8c-49d7-a5ba-9a04e99b6c09-kube-api-access-xfv9t\") pod \"payload-processing-e2e-worker-w4-3eb31c-589ff8b4b8-v89hw\" (UID: \"1df0a5e9-fb8c-49d7-a5ba-9a04e99b6c09\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-3eb31c-589ff8b4b8-v89hw" Nov 04 04:58:31.886414 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:31.886377 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1df0a5e9-fb8c-49d7-a5ba-9a04e99b6c09-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-3eb31c-589ff8b4b8-v89hw\" (UID: \"1df0a5e9-fb8c-49d7-a5ba-9a04e99b6c09\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-3eb31c-589ff8b4b8-v89hw" Nov 04 04:58:31.987944 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:31.987920 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfv9t\" (UniqueName: \"kubernetes.io/projected/1df0a5e9-fb8c-49d7-a5ba-9a04e99b6c09-kube-api-access-xfv9t\") pod \"payload-processing-e2e-worker-w4-3eb31c-589ff8b4b8-v89hw\" (UID: \"1df0a5e9-fb8c-49d7-a5ba-9a04e99b6c09\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-3eb31c-589ff8b4b8-v89hw" Nov 04 04:58:31.988069 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:31.987975 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1df0a5e9-fb8c-49d7-a5ba-9a04e99b6c09-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-3eb31c-589ff8b4b8-v89hw\" (UID: \"1df0a5e9-fb8c-49d7-a5ba-9a04e99b6c09\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-3eb31c-589ff8b4b8-v89hw" Nov 04 04:58:31.988609 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:31.988592 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1df0a5e9-fb8c-49d7-a5ba-9a04e99b6c09-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-3eb31c-589ff8b4b8-v89hw\" (UID: \"1df0a5e9-fb8c-49d7-a5ba-9a04e99b6c09\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-3eb31c-589ff8b4b8-v89hw" Nov 04 04:58:32.045688 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:32.045664 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfv9t\" (UniqueName: \"kubernetes.io/projected/1df0a5e9-fb8c-49d7-a5ba-9a04e99b6c09-kube-api-access-xfv9t\") pod \"payload-processing-e2e-worker-w4-3eb31c-589ff8b4b8-v89hw\" (UID: \"1df0a5e9-fb8c-49d7-a5ba-9a04e99b6c09\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-3eb31c-589ff8b4b8-v89hw" Nov 04 04:58:32.254032 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:32.254009 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-3eb31c-5bc57d7f87-xfpgv"] Nov 04 04:58:32.256528 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:58:32.256504 4509 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc425a743_509e_4356_86a2_8a1779db79ae.slice/crio-f7072aa1782a673e9f5f9be3ebee8e54aa124fea166c794183adce5877042fdd WatchSource:0}: Error finding container f7072aa1782a673e9f5f9be3ebee8e54aa124fea166c794183adce5877042fdd: Status 404 returned error can't find the container with id f7072aa1782a673e9f5f9be3ebee8e54aa124fea166c794183adce5877042fdd Nov 04 04:58:32.344939 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:32.344918 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-3eb31c-589ff8b4b8-v89hw" Nov 04 04:58:32.732476 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:32.732455 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-3eb31c-589ff8b4b8-v89hw"] Nov 04 04:58:32.734676 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:58:32.734651 4509 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1df0a5e9_fb8c_49d7_a5ba_9a04e99b6c09.slice/crio-ffc0ac2901187b3f7d7349055df9b469fd297c1dfc10448e1cb9913c7cc6aedc WatchSource:0}: Error finding container ffc0ac2901187b3f7d7349055df9b469fd297c1dfc10448e1cb9913c7cc6aedc: Status 404 returned error can't find the container with id ffc0ac2901187b3f7d7349055df9b469fd297c1dfc10448e1cb9913c7cc6aedc Nov 04 04:58:32.979085 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:32.979058 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-3eb31c-589ff8b4b8-v89hw" event={"ID":"1df0a5e9-fb8c-49d7-a5ba-9a04e99b6c09","Type":"ContainerStarted","Data":"e866d18b9f28b4f043c2ccb972699f0011f4a2e238b534ea473e444638f23f81"} Nov 04 04:58:32.979212 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:32.979115 4509 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-3eb31c-589ff8b4b8-v89hw" Nov 04 04:58:32.979212 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:32.979138 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-3eb31c-589ff8b4b8-v89hw" event={"ID":"1df0a5e9-fb8c-49d7-a5ba-9a04e99b6c09","Type":"ContainerStarted","Data":"ffc0ac2901187b3f7d7349055df9b469fd297c1dfc10448e1cb9913c7cc6aedc"} Nov 04 04:58:32.980606 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:32.980584 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3eb31c-5bc57d7f87-xfpgv" event={"ID":"c425a743-509e-4356-86a2-8a1779db79ae","Type":"ContainerStarted","Data":"9fe483b9696f72566747948c6bd8af9100050838fe1d8a66109d8d9aabaf9a9b"} Nov 04 04:58:32.980684 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:32.980623 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3eb31c-5bc57d7f87-xfpgv" event={"ID":"c425a743-509e-4356-86a2-8a1779db79ae","Type":"ContainerStarted","Data":"f7072aa1782a673e9f5f9be3ebee8e54aa124fea166c794183adce5877042fdd"} Nov 04 04:58:32.980684 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:32.980659 4509 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3eb31c-5bc57d7f87-xfpgv" Nov 04 04:58:33.099629 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:33.099547 4509 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-3eb31c-589ff8b4b8-v89hw" podStartSLOduration=2.099533128 podStartE2EDuration="2.099533128s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 04:58:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:58:33.099504037 +0000 UTC m=+676.977227559" watchObservedRunningTime="2026-11-04 04:58:33.099533128 +0000 UTC m=+676.977256649" Nov 04 04:58:34.048596 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:34.048551 4509 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3eb31c-5bc57d7f87-xfpgv" podStartSLOduration=3.048537864 podStartE2EDuration="3.048537864s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 04:58:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:58:33.226508023 +0000 UTC m=+677.104231544" watchObservedRunningTime="2026-11-04 04:58:34.048537864 +0000 UTC m=+677.926261431" Nov 04 04:58:34.050098 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:34.050080 4509 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-cedfca-67b95cc659-6v86t"] Nov 04 04:58:34.053651 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:34.053637 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-cedfca-67b95cc659-6v86t" Nov 04 04:58:34.108308 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:34.108270 4509 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-cedfca-dockercfg-5vrpq\"" Nov 04 04:58:34.108521 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:34.108367 4509 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-cedfca\"" Nov 04 04:58:34.111089 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:34.111066 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-cedfca-67b95cc659-6v86t"] Nov 04 04:58:34.205118 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:34.205088 4509 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-cedfca-c7c8f5b7d-hdqcw"] Nov 04 04:58:34.208317 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:34.208289 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vs8t\" (UniqueName: \"kubernetes.io/projected/5f041060-cb24-4d0a-be92-aabd9a536c92-kube-api-access-2vs8t\") pod \"payload-pre-processing-e2e-worker-w6-cedfca-67b95cc659-6v86t\" (UID: \"5f041060-cb24-4d0a-be92-aabd9a536c92\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-cedfca-67b95cc659-6v86t" Nov 04 04:58:34.208527 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:34.208503 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f041060-cb24-4d0a-be92-aabd9a536c92-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-cedfca-67b95cc659-6v86t\" (UID: \"5f041060-cb24-4d0a-be92-aabd9a536c92\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-cedfca-67b95cc659-6v86t" Nov 04 04:58:34.208628 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:34.208614 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-cedfca-c7c8f5b7d-hdqcw" Nov 04 04:58:34.266095 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:34.266055 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-cedfca-c7c8f5b7d-hdqcw"] Nov 04 04:58:34.309201 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:34.309134 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7xcg\" (UniqueName: \"kubernetes.io/projected/c6376390-553a-41f1-87d3-31e63d9e929d-kube-api-access-q7xcg\") pod \"payload-processing-e2e-worker-w6-cedfca-c7c8f5b7d-hdqcw\" (UID: \"c6376390-553a-41f1-87d3-31e63d9e929d\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-cedfca-c7c8f5b7d-hdqcw" Nov 04 04:58:34.309201 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:34.309185 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f041060-cb24-4d0a-be92-aabd9a536c92-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-cedfca-67b95cc659-6v86t\" (UID: \"5f041060-cb24-4d0a-be92-aabd9a536c92\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-cedfca-67b95cc659-6v86t" Nov 04 04:58:34.309431 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:34.309340 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vs8t\" (UniqueName: \"kubernetes.io/projected/5f041060-cb24-4d0a-be92-aabd9a536c92-kube-api-access-2vs8t\") pod \"payload-pre-processing-e2e-worker-w6-cedfca-67b95cc659-6v86t\" (UID: \"5f041060-cb24-4d0a-be92-aabd9a536c92\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-cedfca-67b95cc659-6v86t" Nov 04 04:58:34.309431 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:34.309366 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6376390-553a-41f1-87d3-31e63d9e929d-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-cedfca-c7c8f5b7d-hdqcw\" (UID: \"c6376390-553a-41f1-87d3-31e63d9e929d\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-cedfca-c7c8f5b7d-hdqcw" Nov 04 04:58:34.309920 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:34.309899 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f041060-cb24-4d0a-be92-aabd9a536c92-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-cedfca-67b95cc659-6v86t\" (UID: \"5f041060-cb24-4d0a-be92-aabd9a536c92\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-cedfca-67b95cc659-6v86t" Nov 04 04:58:34.366383 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:34.366360 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vs8t\" (UniqueName: \"kubernetes.io/projected/5f041060-cb24-4d0a-be92-aabd9a536c92-kube-api-access-2vs8t\") pod \"payload-pre-processing-e2e-worker-w6-cedfca-67b95cc659-6v86t\" (UID: \"5f041060-cb24-4d0a-be92-aabd9a536c92\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-cedfca-67b95cc659-6v86t" Nov 04 04:58:34.368176 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:34.368145 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-cedfca-67b95cc659-6v86t" Nov 04 04:58:34.410436 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:34.410411 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6376390-553a-41f1-87d3-31e63d9e929d-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-cedfca-c7c8f5b7d-hdqcw\" (UID: \"c6376390-553a-41f1-87d3-31e63d9e929d\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-cedfca-c7c8f5b7d-hdqcw" Nov 04 04:58:34.410559 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:34.410470 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7xcg\" (UniqueName: \"kubernetes.io/projected/c6376390-553a-41f1-87d3-31e63d9e929d-kube-api-access-q7xcg\") pod \"payload-processing-e2e-worker-w6-cedfca-c7c8f5b7d-hdqcw\" (UID: \"c6376390-553a-41f1-87d3-31e63d9e929d\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-cedfca-c7c8f5b7d-hdqcw" Nov 04 04:58:34.411035 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:34.411018 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6376390-553a-41f1-87d3-31e63d9e929d-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-cedfca-c7c8f5b7d-hdqcw\" (UID: \"c6376390-553a-41f1-87d3-31e63d9e929d\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-cedfca-c7c8f5b7d-hdqcw" Nov 04 04:58:34.468222 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:34.468201 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7xcg\" (UniqueName: \"kubernetes.io/projected/c6376390-553a-41f1-87d3-31e63d9e929d-kube-api-access-q7xcg\") pod \"payload-processing-e2e-worker-w6-cedfca-c7c8f5b7d-hdqcw\" (UID: \"c6376390-553a-41f1-87d3-31e63d9e929d\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-cedfca-c7c8f5b7d-hdqcw" Nov 04 04:58:34.521011 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:34.520991 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-cedfca-c7c8f5b7d-hdqcw" Nov 04 04:58:34.739760 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:34.738938 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-cedfca-67b95cc659-6v86t"] Nov 04 04:58:34.893908 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:34.893857 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-cedfca-c7c8f5b7d-hdqcw"] Nov 04 04:58:34.896165 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:58:34.896137 4509 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6376390_553a_41f1_87d3_31e63d9e929d.slice/crio-f304bb19495c418193c8503ce5d19999b026ed2322502e61a5f7313b7d907790 WatchSource:0}: Error finding container f304bb19495c418193c8503ce5d19999b026ed2322502e61a5f7313b7d907790: Status 404 returned error can't find the container with id f304bb19495c418193c8503ce5d19999b026ed2322502e61a5f7313b7d907790 Nov 04 04:58:34.991446 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:34.991421 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-cedfca-c7c8f5b7d-hdqcw" event={"ID":"c6376390-553a-41f1-87d3-31e63d9e929d","Type":"ContainerStarted","Data":"41fdd7a11af2f438301f249dec8733847f297b94e70fb208b7d4258910600bdf"} Nov 04 04:58:34.991561 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:34.991474 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-cedfca-c7c8f5b7d-hdqcw" event={"ID":"c6376390-553a-41f1-87d3-31e63d9e929d","Type":"ContainerStarted","Data":"f304bb19495c418193c8503ce5d19999b026ed2322502e61a5f7313b7d907790"} Nov 04 04:58:34.991561 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:34.991545 4509 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-cedfca-c7c8f5b7d-hdqcw" Nov 04 04:58:34.993065 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:34.993044 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-cedfca-67b95cc659-6v86t" event={"ID":"5f041060-cb24-4d0a-be92-aabd9a536c92","Type":"ContainerStarted","Data":"d683c478c107507a70e1739f169c06fcda658c6ea0c5ab0a729b212b953e95df"} Nov 04 04:58:34.993153 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:34.993078 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-cedfca-67b95cc659-6v86t" event={"ID":"5f041060-cb24-4d0a-be92-aabd9a536c92","Type":"ContainerStarted","Data":"18289411500cfdefece7185753c92620a504329621c613d3068a0cf61da23f6e"} Nov 04 04:58:34.993153 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:34.993143 4509 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-cedfca-67b95cc659-6v86t" Nov 04 04:58:35.104917 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:35.104877 4509 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w6-cedfca-c7c8f5b7d-hdqcw" podStartSLOduration=1.104864111 podStartE2EDuration="1.104864111s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 04:58:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:58:35.104495918 +0000 UTC m=+678.982219440" watchObservedRunningTime="2026-11-04 04:58:35.104864111 +0000 UTC m=+678.982587764" Nov 04 04:58:35.234019 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:35.233968 4509 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-cedfca-67b95cc659-6v86t" podStartSLOduration=1.233951953 podStartE2EDuration="1.233951953s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 04:58:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:58:35.232650627 +0000 UTC m=+679.110374149" watchObservedRunningTime="2026-11-04 04:58:35.233951953 +0000 UTC m=+679.111675473" Nov 04 04:58:38.925786 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:38.925755 4509 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-370078-6c55687f5f-gwlxz"] Nov 04 04:58:38.929454 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:38.929434 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-370078-6c55687f5f-gwlxz" Nov 04 04:58:38.986393 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:38.986372 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-370078-6c55687f5f-gwlxz"] Nov 04 04:58:39.034520 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:39.034494 4509 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-370078\"" Nov 04 04:58:39.034869 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:39.034852 4509 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-370078-dockercfg-w5f84\"" Nov 04 04:58:39.064518 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:39.064487 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdwlb\" (UniqueName: \"kubernetes.io/projected/bbe04b59-da69-46ac-ba51-5b617ca5f174-kube-api-access-jdwlb\") pod \"payload-pre-processing-e2e-worker-w2-370078-6c55687f5f-gwlxz\" (UID: \"bbe04b59-da69-46ac-ba51-5b617ca5f174\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-370078-6c55687f5f-gwlxz" Nov 04 04:58:39.064676 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:39.064632 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbe04b59-da69-46ac-ba51-5b617ca5f174-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-370078-6c55687f5f-gwlxz\" (UID: \"bbe04b59-da69-46ac-ba51-5b617ca5f174\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-370078-6c55687f5f-gwlxz" Nov 04 04:58:39.078427 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:39.078403 4509 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-370078-dc9467d44-l4644"] Nov 04 04:58:39.081734 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:39.081719 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-370078-dc9467d44-l4644" Nov 04 04:58:39.148901 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:39.148865 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-370078-dc9467d44-l4644"] Nov 04 04:58:39.165167 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:39.165143 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbe04b59-da69-46ac-ba51-5b617ca5f174-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-370078-6c55687f5f-gwlxz\" (UID: \"bbe04b59-da69-46ac-ba51-5b617ca5f174\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-370078-6c55687f5f-gwlxz" Nov 04 04:58:39.165272 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:39.165208 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdwlb\" (UniqueName: \"kubernetes.io/projected/bbe04b59-da69-46ac-ba51-5b617ca5f174-kube-api-access-jdwlb\") pod \"payload-pre-processing-e2e-worker-w2-370078-6c55687f5f-gwlxz\" (UID: \"bbe04b59-da69-46ac-ba51-5b617ca5f174\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-370078-6c55687f5f-gwlxz" Nov 04 04:58:39.165272 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:39.165249 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqscs\" (UniqueName: \"kubernetes.io/projected/64a2b08c-1fc3-4334-96f9-97bfc88859b2-kube-api-access-wqscs\") pod \"payload-processing-e2e-worker-w2-370078-dc9467d44-l4644\" (UID: \"64a2b08c-1fc3-4334-96f9-97bfc88859b2\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-370078-dc9467d44-l4644" Nov 04 04:58:39.165408 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:39.165278 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64a2b08c-1fc3-4334-96f9-97bfc88859b2-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-370078-dc9467d44-l4644\" (UID: \"64a2b08c-1fc3-4334-96f9-97bfc88859b2\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-370078-dc9467d44-l4644" Nov 04 04:58:39.165825 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:39.165805 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbe04b59-da69-46ac-ba51-5b617ca5f174-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-370078-6c55687f5f-gwlxz\" (UID: \"bbe04b59-da69-46ac-ba51-5b617ca5f174\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-370078-6c55687f5f-gwlxz" Nov 04 04:58:39.222442 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:39.222422 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdwlb\" (UniqueName: \"kubernetes.io/projected/bbe04b59-da69-46ac-ba51-5b617ca5f174-kube-api-access-jdwlb\") pod \"payload-pre-processing-e2e-worker-w2-370078-6c55687f5f-gwlxz\" (UID: \"bbe04b59-da69-46ac-ba51-5b617ca5f174\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-370078-6c55687f5f-gwlxz" Nov 04 04:58:39.242239 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:39.242217 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-370078-6c55687f5f-gwlxz" Nov 04 04:58:39.266566 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:39.266543 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqscs\" (UniqueName: \"kubernetes.io/projected/64a2b08c-1fc3-4334-96f9-97bfc88859b2-kube-api-access-wqscs\") pod \"payload-processing-e2e-worker-w2-370078-dc9467d44-l4644\" (UID: \"64a2b08c-1fc3-4334-96f9-97bfc88859b2\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-370078-dc9467d44-l4644" Nov 04 04:58:39.266665 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:39.266576 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64a2b08c-1fc3-4334-96f9-97bfc88859b2-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-370078-dc9467d44-l4644\" (UID: \"64a2b08c-1fc3-4334-96f9-97bfc88859b2\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-370078-dc9467d44-l4644" Nov 04 04:58:39.267281 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:39.267261 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64a2b08c-1fc3-4334-96f9-97bfc88859b2-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-370078-dc9467d44-l4644\" (UID: \"64a2b08c-1fc3-4334-96f9-97bfc88859b2\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-370078-dc9467d44-l4644" Nov 04 04:58:39.325661 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:39.325636 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqscs\" (UniqueName: \"kubernetes.io/projected/64a2b08c-1fc3-4334-96f9-97bfc88859b2-kube-api-access-wqscs\") pod \"payload-processing-e2e-worker-w2-370078-dc9467d44-l4644\" (UID: \"64a2b08c-1fc3-4334-96f9-97bfc88859b2\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-370078-dc9467d44-l4644" Nov 04 04:58:39.394020 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:39.393994 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-370078-dc9467d44-l4644" Nov 04 04:58:39.611590 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:39.611566 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-370078-6c55687f5f-gwlxz"] Nov 04 04:58:39.613618 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:58:39.613597 4509 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbe04b59_da69_46ac_ba51_5b617ca5f174.slice/crio-d6e797ce9e249fd1517e02bcde7b5e767e1229c2fc76610be28776dc99c88f82 WatchSource:0}: Error finding container d6e797ce9e249fd1517e02bcde7b5e767e1229c2fc76610be28776dc99c88f82: Status 404 returned error can't find the container with id d6e797ce9e249fd1517e02bcde7b5e767e1229c2fc76610be28776dc99c88f82 Nov 04 04:58:39.763828 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:39.763806 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-370078-dc9467d44-l4644"] Nov 04 04:58:39.765751 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:58:39.765725 4509 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64a2b08c_1fc3_4334_96f9_97bfc88859b2.slice/crio-4ec0a2e01f9676bb6189efb8b39b9f82ad5ff462dedfb331d19db3fc05f79b8c WatchSource:0}: Error finding container 4ec0a2e01f9676bb6189efb8b39b9f82ad5ff462dedfb331d19db3fc05f79b8c: Status 404 returned error can't find the container with id 4ec0a2e01f9676bb6189efb8b39b9f82ad5ff462dedfb331d19db3fc05f79b8c Nov 04 04:58:39.967158 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:39.967135 4509 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-4cce33-56666d7678-r88jh" Nov 04 04:58:40.035743 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:40.035708 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-370078-dc9467d44-l4644" event={"ID":"64a2b08c-1fc3-4334-96f9-97bfc88859b2","Type":"ContainerStarted","Data":"eb0fde9670adda43659fde6b48727ea0b095c807b57412b34c78854fb02799b3"} Nov 04 04:58:40.035915 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:40.035764 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-370078-dc9467d44-l4644" event={"ID":"64a2b08c-1fc3-4334-96f9-97bfc88859b2","Type":"ContainerStarted","Data":"4ec0a2e01f9676bb6189efb8b39b9f82ad5ff462dedfb331d19db3fc05f79b8c"} Nov 04 04:58:40.035915 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:40.035795 4509 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-370078-dc9467d44-l4644" Nov 04 04:58:40.037289 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:40.037267 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-370078-6c55687f5f-gwlxz" event={"ID":"bbe04b59-da69-46ac-ba51-5b617ca5f174","Type":"ContainerStarted","Data":"1a3f43f3121445a046535453d28cc44f61593dee2ed5566852feabffc5869d13"} Nov 04 04:58:40.037410 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:40.037307 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-370078-6c55687f5f-gwlxz" event={"ID":"bbe04b59-da69-46ac-ba51-5b617ca5f174","Type":"ContainerStarted","Data":"d6e797ce9e249fd1517e02bcde7b5e767e1229c2fc76610be28776dc99c88f82"} Nov 04 04:58:40.037410 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:40.037361 4509 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-370078-6c55687f5f-gwlxz" Nov 04 04:58:40.208820 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:40.208729 4509 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-370078-6c55687f5f-gwlxz" podStartSLOduration=2.208706599 podStartE2EDuration="2.208706599s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 04:58:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:58:40.204606291 +0000 UTC m=+684.082329810" watchObservedRunningTime="2026-11-04 04:58:40.208706599 +0000 UTC m=+684.086430121" Nov 04 04:58:40.330106 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:40.324551 4509 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-370078-dc9467d44-l4644" podStartSLOduration=1.324529596 podStartE2EDuration="1.324529596s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 04:58:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:58:40.322748796 +0000 UTC m=+684.200472320" watchObservedRunningTime="2026-11-04 04:58:40.324529596 +0000 UTC m=+684.202253122" Nov 04 04:58:41.649626 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:41.649586 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-42dnm"] Nov 04 04:58:43.988512 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:43.988482 4509 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-3eb31c-589ff8b4b8-v89hw" Nov 04 04:58:43.988966 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:43.988557 4509 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3eb31c-5bc57d7f87-xfpgv" Nov 04 04:58:44.453707 ip-10-0-133-180 kubenswrapper[4509]: E1104 04:58:44.453628 4509 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmdhl" containerName="extractor" Nov 04 04:58:46.001501 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:46.001468 4509 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-cedfca-67b95cc659-6v86t" Nov 04 04:58:46.001921 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:46.001631 4509 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-cedfca-c7c8f5b7d-hdqcw" Nov 04 04:58:46.106382 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:46.106352 4509 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-0aae2f-5ff7778f8c-4tbhr"] Nov 04 04:58:46.110206 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:46.110192 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-0aae2f-5ff7778f8c-4tbhr" Nov 04 04:58:46.138828 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:46.138364 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-0aae2f-5ff7778f8c-4tbhr"] Nov 04 04:58:46.182348 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:46.170898 4509 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-0aae2f-dockercfg-rm7mw\"" Nov 04 04:58:46.182348 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:46.171033 4509 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-0aae2f\"" Nov 04 04:58:46.222837 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:46.222801 4509 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-0aae2f-74664f8f76-rzd5v"] Nov 04 04:58:46.226848 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:46.226719 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-0aae2f-74664f8f76-rzd5v" Nov 04 04:58:46.248303 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:46.248278 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gt9tx\" (UniqueName: \"kubernetes.io/projected/e1f20ac4-af33-4c51-8ae3-a286a4c9bc2c-kube-api-access-gt9tx\") pod \"payload-pre-processing-e2e-worker-w0-0aae2f-5ff7778f8c-4tbhr\" (UID: \"e1f20ac4-af33-4c51-8ae3-a286a4c9bc2c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-0aae2f-5ff7778f8c-4tbhr" Nov 04 04:58:46.248470 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:46.248401 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1f20ac4-af33-4c51-8ae3-a286a4c9bc2c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-0aae2f-5ff7778f8c-4tbhr\" (UID: \"e1f20ac4-af33-4c51-8ae3-a286a4c9bc2c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-0aae2f-5ff7778f8c-4tbhr" Nov 04 04:58:46.288075 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:46.287648 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-0aae2f-74664f8f76-rzd5v"] Nov 04 04:58:46.349373 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:46.349314 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f1d0cf5-2aa4-49c7-9153-4bc44df28fac-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-0aae2f-74664f8f76-rzd5v\" (UID: \"2f1d0cf5-2aa4-49c7-9153-4bc44df28fac\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-0aae2f-74664f8f76-rzd5v" Nov 04 04:58:46.349525 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:46.349398 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjhzd\" (UniqueName: \"kubernetes.io/projected/2f1d0cf5-2aa4-49c7-9153-4bc44df28fac-kube-api-access-sjhzd\") pod \"payload-processing-e2e-worker-w0-0aae2f-74664f8f76-rzd5v\" (UID: \"2f1d0cf5-2aa4-49c7-9153-4bc44df28fac\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-0aae2f-74664f8f76-rzd5v" Nov 04 04:58:46.349525 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:46.349459 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gt9tx\" (UniqueName: \"kubernetes.io/projected/e1f20ac4-af33-4c51-8ae3-a286a4c9bc2c-kube-api-access-gt9tx\") pod \"payload-pre-processing-e2e-worker-w0-0aae2f-5ff7778f8c-4tbhr\" (UID: \"e1f20ac4-af33-4c51-8ae3-a286a4c9bc2c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-0aae2f-5ff7778f8c-4tbhr" Nov 04 04:58:46.349525 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:46.349505 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1f20ac4-af33-4c51-8ae3-a286a4c9bc2c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-0aae2f-5ff7778f8c-4tbhr\" (UID: \"e1f20ac4-af33-4c51-8ae3-a286a4c9bc2c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-0aae2f-5ff7778f8c-4tbhr" Nov 04 04:58:46.350172 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:46.350155 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1f20ac4-af33-4c51-8ae3-a286a4c9bc2c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-0aae2f-5ff7778f8c-4tbhr\" (UID: \"e1f20ac4-af33-4c51-8ae3-a286a4c9bc2c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-0aae2f-5ff7778f8c-4tbhr" Nov 04 04:58:46.406157 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:46.406130 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gt9tx\" (UniqueName: \"kubernetes.io/projected/e1f20ac4-af33-4c51-8ae3-a286a4c9bc2c-kube-api-access-gt9tx\") pod \"payload-pre-processing-e2e-worker-w0-0aae2f-5ff7778f8c-4tbhr\" (UID: \"e1f20ac4-af33-4c51-8ae3-a286a4c9bc2c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-0aae2f-5ff7778f8c-4tbhr" Nov 04 04:58:46.440548 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:46.440526 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-0aae2f-5ff7778f8c-4tbhr" Nov 04 04:58:46.451252 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:46.451131 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f1d0cf5-2aa4-49c7-9153-4bc44df28fac-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-0aae2f-74664f8f76-rzd5v\" (UID: \"2f1d0cf5-2aa4-49c7-9153-4bc44df28fac\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-0aae2f-74664f8f76-rzd5v" Nov 04 04:58:46.451252 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:46.451164 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjhzd\" (UniqueName: \"kubernetes.io/projected/2f1d0cf5-2aa4-49c7-9153-4bc44df28fac-kube-api-access-sjhzd\") pod \"payload-processing-e2e-worker-w0-0aae2f-74664f8f76-rzd5v\" (UID: \"2f1d0cf5-2aa4-49c7-9153-4bc44df28fac\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-0aae2f-74664f8f76-rzd5v" Nov 04 04:58:46.451819 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:46.451800 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f1d0cf5-2aa4-49c7-9153-4bc44df28fac-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-0aae2f-74664f8f76-rzd5v\" (UID: \"2f1d0cf5-2aa4-49c7-9153-4bc44df28fac\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-0aae2f-74664f8f76-rzd5v" Nov 04 04:58:46.508979 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:46.508940 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjhzd\" (UniqueName: \"kubernetes.io/projected/2f1d0cf5-2aa4-49c7-9153-4bc44df28fac-kube-api-access-sjhzd\") pod \"payload-processing-e2e-worker-w0-0aae2f-74664f8f76-rzd5v\" (UID: \"2f1d0cf5-2aa4-49c7-9153-4bc44df28fac\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-0aae2f-74664f8f76-rzd5v" Nov 04 04:58:46.543647 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:46.543593 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-0aae2f-74664f8f76-rzd5v" Nov 04 04:58:46.835981 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:46.835943 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-0aae2f-5ff7778f8c-4tbhr"] Nov 04 04:58:46.838095 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:58:46.838066 4509 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1f20ac4_af33_4c51_8ae3_a286a4c9bc2c.slice/crio-70edbca3af1fe3e9f644639ec3406d05531856b05e21b3bf66cedccc7671afe1 WatchSource:0}: Error finding container 70edbca3af1fe3e9f644639ec3406d05531856b05e21b3bf66cedccc7671afe1: Status 404 returned error can't find the container with id 70edbca3af1fe3e9f644639ec3406d05531856b05e21b3bf66cedccc7671afe1 Nov 04 04:58:46.942481 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:46.942461 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-0aae2f-74664f8f76-rzd5v"] Nov 04 04:58:46.943969 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:58:46.943946 4509 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f1d0cf5_2aa4_49c7_9153_4bc44df28fac.slice/crio-8c122dd7e58a8662c323cbdcab4f10d430f1c381fa8a1287960f70746a29386b WatchSource:0}: Error finding container 8c122dd7e58a8662c323cbdcab4f10d430f1c381fa8a1287960f70746a29386b: Status 404 returned error can't find the container with id 8c122dd7e58a8662c323cbdcab4f10d430f1c381fa8a1287960f70746a29386b Nov 04 04:58:47.093955 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:47.093880 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-0aae2f-5ff7778f8c-4tbhr" event={"ID":"e1f20ac4-af33-4c51-8ae3-a286a4c9bc2c","Type":"ContainerStarted","Data":"08f4b0e09701b0cc67798a27385f32cf7c5aa1f27a06a63ed3afbd73c6c74079"} Nov 04 04:58:47.093955 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:47.093945 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-0aae2f-5ff7778f8c-4tbhr" event={"ID":"e1f20ac4-af33-4c51-8ae3-a286a4c9bc2c","Type":"ContainerStarted","Data":"70edbca3af1fe3e9f644639ec3406d05531856b05e21b3bf66cedccc7671afe1"} Nov 04 04:58:47.094438 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:47.093976 4509 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-0aae2f-5ff7778f8c-4tbhr" Nov 04 04:58:47.095540 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:47.095516 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-0aae2f-74664f8f76-rzd5v" event={"ID":"2f1d0cf5-2aa4-49c7-9153-4bc44df28fac","Type":"ContainerStarted","Data":"bc2a4e535a03a824ce7352aadb71e3507d7d92f70e2b9156e11966ff5d27f647"} Nov 04 04:58:47.095634 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:47.095564 4509 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-0aae2f-74664f8f76-rzd5v" Nov 04 04:58:47.095634 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:47.095579 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-0aae2f-74664f8f76-rzd5v" event={"ID":"2f1d0cf5-2aa4-49c7-9153-4bc44df28fac","Type":"ContainerStarted","Data":"8c122dd7e58a8662c323cbdcab4f10d430f1c381fa8a1287960f70746a29386b"} Nov 04 04:58:47.207260 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:47.207214 4509 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-0aae2f-5ff7778f8c-4tbhr" podStartSLOduration=1.207201198 podStartE2EDuration="1.207201198s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 04:58:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:58:47.206500169 +0000 UTC m=+691.084223689" watchObservedRunningTime="2026-11-04 04:58:47.207201198 +0000 UTC m=+691.084924718" Nov 04 04:58:47.322727 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:47.322667 4509 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-0aae2f-74664f8f76-rzd5v" podStartSLOduration=1.322649047 podStartE2EDuration="1.322649047s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 04:58:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:58:47.32160439 +0000 UTC m=+691.199327911" watchObservedRunningTime="2026-11-04 04:58:47.322649047 +0000 UTC m=+691.200372570" Nov 04 04:58:48.181922 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:48.181890 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-42dnm"] Nov 04 04:58:48.580663 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:48.580622 4509 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-1874d8-78f4d9f8b7-9d2tj"] Nov 04 04:58:48.584405 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:48.584385 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-1874d8-78f4d9f8b7-9d2tj" Nov 04 04:58:48.688648 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:48.687813 4509 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-1874d8-dockercfg-54m2q\"" Nov 04 04:58:48.688648 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:48.687948 4509 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-1874d8\"" Nov 04 04:58:48.690458 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:48.690434 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-1874d8-78f4d9f8b7-9d2tj"] Nov 04 04:58:48.693631 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:48.693599 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnkns\" (UniqueName: \"kubernetes.io/projected/e5103c9a-0929-422e-8c95-b342181f0669-kube-api-access-lnkns\") pod \"payload-pre-processing-e2e-worker-w1-1874d8-78f4d9f8b7-9d2tj\" (UID: \"e5103c9a-0929-422e-8c95-b342181f0669\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-1874d8-78f4d9f8b7-9d2tj" Nov 04 04:58:48.693893 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:48.693865 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5103c9a-0929-422e-8c95-b342181f0669-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-1874d8-78f4d9f8b7-9d2tj\" (UID: \"e5103c9a-0929-422e-8c95-b342181f0669\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-1874d8-78f4d9f8b7-9d2tj" Nov 04 04:58:48.797555 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:48.795700 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnkns\" (UniqueName: \"kubernetes.io/projected/e5103c9a-0929-422e-8c95-b342181f0669-kube-api-access-lnkns\") pod \"payload-pre-processing-e2e-worker-w1-1874d8-78f4d9f8b7-9d2tj\" (UID: \"e5103c9a-0929-422e-8c95-b342181f0669\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-1874d8-78f4d9f8b7-9d2tj" Nov 04 04:58:48.797555 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:48.795871 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5103c9a-0929-422e-8c95-b342181f0669-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-1874d8-78f4d9f8b7-9d2tj\" (UID: \"e5103c9a-0929-422e-8c95-b342181f0669\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-1874d8-78f4d9f8b7-9d2tj" Nov 04 04:58:48.797555 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:48.796856 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5103c9a-0929-422e-8c95-b342181f0669-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-1874d8-78f4d9f8b7-9d2tj\" (UID: \"e5103c9a-0929-422e-8c95-b342181f0669\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-1874d8-78f4d9f8b7-9d2tj" Nov 04 04:58:48.853994 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:48.853917 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnkns\" (UniqueName: \"kubernetes.io/projected/e5103c9a-0929-422e-8c95-b342181f0669-kube-api-access-lnkns\") pod \"payload-pre-processing-e2e-worker-w1-1874d8-78f4d9f8b7-9d2tj\" (UID: \"e5103c9a-0929-422e-8c95-b342181f0669\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-1874d8-78f4d9f8b7-9d2tj" Nov 04 04:58:48.899925 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:48.899888 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-1874d8-78f4d9f8b7-9d2tj" Nov 04 04:58:49.276060 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:49.276021 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-1874d8-78f4d9f8b7-9d2tj"] Nov 04 04:58:49.289957 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:58:49.289919 4509 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5103c9a_0929_422e_8c95_b342181f0669.slice/crio-1b4d6fd92b32bde69c73fb0eaaa01cc9edafcb5ca839a77dbeba64bc5748a516 WatchSource:0}: Error finding container 1b4d6fd92b32bde69c73fb0eaaa01cc9edafcb5ca839a77dbeba64bc5748a516: Status 404 returned error can't find the container with id 1b4d6fd92b32bde69c73fb0eaaa01cc9edafcb5ca839a77dbeba64bc5748a516 Nov 04 04:58:49.392533 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:49.392501 4509 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-g4ctm"] Nov 04 04:58:49.396760 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:49.396735 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-g4ctm" Nov 04 04:58:49.439476 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:49.439394 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-g4ctm"] Nov 04 04:58:49.486351 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:49.486295 4509 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-3eb31c-openshift-default-dockercfg-bscg9\"" Nov 04 04:58:49.502949 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:49.502910 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jrtf\" (UniqueName: \"kubernetes.io/projected/c985ba5a-3feb-4b17-959e-b4ca34a5dfbc-kube-api-access-4jrtf\") pod \"e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-g4ctm\" (UID: \"c985ba5a-3feb-4b17-959e-b4ca34a5dfbc\") " pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-g4ctm" Nov 04 04:58:49.503130 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:49.502972 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c985ba5a-3feb-4b17-959e-b4ca34a5dfbc-workload-socket\") pod \"e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-g4ctm\" (UID: \"c985ba5a-3feb-4b17-959e-b4ca34a5dfbc\") " pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-g4ctm" Nov 04 04:58:49.503130 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:49.503043 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c985ba5a-3feb-4b17-959e-b4ca34a5dfbc-credential-socket\") pod \"e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-g4ctm\" (UID: \"c985ba5a-3feb-4b17-959e-b4ca34a5dfbc\") " pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-g4ctm" Nov 04 04:58:49.503130 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:49.503075 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c985ba5a-3feb-4b17-959e-b4ca34a5dfbc-istio-token\") pod \"e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-g4ctm\" (UID: \"c985ba5a-3feb-4b17-959e-b4ca34a5dfbc\") " pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-g4ctm" Nov 04 04:58:49.503130 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:49.503098 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c985ba5a-3feb-4b17-959e-b4ca34a5dfbc-istio-envoy\") pod \"e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-g4ctm\" (UID: \"c985ba5a-3feb-4b17-959e-b4ca34a5dfbc\") " pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-g4ctm" Nov 04 04:58:49.503371 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:49.503233 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c985ba5a-3feb-4b17-959e-b4ca34a5dfbc-istiod-ca-cert\") pod \"e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-g4ctm\" (UID: \"c985ba5a-3feb-4b17-959e-b4ca34a5dfbc\") " pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-g4ctm" Nov 04 04:58:49.503371 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:49.503345 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c985ba5a-3feb-4b17-959e-b4ca34a5dfbc-istio-data\") pod \"e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-g4ctm\" (UID: \"c985ba5a-3feb-4b17-959e-b4ca34a5dfbc\") " pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-g4ctm" Nov 04 04:58:49.503499 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:49.503470 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c985ba5a-3feb-4b17-959e-b4ca34a5dfbc-istio-podinfo\") pod \"e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-g4ctm\" (UID: \"c985ba5a-3feb-4b17-959e-b4ca34a5dfbc\") " pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-g4ctm" Nov 04 04:58:49.503559 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:49.503512 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c985ba5a-3feb-4b17-959e-b4ca34a5dfbc-workload-certs\") pod \"e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-g4ctm\" (UID: \"c985ba5a-3feb-4b17-959e-b4ca34a5dfbc\") " pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-g4ctm" Nov 04 04:58:49.605016 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:49.604937 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jrtf\" (UniqueName: \"kubernetes.io/projected/c985ba5a-3feb-4b17-959e-b4ca34a5dfbc-kube-api-access-4jrtf\") pod \"e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-g4ctm\" (UID: \"c985ba5a-3feb-4b17-959e-b4ca34a5dfbc\") " pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-g4ctm" Nov 04 04:58:49.605016 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:49.604986 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c985ba5a-3feb-4b17-959e-b4ca34a5dfbc-workload-socket\") pod \"e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-g4ctm\" (UID: \"c985ba5a-3feb-4b17-959e-b4ca34a5dfbc\") " pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-g4ctm" Nov 04 04:58:49.605016 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:49.605019 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c985ba5a-3feb-4b17-959e-b4ca34a5dfbc-credential-socket\") pod \"e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-g4ctm\" (UID: \"c985ba5a-3feb-4b17-959e-b4ca34a5dfbc\") " pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-g4ctm" Nov 04 04:58:49.605287 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:49.605046 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c985ba5a-3feb-4b17-959e-b4ca34a5dfbc-istio-token\") pod \"e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-g4ctm\" (UID: \"c985ba5a-3feb-4b17-959e-b4ca34a5dfbc\") " pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-g4ctm" Nov 04 04:58:49.605287 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:49.605072 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c985ba5a-3feb-4b17-959e-b4ca34a5dfbc-istio-envoy\") pod \"e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-g4ctm\" (UID: \"c985ba5a-3feb-4b17-959e-b4ca34a5dfbc\") " pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-g4ctm" Nov 04 04:58:49.605287 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:49.605129 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c985ba5a-3feb-4b17-959e-b4ca34a5dfbc-istiod-ca-cert\") pod \"e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-g4ctm\" (UID: \"c985ba5a-3feb-4b17-959e-b4ca34a5dfbc\") " pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-g4ctm" Nov 04 04:58:49.605287 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:49.605159 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c985ba5a-3feb-4b17-959e-b4ca34a5dfbc-istio-data\") pod \"e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-g4ctm\" (UID: \"c985ba5a-3feb-4b17-959e-b4ca34a5dfbc\") " pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-g4ctm" Nov 04 04:58:49.605287 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:49.605218 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c985ba5a-3feb-4b17-959e-b4ca34a5dfbc-istio-podinfo\") pod \"e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-g4ctm\" (UID: \"c985ba5a-3feb-4b17-959e-b4ca34a5dfbc\") " pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-g4ctm" Nov 04 04:58:49.605287 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:49.605250 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c985ba5a-3feb-4b17-959e-b4ca34a5dfbc-workload-certs\") pod \"e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-g4ctm\" (UID: \"c985ba5a-3feb-4b17-959e-b4ca34a5dfbc\") " pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-g4ctm" Nov 04 04:58:49.605625 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:49.605482 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c985ba5a-3feb-4b17-959e-b4ca34a5dfbc-credential-socket\") pod \"e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-g4ctm\" (UID: \"c985ba5a-3feb-4b17-959e-b4ca34a5dfbc\") " pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-g4ctm" Nov 04 04:58:49.605625 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:49.605496 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c985ba5a-3feb-4b17-959e-b4ca34a5dfbc-workload-socket\") pod \"e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-g4ctm\" (UID: \"c985ba5a-3feb-4b17-959e-b4ca34a5dfbc\") " pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-g4ctm" Nov 04 04:58:49.605730 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:49.605637 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c985ba5a-3feb-4b17-959e-b4ca34a5dfbc-workload-certs\") pod \"e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-g4ctm\" (UID: \"c985ba5a-3feb-4b17-959e-b4ca34a5dfbc\") " pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-g4ctm" Nov 04 04:58:49.605730 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:49.605720 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c985ba5a-3feb-4b17-959e-b4ca34a5dfbc-istio-data\") pod \"e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-g4ctm\" (UID: \"c985ba5a-3feb-4b17-959e-b4ca34a5dfbc\") " pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-g4ctm" Nov 04 04:58:49.606153 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:49.606130 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c985ba5a-3feb-4b17-959e-b4ca34a5dfbc-istiod-ca-cert\") pod \"e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-g4ctm\" (UID: \"c985ba5a-3feb-4b17-959e-b4ca34a5dfbc\") " pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-g4ctm" Nov 04 04:58:49.607950 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:49.607930 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c985ba5a-3feb-4b17-959e-b4ca34a5dfbc-istio-envoy\") pod \"e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-g4ctm\" (UID: \"c985ba5a-3feb-4b17-959e-b4ca34a5dfbc\") " pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-g4ctm" Nov 04 04:58:49.608115 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:49.608098 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c985ba5a-3feb-4b17-959e-b4ca34a5dfbc-istio-podinfo\") pod \"e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-g4ctm\" (UID: \"c985ba5a-3feb-4b17-959e-b4ca34a5dfbc\") " pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-g4ctm" Nov 04 04:58:49.686807 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:49.686771 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c985ba5a-3feb-4b17-959e-b4ca34a5dfbc-istio-token\") pod \"e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-g4ctm\" (UID: \"c985ba5a-3feb-4b17-959e-b4ca34a5dfbc\") " pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-g4ctm" Nov 04 04:58:49.686993 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:49.686968 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jrtf\" (UniqueName: \"kubernetes.io/projected/c985ba5a-3feb-4b17-959e-b4ca34a5dfbc-kube-api-access-4jrtf\") pod \"e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-g4ctm\" (UID: \"c985ba5a-3feb-4b17-959e-b4ca34a5dfbc\") " pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-g4ctm" Nov 04 04:58:49.712130 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:49.712106 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-g4ctm" Nov 04 04:58:50.115112 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:50.115079 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-1874d8-78f4d9f8b7-9d2tj" event={"ID":"e5103c9a-0929-422e-8c95-b342181f0669","Type":"ContainerStarted","Data":"0a0e3c88f305b3df3b86b01e30a32fbf0d1eb8586f55658dcdd43e85de8750fe"} Nov 04 04:58:50.115282 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:50.115133 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-1874d8-78f4d9f8b7-9d2tj" event={"ID":"e5103c9a-0929-422e-8c95-b342181f0669","Type":"ContainerStarted","Data":"1b4d6fd92b32bde69c73fb0eaaa01cc9edafcb5ca839a77dbeba64bc5748a516"} Nov 04 04:58:50.115282 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:50.115271 4509 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-1874d8-78f4d9f8b7-9d2tj" Nov 04 04:58:50.192384 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:50.192272 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-g4ctm"] Nov 04 04:58:50.195047 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:58:50.195017 4509 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc985ba5a_3feb_4b17_959e_b4ca34a5dfbc.slice/crio-5cb79bd4b679c69b25150a63b79a9964a5f4c1a5aabbb6698ee7c452658fb98a WatchSource:0}: Error finding container 5cb79bd4b679c69b25150a63b79a9964a5f4c1a5aabbb6698ee7c452658fb98a: Status 404 returned error can't find the container with id 5cb79bd4b679c69b25150a63b79a9964a5f4c1a5aabbb6698ee7c452658fb98a Nov 04 04:58:50.197984 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:50.197952 4509 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 04 04:58:50.198061 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:50.198029 4509 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 04 04:58:50.198097 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:50.198072 4509 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 04 04:58:50.240378 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:50.240316 4509 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-1874d8-78f4d9f8b7-9d2tj" podStartSLOduration=2.240304886 podStartE2EDuration="2.240304886s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 04:58:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:58:50.239625804 +0000 UTC m=+694.117349324" watchObservedRunningTime="2026-11-04 04:58:50.240304886 +0000 UTC m=+694.118028423" Nov 04 04:58:51.045468 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:51.045428 4509 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-370078-6c55687f5f-gwlxz" Nov 04 04:58:51.046028 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:51.046010 4509 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-370078-dc9467d44-l4644" Nov 04 04:58:51.103350 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:51.103012 4509 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-dab70d-695b6bf8dc-vlnmz"] Nov 04 04:58:51.113788 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:51.113750 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-dab70d-695b6bf8dc-vlnmz" Nov 04 04:58:51.123451 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:51.122858 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-g4ctm" event={"ID":"c985ba5a-3feb-4b17-959e-b4ca34a5dfbc","Type":"ContainerStarted","Data":"daaaa3814f302b7704286c9b49a77219ed4ba6ae4241d9862bce265ef64d2722"} Nov 04 04:58:51.123451 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:51.122941 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-g4ctm" event={"ID":"c985ba5a-3feb-4b17-959e-b4ca34a5dfbc","Type":"ContainerStarted","Data":"5cb79bd4b679c69b25150a63b79a9964a5f4c1a5aabbb6698ee7c452658fb98a"} Nov 04 04:58:51.152162 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:51.152141 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-dab70d-695b6bf8dc-vlnmz"] Nov 04 04:58:51.195569 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:51.195518 4509 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-dab70d\"" Nov 04 04:58:51.269975 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:51.238504 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qw5s\" (UniqueName: \"kubernetes.io/projected/3137d988-598c-4f6a-a797-a08ed4448b8a-kube-api-access-4qw5s\") pod \"payload-pre-processing-e2e-worker-w3-dab70d-695b6bf8dc-vlnmz\" (UID: \"3137d988-598c-4f6a-a797-a08ed4448b8a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-dab70d-695b6bf8dc-vlnmz" Nov 04 04:58:51.269975 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:51.238691 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3137d988-598c-4f6a-a797-a08ed4448b8a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-dab70d-695b6bf8dc-vlnmz\" (UID: \"3137d988-598c-4f6a-a797-a08ed4448b8a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-dab70d-695b6bf8dc-vlnmz" Nov 04 04:58:51.269975 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:51.253698 4509 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-dab70d-dockercfg-mlmqr\"" Nov 04 04:58:51.270415 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:51.270390 4509 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-dab70d-6ddd49994d-ggggw"] Nov 04 04:58:51.338450 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:51.338354 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-dab70d-6ddd49994d-ggggw" Nov 04 04:58:51.340491 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:51.340463 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3137d988-598c-4f6a-a797-a08ed4448b8a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-dab70d-695b6bf8dc-vlnmz\" (UID: \"3137d988-598c-4f6a-a797-a08ed4448b8a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-dab70d-695b6bf8dc-vlnmz" Nov 04 04:58:51.340706 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:51.340682 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-dab70d-6ddd49994d-ggggw"] Nov 04 04:58:51.341730 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:51.341707 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3137d988-598c-4f6a-a797-a08ed4448b8a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-dab70d-695b6bf8dc-vlnmz\" (UID: \"3137d988-598c-4f6a-a797-a08ed4448b8a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-dab70d-695b6bf8dc-vlnmz" Nov 04 04:58:51.341973 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:51.341943 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qw5s\" (UniqueName: \"kubernetes.io/projected/3137d988-598c-4f6a-a797-a08ed4448b8a-kube-api-access-4qw5s\") pod \"payload-pre-processing-e2e-worker-w3-dab70d-695b6bf8dc-vlnmz\" (UID: \"3137d988-598c-4f6a-a797-a08ed4448b8a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-dab70d-695b6bf8dc-vlnmz" Nov 04 04:58:51.427968 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:51.427932 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qw5s\" (UniqueName: \"kubernetes.io/projected/3137d988-598c-4f6a-a797-a08ed4448b8a-kube-api-access-4qw5s\") pod \"payload-pre-processing-e2e-worker-w3-dab70d-695b6bf8dc-vlnmz\" (UID: \"3137d988-598c-4f6a-a797-a08ed4448b8a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-dab70d-695b6bf8dc-vlnmz" Nov 04 04:58:51.442665 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:51.442638 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f71da8bf-e97b-4a2a-bf50-9588fbc73c48-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-dab70d-6ddd49994d-ggggw\" (UID: \"f71da8bf-e97b-4a2a-bf50-9588fbc73c48\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-dab70d-6ddd49994d-ggggw" Nov 04 04:58:51.442797 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:51.442727 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sc5v5\" (UniqueName: \"kubernetes.io/projected/f71da8bf-e97b-4a2a-bf50-9588fbc73c48-kube-api-access-sc5v5\") pod \"payload-processing-e2e-worker-w3-dab70d-6ddd49994d-ggggw\" (UID: \"f71da8bf-e97b-4a2a-bf50-9588fbc73c48\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-dab70d-6ddd49994d-ggggw" Nov 04 04:58:51.543650 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:51.543617 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sc5v5\" (UniqueName: \"kubernetes.io/projected/f71da8bf-e97b-4a2a-bf50-9588fbc73c48-kube-api-access-sc5v5\") pod \"payload-processing-e2e-worker-w3-dab70d-6ddd49994d-ggggw\" (UID: \"f71da8bf-e97b-4a2a-bf50-9588fbc73c48\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-dab70d-6ddd49994d-ggggw" Nov 04 04:58:51.543806 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:51.543715 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f71da8bf-e97b-4a2a-bf50-9588fbc73c48-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-dab70d-6ddd49994d-ggggw\" (UID: \"f71da8bf-e97b-4a2a-bf50-9588fbc73c48\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-dab70d-6ddd49994d-ggggw" Nov 04 04:58:51.544533 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:51.544506 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f71da8bf-e97b-4a2a-bf50-9588fbc73c48-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-dab70d-6ddd49994d-ggggw\" (UID: \"f71da8bf-e97b-4a2a-bf50-9588fbc73c48\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-dab70d-6ddd49994d-ggggw" Nov 04 04:58:51.590398 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:51.587544 4509 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-g4ctm" podStartSLOduration=2.5875251710000002 podStartE2EDuration="2.587525171s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 04:58:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:58:51.586543659 +0000 UTC m=+695.464267181" watchObservedRunningTime="2026-11-04 04:58:51.587525171 +0000 UTC m=+695.465248693" Nov 04 04:58:51.617221 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:51.617178 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sc5v5\" (UniqueName: \"kubernetes.io/projected/f71da8bf-e97b-4a2a-bf50-9588fbc73c48-kube-api-access-sc5v5\") pod \"payload-processing-e2e-worker-w3-dab70d-6ddd49994d-ggggw\" (UID: \"f71da8bf-e97b-4a2a-bf50-9588fbc73c48\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-dab70d-6ddd49994d-ggggw" Nov 04 04:58:51.657612 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:51.657584 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-dab70d-6ddd49994d-ggggw" Nov 04 04:58:51.714652 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:51.714479 4509 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-g4ctm" Nov 04 04:58:51.715975 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:51.715939 4509 patch_prober.go:29] interesting pod/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-g4ctm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Nov 04 04:58:51.716078 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:51.715989 4509 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-g4ctm" podUID="c985ba5a-3feb-4b17-959e-b4ca34a5dfbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Nov 04 04:58:51.727794 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:51.727618 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-dab70d-695b6bf8dc-vlnmz" Nov 04 04:58:51.956625 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:51.956452 4509 patch_prober.go:29] interesting pod/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-m6qb5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 04:58:51.956625 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:51.956501 4509 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-m6qb5" podUID="d751b557-4e07-4657-96ec-59fc0e70fd57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 04:58:52.116664 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:52.116637 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-dab70d-6ddd49994d-ggggw"] Nov 04 04:58:52.119367 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:58:52.118920 4509 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf71da8bf_e97b_4a2a_bf50_9588fbc73c48.slice/crio-3a8987080f89e9bec9f601d13ade7779170bd7fd7edfd589d851e661c4627861 WatchSource:0}: Error finding container 3a8987080f89e9bec9f601d13ade7779170bd7fd7edfd589d851e661c4627861: Status 404 returned error can't find the container with id 3a8987080f89e9bec9f601d13ade7779170bd7fd7edfd589d851e661c4627861 Nov 04 04:58:52.128642 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:52.127986 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-dab70d-6ddd49994d-ggggw" event={"ID":"f71da8bf-e97b-4a2a-bf50-9588fbc73c48","Type":"ContainerStarted","Data":"3a8987080f89e9bec9f601d13ade7779170bd7fd7edfd589d851e661c4627861"} Nov 04 04:58:52.190397 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:52.190360 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-dab70d-695b6bf8dc-vlnmz"] Nov 04 04:58:52.202616 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:58:52.202567 4509 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3137d988_598c_4f6a_a797_a08ed4448b8a.slice/crio-cbd03943f3ed8f758dd07f1e2d3496299dd4c9cfcc0f850d79014f49c42b7ad2 WatchSource:0}: Error finding container cbd03943f3ed8f758dd07f1e2d3496299dd4c9cfcc0f850d79014f49c42b7ad2: Status 404 returned error can't find the container with id cbd03943f3ed8f758dd07f1e2d3496299dd4c9cfcc0f850d79014f49c42b7ad2 Nov 04 04:58:52.712971 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:52.712930 4509 patch_prober.go:29] interesting pod/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-g4ctm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Nov 04 04:58:52.713141 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:52.713001 4509 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-g4ctm" podUID="c985ba5a-3feb-4b17-959e-b4ca34a5dfbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Nov 04 04:58:53.134675 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:53.134310 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-dab70d-6ddd49994d-ggggw" event={"ID":"f71da8bf-e97b-4a2a-bf50-9588fbc73c48","Type":"ContainerStarted","Data":"885724b47455392fd91c42875e22e92c1b22b7bc45ba141b3edf3876a0f8adf7"} Nov 04 04:58:53.134675 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:53.134548 4509 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-dab70d-6ddd49994d-ggggw" Nov 04 04:58:53.135655 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:53.135636 4509 generic.go:356] "Generic (PLEG): container finished" podID="d751b557-4e07-4657-96ec-59fc0e70fd57" containerID="90901cf35b3a5c5999f8076892b42910a049c313e90fe487980febe641e77b6b" exitCode=0 Nov 04 04:58:53.136363 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:53.136345 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-m6qb5" event={"ID":"d751b557-4e07-4657-96ec-59fc0e70fd57","Type":"ContainerDied","Data":"90901cf35b3a5c5999f8076892b42910a049c313e90fe487980febe641e77b6b"} Nov 04 04:58:53.136667 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:53.136654 4509 scope.go:122] "RemoveContainer" containerID="90901cf35b3a5c5999f8076892b42910a049c313e90fe487980febe641e77b6b" Nov 04 04:58:53.137460 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:53.137443 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-dab70d-695b6bf8dc-vlnmz" event={"ID":"3137d988-598c-4f6a-a797-a08ed4448b8a","Type":"ContainerStarted","Data":"410c6a472fbecd80e92b65c59f31cd3a6b7309130c7eac15ff8712ddd723a8fd"} Nov 04 04:58:53.137499 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:53.137475 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-dab70d-695b6bf8dc-vlnmz" event={"ID":"3137d988-598c-4f6a-a797-a08ed4448b8a","Type":"ContainerStarted","Data":"cbd03943f3ed8f758dd07f1e2d3496299dd4c9cfcc0f850d79014f49c42b7ad2"} Nov 04 04:58:53.137530 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:53.137518 4509 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-dab70d-695b6bf8dc-vlnmz" Nov 04 04:58:53.138371 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:53.138346 4509 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 04 04:58:53.138422 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:53.138410 4509 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 04 04:58:53.138454 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:53.138443 4509 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 04 04:58:53.255249 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:53.255204 4509 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-dab70d-6ddd49994d-ggggw" podStartSLOduration=2.255190445 podStartE2EDuration="2.255190445s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 04:58:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:58:53.253402416 +0000 UTC m=+697.131125936" watchObservedRunningTime="2026-11-04 04:58:53.255190445 +0000 UTC m=+697.132913966" Nov 04 04:58:53.531112 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:53.530936 4509 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-dab70d-695b6bf8dc-vlnmz" podStartSLOduration=2.530918047 podStartE2EDuration="2.530918047s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 04:58:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:58:53.530698107 +0000 UTC m=+697.408421629" watchObservedRunningTime="2026-11-04 04:58:53.530918047 +0000 UTC m=+697.408641570" Nov 04 04:58:53.712969 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:53.712541 4509 patch_prober.go:29] interesting pod/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-g4ctm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Nov 04 04:58:53.712969 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:53.712599 4509 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-g4ctm" podUID="c985ba5a-3feb-4b17-959e-b4ca34a5dfbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Nov 04 04:58:54.144341 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:54.144290 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-m6qb5" event={"ID":"d751b557-4e07-4657-96ec-59fc0e70fd57","Type":"ContainerStarted","Data":"33466972d2f1df00a526742b7a18372fa8aa2b7b68e9a3ccab257f6ed7aa65ca"} Nov 04 04:58:54.714137 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:54.714102 4509 patch_prober.go:29] interesting pod/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-g4ctm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Nov 04 04:58:54.714470 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:54.714440 4509 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-g4ctm" podUID="c985ba5a-3feb-4b17-959e-b4ca34a5dfbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Nov 04 04:58:55.036361 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:55.036154 4509 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-m6qb5" Nov 04 04:58:55.036518 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:55.036379 4509 patch_prober.go:29] interesting pod/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-m6qb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Nov 04 04:58:55.036518 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:55.036433 4509 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-m6qb5" podUID="d751b557-4e07-4657-96ec-59fc0e70fd57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Nov 04 04:58:55.713127 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:55.713077 4509 patch_prober.go:29] interesting pod/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-g4ctm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Nov 04 04:58:55.713672 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:55.713144 4509 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-g4ctm" podUID="c985ba5a-3feb-4b17-959e-b4ca34a5dfbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Nov 04 04:58:56.036606 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:56.036525 4509 patch_prober.go:29] interesting pod/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-m6qb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Nov 04 04:58:56.036606 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:56.036578 4509 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-m6qb5" podUID="d751b557-4e07-4657-96ec-59fc0e70fd57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Nov 04 04:58:56.717961 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:56.717927 4509 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-g4ctm" Nov 04 04:58:56.718457 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:56.718233 4509 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-g4ctm" Nov 04 04:58:56.719385 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:56.719365 4509 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-g4ctm" Nov 04 04:58:57.037012 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:57.036939 4509 patch_prober.go:29] interesting pod/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-m6qb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Nov 04 04:58:57.037012 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:57.036989 4509 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-m6qb5" podUID="d751b557-4e07-4657-96ec-59fc0e70fd57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Nov 04 04:58:58.036792 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:58.036757 4509 patch_prober.go:29] interesting pod/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-m6qb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Nov 04 04:58:58.037180 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:58.036822 4509 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-m6qb5" podUID="d751b557-4e07-4657-96ec-59fc0e70fd57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Nov 04 04:58:58.106005 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:58.105968 4509 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-0aae2f-5ff7778f8c-4tbhr" Nov 04 04:58:58.106170 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:58.106033 4509 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-0aae2f-74664f8f76-rzd5v" Nov 04 04:58:59.037317 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:59.037240 4509 patch_prober.go:29] interesting pod/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-m6qb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Nov 04 04:58:59.037714 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:59.037314 4509 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-m6qb5" podUID="d751b557-4e07-4657-96ec-59fc0e70fd57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Nov 04 04:58:59.199019 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:59.198940 4509 generic.go:356] "Generic (PLEG): container finished" podID="67ec6f66-3a36-4d01-be83-4ea5d42ba52e" containerID="38fd3e3b00aad944d8c094dcef7e8eb6d1d7b4f72f2be1288d1c49e77bcd6097" exitCode=0 Nov 04 04:58:59.199019 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:59.198975 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-znf89" event={"ID":"67ec6f66-3a36-4d01-be83-4ea5d42ba52e","Type":"ContainerDied","Data":"38fd3e3b00aad944d8c094dcef7e8eb6d1d7b4f72f2be1288d1c49e77bcd6097"} Nov 04 04:58:59.199370 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:59.199352 4509 scope.go:122] "RemoveContainer" containerID="38fd3e3b00aad944d8c094dcef7e8eb6d1d7b4f72f2be1288d1c49e77bcd6097" Nov 04 04:58:59.200600 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:59.200571 4509 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 04 04:58:59.200686 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:59.200636 4509 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 04 04:58:59.200686 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:58:59.200665 4509 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 04 04:59:00.039702 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:59:00.039678 4509 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-m6qb5" Nov 04 04:59:00.040095 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:59:00.039967 4509 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-m6qb5" Nov 04 04:59:00.040665 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:59:00.040649 4509 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-m6qb5" Nov 04 04:59:00.206677 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:59:00.206642 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-znf89" event={"ID":"67ec6f66-3a36-4d01-be83-4ea5d42ba52e","Type":"ContainerStarted","Data":"2e57e7a19e5469496225664baefc6298927614f7fc55b23797a2a3c0daba3706"} Nov 04 04:59:00.391718 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:59:00.391617 4509 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-znf89" Nov 04 04:59:00.391892 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:59:00.391843 4509 patch_prober.go:29] interesting pod/e2e-worker-w5-4cce33-openshift-default-747768654f-znf89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Nov 04 04:59:00.391969 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:59:00.391908 4509 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-znf89" podUID="67ec6f66-3a36-4d01-be83-4ea5d42ba52e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Nov 04 04:59:01.126401 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:59:01.126374 4509 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-1874d8-78f4d9f8b7-9d2tj" Nov 04 04:59:01.392387 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:59:01.392291 4509 patch_prober.go:29] interesting pod/e2e-worker-w5-4cce33-openshift-default-747768654f-znf89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Nov 04 04:59:01.392545 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:59:01.392404 4509 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-znf89" podUID="67ec6f66-3a36-4d01-be83-4ea5d42ba52e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Nov 04 04:59:01.786968 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:59:01.786939 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-42dnm"] Nov 04 04:59:02.395969 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:59:02.395931 4509 patch_prober.go:29] interesting pod/e2e-worker-w5-4cce33-openshift-default-747768654f-znf89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Nov 04 04:59:02.396773 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:59:02.396736 4509 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-znf89" podUID="67ec6f66-3a36-4d01-be83-4ea5d42ba52e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Nov 04 04:59:03.392483 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:59:03.392434 4509 patch_prober.go:29] interesting pod/e2e-worker-w5-4cce33-openshift-default-747768654f-znf89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Nov 04 04:59:03.392650 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:59:03.392499 4509 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-znf89" podUID="67ec6f66-3a36-4d01-be83-4ea5d42ba52e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Nov 04 04:59:04.146826 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:59:04.146798 4509 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-dab70d-6ddd49994d-ggggw" Nov 04 04:59:04.392010 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:59:04.391975 4509 patch_prober.go:29] interesting pod/e2e-worker-w5-4cce33-openshift-default-747768654f-znf89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Nov 04 04:59:04.392205 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:59:04.392039 4509 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-znf89" podUID="67ec6f66-3a36-4d01-be83-4ea5d42ba52e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Nov 04 04:59:05.158524 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:59:05.158495 4509 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-dab70d-695b6bf8dc-vlnmz" Nov 04 04:59:05.397221 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:59:05.397190 4509 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-znf89" Nov 04 04:59:05.397590 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:59:05.397568 4509 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-znf89" Nov 04 04:59:05.398206 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:59:05.398187 4509 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-znf89" Nov 04 04:59:14.455051 ip-10-0-133-180 kubenswrapper[4509]: E1104 04:59:14.455015 4509 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmdhl" containerName="extractor" Nov 04 04:59:21.763354 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:59:21.747276 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-42dnm"] Nov 04 04:59:32.429967 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:59:32.429932 4509 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-cedfca/simulator-w6-cedfca-kserve-687878f579-n9vnt"] Nov 04 04:59:32.433789 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:59:32.433771 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-cedfca/simulator-w6-cedfca-kserve-687878f579-n9vnt" Nov 04 04:59:32.490148 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:59:32.490120 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-cedfca/simulator-w6-cedfca-kserve-687878f579-n9vnt"] Nov 04 04:59:32.493830 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:59:32.493803 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cea7867b-14f2-4967-8467-1a4d4e698f10-model-cache\") pod \"simulator-w6-cedfca-kserve-687878f579-n9vnt\" (UID: \"cea7867b-14f2-4967-8467-1a4d4e698f10\") " pod="e2e-models-e2e-worker-w6-cedfca/simulator-w6-cedfca-kserve-687878f579-n9vnt" Nov 04 04:59:32.493948 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:59:32.493879 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cea7867b-14f2-4967-8467-1a4d4e698f10-dshm\") pod \"simulator-w6-cedfca-kserve-687878f579-n9vnt\" (UID: \"cea7867b-14f2-4967-8467-1a4d4e698f10\") " pod="e2e-models-e2e-worker-w6-cedfca/simulator-w6-cedfca-kserve-687878f579-n9vnt" Nov 04 04:59:32.493948 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:59:32.493927 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcgnk\" (UniqueName: \"kubernetes.io/projected/cea7867b-14f2-4967-8467-1a4d4e698f10-kube-api-access-pcgnk\") pod \"simulator-w6-cedfca-kserve-687878f579-n9vnt\" (UID: \"cea7867b-14f2-4967-8467-1a4d4e698f10\") " pod="e2e-models-e2e-worker-w6-cedfca/simulator-w6-cedfca-kserve-687878f579-n9vnt" Nov 04 04:59:32.494061 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:59:32.493997 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cea7867b-14f2-4967-8467-1a4d4e698f10-home\") pod \"simulator-w6-cedfca-kserve-687878f579-n9vnt\" (UID: \"cea7867b-14f2-4967-8467-1a4d4e698f10\") " pod="e2e-models-e2e-worker-w6-cedfca/simulator-w6-cedfca-kserve-687878f579-n9vnt" Nov 04 04:59:32.494123 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:59:32.494052 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cea7867b-14f2-4967-8467-1a4d4e698f10-tls-certs\") pod \"simulator-w6-cedfca-kserve-687878f579-n9vnt\" (UID: \"cea7867b-14f2-4967-8467-1a4d4e698f10\") " pod="e2e-models-e2e-worker-w6-cedfca/simulator-w6-cedfca-kserve-687878f579-n9vnt" Nov 04 04:59:32.494123 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:59:32.494104 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cea7867b-14f2-4967-8467-1a4d4e698f10-tmp-dir\") pod \"simulator-w6-cedfca-kserve-687878f579-n9vnt\" (UID: \"cea7867b-14f2-4967-8467-1a4d4e698f10\") " pod="e2e-models-e2e-worker-w6-cedfca/simulator-w6-cedfca-kserve-687878f579-n9vnt" Nov 04 04:59:32.537095 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:59:32.537075 4509 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-cedfca\"/\"kube-root-ca.crt\"" Nov 04 04:59:32.537271 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:59:32.537252 4509 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-cedfca\"/\"openshift-service-ca.crt\"" Nov 04 04:59:32.537358 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:59:32.537343 4509 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-cedfca\"/\"default-dockercfg-f7vjm\"" Nov 04 04:59:32.537460 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:59:32.537444 4509 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-cedfca\"/\"simulator-w6-cedfca-kserve-self-signed-certs\"" Nov 04 04:59:32.595450 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:59:32.595416 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cea7867b-14f2-4967-8467-1a4d4e698f10-model-cache\") pod \"simulator-w6-cedfca-kserve-687878f579-n9vnt\" (UID: \"cea7867b-14f2-4967-8467-1a4d4e698f10\") " pod="e2e-models-e2e-worker-w6-cedfca/simulator-w6-cedfca-kserve-687878f579-n9vnt" Nov 04 04:59:32.595650 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:59:32.595493 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cea7867b-14f2-4967-8467-1a4d4e698f10-dshm\") pod \"simulator-w6-cedfca-kserve-687878f579-n9vnt\" (UID: \"cea7867b-14f2-4967-8467-1a4d4e698f10\") " pod="e2e-models-e2e-worker-w6-cedfca/simulator-w6-cedfca-kserve-687878f579-n9vnt" Nov 04 04:59:32.595650 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:59:32.595537 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcgnk\" (UniqueName: \"kubernetes.io/projected/cea7867b-14f2-4967-8467-1a4d4e698f10-kube-api-access-pcgnk\") pod \"simulator-w6-cedfca-kserve-687878f579-n9vnt\" (UID: \"cea7867b-14f2-4967-8467-1a4d4e698f10\") " pod="e2e-models-e2e-worker-w6-cedfca/simulator-w6-cedfca-kserve-687878f579-n9vnt" Nov 04 04:59:32.595650 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:59:32.595564 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cea7867b-14f2-4967-8467-1a4d4e698f10-home\") pod \"simulator-w6-cedfca-kserve-687878f579-n9vnt\" (UID: \"cea7867b-14f2-4967-8467-1a4d4e698f10\") " pod="e2e-models-e2e-worker-w6-cedfca/simulator-w6-cedfca-kserve-687878f579-n9vnt" Nov 04 04:59:32.595650 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:59:32.595601 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cea7867b-14f2-4967-8467-1a4d4e698f10-tls-certs\") pod \"simulator-w6-cedfca-kserve-687878f579-n9vnt\" (UID: \"cea7867b-14f2-4967-8467-1a4d4e698f10\") " pod="e2e-models-e2e-worker-w6-cedfca/simulator-w6-cedfca-kserve-687878f579-n9vnt" Nov 04 04:59:32.595650 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:59:32.595638 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cea7867b-14f2-4967-8467-1a4d4e698f10-tmp-dir\") pod \"simulator-w6-cedfca-kserve-687878f579-n9vnt\" (UID: \"cea7867b-14f2-4967-8467-1a4d4e698f10\") " pod="e2e-models-e2e-worker-w6-cedfca/simulator-w6-cedfca-kserve-687878f579-n9vnt" Nov 04 04:59:32.596054 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:59:32.596033 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cea7867b-14f2-4967-8467-1a4d4e698f10-tmp-dir\") pod \"simulator-w6-cedfca-kserve-687878f579-n9vnt\" (UID: \"cea7867b-14f2-4967-8467-1a4d4e698f10\") " pod="e2e-models-e2e-worker-w6-cedfca/simulator-w6-cedfca-kserve-687878f579-n9vnt" Nov 04 04:59:32.596275 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:59:32.596247 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cea7867b-14f2-4967-8467-1a4d4e698f10-model-cache\") pod \"simulator-w6-cedfca-kserve-687878f579-n9vnt\" (UID: \"cea7867b-14f2-4967-8467-1a4d4e698f10\") " pod="e2e-models-e2e-worker-w6-cedfca/simulator-w6-cedfca-kserve-687878f579-n9vnt" Nov 04 04:59:32.596690 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:59:32.596662 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cea7867b-14f2-4967-8467-1a4d4e698f10-home\") pod \"simulator-w6-cedfca-kserve-687878f579-n9vnt\" (UID: \"cea7867b-14f2-4967-8467-1a4d4e698f10\") " pod="e2e-models-e2e-worker-w6-cedfca/simulator-w6-cedfca-kserve-687878f579-n9vnt" Nov 04 04:59:32.598933 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:59:32.598913 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cea7867b-14f2-4967-8467-1a4d4e698f10-dshm\") pod \"simulator-w6-cedfca-kserve-687878f579-n9vnt\" (UID: \"cea7867b-14f2-4967-8467-1a4d4e698f10\") " pod="e2e-models-e2e-worker-w6-cedfca/simulator-w6-cedfca-kserve-687878f579-n9vnt" Nov 04 04:59:32.600492 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:59:32.600465 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cea7867b-14f2-4967-8467-1a4d4e698f10-tls-certs\") pod \"simulator-w6-cedfca-kserve-687878f579-n9vnt\" (UID: \"cea7867b-14f2-4967-8467-1a4d4e698f10\") " pod="e2e-models-e2e-worker-w6-cedfca/simulator-w6-cedfca-kserve-687878f579-n9vnt" Nov 04 04:59:32.653528 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:59:32.653506 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcgnk\" (UniqueName: \"kubernetes.io/projected/cea7867b-14f2-4967-8467-1a4d4e698f10-kube-api-access-pcgnk\") pod \"simulator-w6-cedfca-kserve-687878f579-n9vnt\" (UID: \"cea7867b-14f2-4967-8467-1a4d4e698f10\") " pod="e2e-models-e2e-worker-w6-cedfca/simulator-w6-cedfca-kserve-687878f579-n9vnt" Nov 04 04:59:32.744353 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:59:32.744316 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-cedfca/simulator-w6-cedfca-kserve-687878f579-n9vnt" Nov 04 04:59:33.114537 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:59:33.114515 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-cedfca/simulator-w6-cedfca-kserve-687878f579-n9vnt"] Nov 04 04:59:33.116073 ip-10-0-133-180 kubenswrapper[4509]: W1104 04:59:33.116046 4509 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcea7867b_14f2_4967_8467_1a4d4e698f10.slice/crio-f598d0a1ef07db779fed61ce3592fb7303607661f6b7a0342094df910f60d8dc WatchSource:0}: Error finding container f598d0a1ef07db779fed61ce3592fb7303607661f6b7a0342094df910f60d8dc: Status 404 returned error can't find the container with id f598d0a1ef07db779fed61ce3592fb7303607661f6b7a0342094df910f60d8dc Nov 04 04:59:33.378608 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:59:33.378533 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-cedfca/simulator-w6-cedfca-kserve-687878f579-n9vnt" event={"ID":"cea7867b-14f2-4967-8467-1a4d4e698f10","Type":"ContainerStarted","Data":"7f44fc3a1b6b633d1d9773b9a6c0bf26610cb463720e7305cd959a53fe32a118"} Nov 04 04:59:33.378608 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:59:33.378570 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-cedfca/simulator-w6-cedfca-kserve-687878f579-n9vnt" event={"ID":"cea7867b-14f2-4967-8467-1a4d4e698f10","Type":"ContainerStarted","Data":"f598d0a1ef07db779fed61ce3592fb7303607661f6b7a0342094df910f60d8dc"} Nov 04 04:59:33.493273 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:59:33.493230 4509 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-cedfca/simulator-w6-cedfca-kserve-687878f579-n9vnt" podStartSLOduration=1.49321699 podStartE2EDuration="1.49321699s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 04:59:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:59:33.492822446 +0000 UTC m=+737.370545965" watchObservedRunningTime="2026-11-04 04:59:33.49321699 +0000 UTC m=+737.370940509" Nov 04 04:59:33.745286 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:59:33.745257 4509 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-cedfca/simulator-w6-cedfca-kserve-687878f579-n9vnt" Nov 04 04:59:38.232143 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:59:38.232110 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-42dnm"] Nov 04 04:59:42.744865 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:59:42.744830 4509 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-cedfca/simulator-w6-cedfca-kserve-687878f579-n9vnt" Nov 04 04:59:42.757384 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:59:42.757364 4509 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-cedfca/simulator-w6-cedfca-kserve-687878f579-n9vnt" Nov 04 04:59:43.439128 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:59:43.439100 4509 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-cedfca/simulator-w6-cedfca-kserve-687878f579-n9vnt" Nov 04 04:59:44.455256 ip-10-0-133-180 kubenswrapper[4509]: E1104 04:59:44.455222 4509 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmdhl" containerName="extractor" Nov 04 04:59:57.247442 ip-10-0-133-180 kubenswrapper[4509]: I1104 04:59:57.247351 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-42dnm"] Nov 04 05:00:00.080587 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:00.080523 4509 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-3eb31c/premium-w4-3eb31c-kserve-58bc6d6c55-kh2c4"] Nov 04 05:00:00.085510 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:00.085492 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3eb31c/premium-w4-3eb31c-kserve-58bc6d6c55-kh2c4" Nov 04 05:00:00.098308 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:00.098285 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9181aed8-6d05-4691-be54-f26391920734-dshm\") pod \"premium-w4-3eb31c-kserve-58bc6d6c55-kh2c4\" (UID: \"9181aed8-6d05-4691-be54-f26391920734\") " pod="e2e-models-e2e-worker-w4-3eb31c/premium-w4-3eb31c-kserve-58bc6d6c55-kh2c4" Nov 04 05:00:00.098412 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:00.098390 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9181aed8-6d05-4691-be54-f26391920734-model-cache\") pod \"premium-w4-3eb31c-kserve-58bc6d6c55-kh2c4\" (UID: \"9181aed8-6d05-4691-be54-f26391920734\") " pod="e2e-models-e2e-worker-w4-3eb31c/premium-w4-3eb31c-kserve-58bc6d6c55-kh2c4" Nov 04 05:00:00.098494 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:00.098475 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9181aed8-6d05-4691-be54-f26391920734-home\") pod \"premium-w4-3eb31c-kserve-58bc6d6c55-kh2c4\" (UID: \"9181aed8-6d05-4691-be54-f26391920734\") " pod="e2e-models-e2e-worker-w4-3eb31c/premium-w4-3eb31c-kserve-58bc6d6c55-kh2c4" Nov 04 05:00:00.098528 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:00.098507 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9181aed8-6d05-4691-be54-f26391920734-tmp-dir\") pod \"premium-w4-3eb31c-kserve-58bc6d6c55-kh2c4\" (UID: \"9181aed8-6d05-4691-be54-f26391920734\") " pod="e2e-models-e2e-worker-w4-3eb31c/premium-w4-3eb31c-kserve-58bc6d6c55-kh2c4" Nov 04 05:00:00.098575 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:00.098563 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5m6t\" (UniqueName: \"kubernetes.io/projected/9181aed8-6d05-4691-be54-f26391920734-kube-api-access-n5m6t\") pod \"premium-w4-3eb31c-kserve-58bc6d6c55-kh2c4\" (UID: \"9181aed8-6d05-4691-be54-f26391920734\") " pod="e2e-models-e2e-worker-w4-3eb31c/premium-w4-3eb31c-kserve-58bc6d6c55-kh2c4" Nov 04 05:00:00.098613 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:00.098601 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9181aed8-6d05-4691-be54-f26391920734-tls-certs\") pod \"premium-w4-3eb31c-kserve-58bc6d6c55-kh2c4\" (UID: \"9181aed8-6d05-4691-be54-f26391920734\") " pod="e2e-models-e2e-worker-w4-3eb31c/premium-w4-3eb31c-kserve-58bc6d6c55-kh2c4" Nov 04 05:00:00.140852 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:00.140831 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-3eb31c/premium-w4-3eb31c-kserve-58bc6d6c55-kh2c4"] Nov 04 05:00:00.188124 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:00.188045 4509 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-3eb31c\"/\"openshift-service-ca.crt\"" Nov 04 05:00:00.188286 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:00.188099 4509 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-3eb31c\"/\"kube-root-ca.crt\"" Nov 04 05:00:00.188404 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:00.188277 4509 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-3eb31c\"/\"premium-w4-3eb31c-kserve-self-signed-certs\"" Nov 04 05:00:00.188492 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:00.188474 4509 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-3eb31c\"/\"default-dockercfg-pb6qn\"" Nov 04 05:00:00.193792 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:00.193769 4509 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-cedfca-29896140-k8xk6"] Nov 04 05:00:00.198043 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:00.198024 4509 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1874d8-29896140-jrzlz"] Nov 04 05:00:00.198155 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:00.198141 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-cedfca-29896140-k8xk6" Nov 04 05:00:00.199082 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:00.199060 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9181aed8-6d05-4691-be54-f26391920734-model-cache\") pod \"premium-w4-3eb31c-kserve-58bc6d6c55-kh2c4\" (UID: \"9181aed8-6d05-4691-be54-f26391920734\") " pod="e2e-models-e2e-worker-w4-3eb31c/premium-w4-3eb31c-kserve-58bc6d6c55-kh2c4" Nov 04 05:00:00.199184 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:00.199100 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4baf9fe5-84d1-48a1-8c10-8080a1e6a5db-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-cedfca-29896140-k8xk6\" (UID: \"4baf9fe5-84d1-48a1-8c10-8080a1e6a5db\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-cedfca-29896140-k8xk6" Nov 04 05:00:00.199184 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:00.199151 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9181aed8-6d05-4691-be54-f26391920734-home\") pod \"premium-w4-3eb31c-kserve-58bc6d6c55-kh2c4\" (UID: \"9181aed8-6d05-4691-be54-f26391920734\") " pod="e2e-models-e2e-worker-w4-3eb31c/premium-w4-3eb31c-kserve-58bc6d6c55-kh2c4" Nov 04 05:00:00.199294 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:00.199183 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9181aed8-6d05-4691-be54-f26391920734-tmp-dir\") pod \"premium-w4-3eb31c-kserve-58bc6d6c55-kh2c4\" (UID: \"9181aed8-6d05-4691-be54-f26391920734\") " pod="e2e-models-e2e-worker-w4-3eb31c/premium-w4-3eb31c-kserve-58bc6d6c55-kh2c4" Nov 04 05:00:00.199294 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:00.199270 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5m6t\" (UniqueName: \"kubernetes.io/projected/9181aed8-6d05-4691-be54-f26391920734-kube-api-access-n5m6t\") pod \"premium-w4-3eb31c-kserve-58bc6d6c55-kh2c4\" (UID: \"9181aed8-6d05-4691-be54-f26391920734\") " pod="e2e-models-e2e-worker-w4-3eb31c/premium-w4-3eb31c-kserve-58bc6d6c55-kh2c4" Nov 04 05:00:00.199412 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:00.199346 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9181aed8-6d05-4691-be54-f26391920734-tls-certs\") pod \"premium-w4-3eb31c-kserve-58bc6d6c55-kh2c4\" (UID: \"9181aed8-6d05-4691-be54-f26391920734\") " pod="e2e-models-e2e-worker-w4-3eb31c/premium-w4-3eb31c-kserve-58bc6d6c55-kh2c4" Nov 04 05:00:00.199464 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:00.199419 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fjfh\" (UniqueName: \"kubernetes.io/projected/4baf9fe5-84d1-48a1-8c10-8080a1e6a5db-kube-api-access-2fjfh\") pod \"maas-api-key-cleanup-e2e-worker-w6-cedfca-29896140-k8xk6\" (UID: \"4baf9fe5-84d1-48a1-8c10-8080a1e6a5db\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-cedfca-29896140-k8xk6" Nov 04 05:00:00.199464 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:00.199453 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9181aed8-6d05-4691-be54-f26391920734-dshm\") pod \"premium-w4-3eb31c-kserve-58bc6d6c55-kh2c4\" (UID: \"9181aed8-6d05-4691-be54-f26391920734\") " pod="e2e-models-e2e-worker-w4-3eb31c/premium-w4-3eb31c-kserve-58bc6d6c55-kh2c4" Nov 04 05:00:00.199561 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:00.199421 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9181aed8-6d05-4691-be54-f26391920734-model-cache\") pod \"premium-w4-3eb31c-kserve-58bc6d6c55-kh2c4\" (UID: \"9181aed8-6d05-4691-be54-f26391920734\") " pod="e2e-models-e2e-worker-w4-3eb31c/premium-w4-3eb31c-kserve-58bc6d6c55-kh2c4" Nov 04 05:00:00.199561 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:00.199539 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9181aed8-6d05-4691-be54-f26391920734-home\") pod \"premium-w4-3eb31c-kserve-58bc6d6c55-kh2c4\" (UID: \"9181aed8-6d05-4691-be54-f26391920734\") " pod="e2e-models-e2e-worker-w4-3eb31c/premium-w4-3eb31c-kserve-58bc6d6c55-kh2c4" Nov 04 05:00:00.199854 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:00.199833 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9181aed8-6d05-4691-be54-f26391920734-tmp-dir\") pod \"premium-w4-3eb31c-kserve-58bc6d6c55-kh2c4\" (UID: \"9181aed8-6d05-4691-be54-f26391920734\") " pod="e2e-models-e2e-worker-w4-3eb31c/premium-w4-3eb31c-kserve-58bc6d6c55-kh2c4" Nov 04 05:00:00.201833 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:00.201810 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9181aed8-6d05-4691-be54-f26391920734-tls-certs\") pod \"premium-w4-3eb31c-kserve-58bc6d6c55-kh2c4\" (UID: \"9181aed8-6d05-4691-be54-f26391920734\") " pod="e2e-models-e2e-worker-w4-3eb31c/premium-w4-3eb31c-kserve-58bc6d6c55-kh2c4" Nov 04 05:00:00.201911 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:00.201891 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9181aed8-6d05-4691-be54-f26391920734-dshm\") pod \"premium-w4-3eb31c-kserve-58bc6d6c55-kh2c4\" (UID: \"9181aed8-6d05-4691-be54-f26391920734\") " pod="e2e-models-e2e-worker-w4-3eb31c/premium-w4-3eb31c-kserve-58bc6d6c55-kh2c4" Nov 04 05:00:00.202385 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:00.202367 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1874d8-29896140-jrzlz" Nov 04 05:00:00.240655 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:00.240635 4509 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-370078-29896140-9tcm2"] Nov 04 05:00:00.243956 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:00.243939 4509 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29896140-kkj8l"] Nov 04 05:00:00.244066 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:00.244051 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-370078-29896140-9tcm2" Nov 04 05:00:00.247166 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:00.247151 4509 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dab70d-29896140-w2fbg"] Nov 04 05:00:00.247283 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:00.247267 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29896140-kkj8l" Nov 04 05:00:00.250747 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:00.250728 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dab70d-29896140-w2fbg" Nov 04 05:00:00.254031 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:00.254012 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-cedfca-29896140-k8xk6"] Nov 04 05:00:00.296139 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:00.290141 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1874d8-29896140-jrzlz"] Nov 04 05:00:00.296139 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:00.292424 4509 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 05:00:00.296139 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:00.292587 4509 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 05:00:00.347791 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:00.336995 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5m6t\" (UniqueName: \"kubernetes.io/projected/9181aed8-6d05-4691-be54-f26391920734-kube-api-access-n5m6t\") pod \"premium-w4-3eb31c-kserve-58bc6d6c55-kh2c4\" (UID: \"9181aed8-6d05-4691-be54-f26391920734\") " pod="e2e-models-e2e-worker-w4-3eb31c/premium-w4-3eb31c-kserve-58bc6d6c55-kh2c4" Nov 04 05:00:00.347791 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:00.342491 4509 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-krgvj\"" Nov 04 05:00:00.348418 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:00.348390 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dab70d-29896140-w2fbg"] Nov 04 05:00:00.352102 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:00.352062 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fjfh\" (UniqueName: \"kubernetes.io/projected/4baf9fe5-84d1-48a1-8c10-8080a1e6a5db-kube-api-access-2fjfh\") pod \"maas-api-key-cleanup-e2e-worker-w6-cedfca-29896140-k8xk6\" (UID: \"4baf9fe5-84d1-48a1-8c10-8080a1e6a5db\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-cedfca-29896140-k8xk6" Nov 04 05:00:00.352598 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:00.352567 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4baf9fe5-84d1-48a1-8c10-8080a1e6a5db-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-cedfca-29896140-k8xk6\" (UID: \"4baf9fe5-84d1-48a1-8c10-8080a1e6a5db\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-cedfca-29896140-k8xk6" Nov 04 05:00:00.353312 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:00.353287 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29896140-kkj8l"] Nov 04 05:00:00.353464 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:00.353294 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4baf9fe5-84d1-48a1-8c10-8080a1e6a5db-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-cedfca-29896140-k8xk6\" (UID: \"4baf9fe5-84d1-48a1-8c10-8080a1e6a5db\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-cedfca-29896140-k8xk6" Nov 04 05:00:00.356361 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:00.356341 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-370078-29896140-9tcm2"] Nov 04 05:00:00.396862 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:00.396842 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3eb31c/premium-w4-3eb31c-kserve-58bc6d6c55-kh2c4" Nov 04 05:00:00.410703 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:00.410677 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fjfh\" (UniqueName: \"kubernetes.io/projected/4baf9fe5-84d1-48a1-8c10-8080a1e6a5db-kube-api-access-2fjfh\") pod \"maas-api-key-cleanup-e2e-worker-w6-cedfca-29896140-k8xk6\" (UID: \"4baf9fe5-84d1-48a1-8c10-8080a1e6a5db\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-cedfca-29896140-k8xk6" Nov 04 05:00:00.453432 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:00.453401 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/467298da-84d2-45fa-bf41-c09422256efa-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-dab70d-29896140-w2fbg\" (UID: \"467298da-84d2-45fa-bf41-c09422256efa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dab70d-29896140-w2fbg" Nov 04 05:00:00.453555 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:00.453533 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6tbv\" (UniqueName: \"kubernetes.io/projected/7114f0d1-b384-4bca-b342-29688c0ec8a5-kube-api-access-r6tbv\") pod \"maas-api-key-cleanup-e2e-worker-w2-370078-29896140-9tcm2\" (UID: \"7114f0d1-b384-4bca-b342-29688c0ec8a5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-370078-29896140-9tcm2" Nov 04 05:00:00.453623 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:00.453565 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7114f0d1-b384-4bca-b342-29688c0ec8a5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-370078-29896140-9tcm2\" (UID: \"7114f0d1-b384-4bca-b342-29688c0ec8a5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-370078-29896140-9tcm2" Nov 04 05:00:00.453623 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:00.453611 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/941eb4d4-9d72-412b-b5ac-1bfbd65b439c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-1874d8-29896140-jrzlz\" (UID: \"941eb4d4-9d72-412b-b5ac-1bfbd65b439c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1874d8-29896140-jrzlz" Nov 04 05:00:00.453719 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:00.453697 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4n9xv\" (UniqueName: \"kubernetes.io/projected/467298da-84d2-45fa-bf41-c09422256efa-kube-api-access-4n9xv\") pod \"maas-api-key-cleanup-e2e-worker-w3-dab70d-29896140-w2fbg\" (UID: \"467298da-84d2-45fa-bf41-c09422256efa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dab70d-29896140-w2fbg" Nov 04 05:00:00.453883 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:00.453861 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdcgh\" (UniqueName: \"kubernetes.io/projected/7124f27c-88a3-4467-8793-cf58ae5c06d2-kube-api-access-sdcgh\") pod \"maas-api-key-cleanup-29896140-kkj8l\" (UID: \"7124f27c-88a3-4467-8793-cf58ae5c06d2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29896140-kkj8l" Nov 04 05:00:00.453956 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:00.453925 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7124f27c-88a3-4467-8793-cf58ae5c06d2-maas-api-service-ca\") pod \"maas-api-key-cleanup-29896140-kkj8l\" (UID: \"7124f27c-88a3-4467-8793-cf58ae5c06d2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29896140-kkj8l" Nov 04 05:00:00.454012 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:00.453986 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrlgk\" (UniqueName: \"kubernetes.io/projected/941eb4d4-9d72-412b-b5ac-1bfbd65b439c-kube-api-access-mrlgk\") pod \"maas-api-key-cleanup-e2e-worker-w1-1874d8-29896140-jrzlz\" (UID: \"941eb4d4-9d72-412b-b5ac-1bfbd65b439c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1874d8-29896140-jrzlz" Nov 04 05:00:00.523415 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:00.523251 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-cedfca-29896140-k8xk6" Nov 04 05:00:00.554687 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:00.554657 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7114f0d1-b384-4bca-b342-29688c0ec8a5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-370078-29896140-9tcm2\" (UID: \"7114f0d1-b384-4bca-b342-29688c0ec8a5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-370078-29896140-9tcm2" Nov 04 05:00:00.554819 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:00.554719 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/941eb4d4-9d72-412b-b5ac-1bfbd65b439c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-1874d8-29896140-jrzlz\" (UID: \"941eb4d4-9d72-412b-b5ac-1bfbd65b439c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1874d8-29896140-jrzlz" Nov 04 05:00:00.554819 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:00.554753 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4n9xv\" (UniqueName: \"kubernetes.io/projected/467298da-84d2-45fa-bf41-c09422256efa-kube-api-access-4n9xv\") pod \"maas-api-key-cleanup-e2e-worker-w3-dab70d-29896140-w2fbg\" (UID: \"467298da-84d2-45fa-bf41-c09422256efa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dab70d-29896140-w2fbg" Nov 04 05:00:00.554819 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:00.554810 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdcgh\" (UniqueName: \"kubernetes.io/projected/7124f27c-88a3-4467-8793-cf58ae5c06d2-kube-api-access-sdcgh\") pod \"maas-api-key-cleanup-29896140-kkj8l\" (UID: \"7124f27c-88a3-4467-8793-cf58ae5c06d2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29896140-kkj8l" Nov 04 05:00:00.554985 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:00.554838 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7124f27c-88a3-4467-8793-cf58ae5c06d2-maas-api-service-ca\") pod \"maas-api-key-cleanup-29896140-kkj8l\" (UID: \"7124f27c-88a3-4467-8793-cf58ae5c06d2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29896140-kkj8l" Nov 04 05:00:00.554985 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:00.554869 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrlgk\" (UniqueName: \"kubernetes.io/projected/941eb4d4-9d72-412b-b5ac-1bfbd65b439c-kube-api-access-mrlgk\") pod \"maas-api-key-cleanup-e2e-worker-w1-1874d8-29896140-jrzlz\" (UID: \"941eb4d4-9d72-412b-b5ac-1bfbd65b439c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1874d8-29896140-jrzlz" Nov 04 05:00:00.554985 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:00.554914 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/467298da-84d2-45fa-bf41-c09422256efa-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-dab70d-29896140-w2fbg\" (UID: \"467298da-84d2-45fa-bf41-c09422256efa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dab70d-29896140-w2fbg" Nov 04 05:00:00.555135 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:00.554985 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6tbv\" (UniqueName: \"kubernetes.io/projected/7114f0d1-b384-4bca-b342-29688c0ec8a5-kube-api-access-r6tbv\") pod \"maas-api-key-cleanup-e2e-worker-w2-370078-29896140-9tcm2\" (UID: \"7114f0d1-b384-4bca-b342-29688c0ec8a5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-370078-29896140-9tcm2" Nov 04 05:00:00.555597 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:00.555573 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/941eb4d4-9d72-412b-b5ac-1bfbd65b439c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-1874d8-29896140-jrzlz\" (UID: \"941eb4d4-9d72-412b-b5ac-1bfbd65b439c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1874d8-29896140-jrzlz" Nov 04 05:00:00.555714 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:00.555630 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7114f0d1-b384-4bca-b342-29688c0ec8a5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-370078-29896140-9tcm2\" (UID: \"7114f0d1-b384-4bca-b342-29688c0ec8a5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-370078-29896140-9tcm2" Nov 04 05:00:00.555852 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:00.555832 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7124f27c-88a3-4467-8793-cf58ae5c06d2-maas-api-service-ca\") pod \"maas-api-key-cleanup-29896140-kkj8l\" (UID: \"7124f27c-88a3-4467-8793-cf58ae5c06d2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29896140-kkj8l" Nov 04 05:00:00.555914 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:00.555881 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/467298da-84d2-45fa-bf41-c09422256efa-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-dab70d-29896140-w2fbg\" (UID: \"467298da-84d2-45fa-bf41-c09422256efa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dab70d-29896140-w2fbg" Nov 04 05:00:00.618440 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:00.617378 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6tbv\" (UniqueName: \"kubernetes.io/projected/7114f0d1-b384-4bca-b342-29688c0ec8a5-kube-api-access-r6tbv\") pod \"maas-api-key-cleanup-e2e-worker-w2-370078-29896140-9tcm2\" (UID: \"7114f0d1-b384-4bca-b342-29688c0ec8a5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-370078-29896140-9tcm2" Nov 04 05:00:00.618440 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:00.617848 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrlgk\" (UniqueName: \"kubernetes.io/projected/941eb4d4-9d72-412b-b5ac-1bfbd65b439c-kube-api-access-mrlgk\") pod \"maas-api-key-cleanup-e2e-worker-w1-1874d8-29896140-jrzlz\" (UID: \"941eb4d4-9d72-412b-b5ac-1bfbd65b439c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1874d8-29896140-jrzlz" Nov 04 05:00:00.618440 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:00.617914 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdcgh\" (UniqueName: \"kubernetes.io/projected/7124f27c-88a3-4467-8793-cf58ae5c06d2-kube-api-access-sdcgh\") pod \"maas-api-key-cleanup-29896140-kkj8l\" (UID: \"7124f27c-88a3-4467-8793-cf58ae5c06d2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29896140-kkj8l" Nov 04 05:00:00.618440 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:00.618374 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4n9xv\" (UniqueName: \"kubernetes.io/projected/467298da-84d2-45fa-bf41-c09422256efa-kube-api-access-4n9xv\") pod \"maas-api-key-cleanup-e2e-worker-w3-dab70d-29896140-w2fbg\" (UID: \"467298da-84d2-45fa-bf41-c09422256efa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dab70d-29896140-w2fbg" Nov 04 05:00:00.774453 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:00.774430 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-3eb31c/premium-w4-3eb31c-kserve-58bc6d6c55-kh2c4"] Nov 04 05:00:00.775862 ip-10-0-133-180 kubenswrapper[4509]: W1104 05:00:00.775835 4509 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9181aed8_6d05_4691_be54_f26391920734.slice/crio-e9fb1b1ad8cf42de91d29df19a0317eee7bc712514d546dd66d0e73a47a48be5 WatchSource:0}: Error finding container e9fb1b1ad8cf42de91d29df19a0317eee7bc712514d546dd66d0e73a47a48be5: Status 404 returned error can't find the container with id e9fb1b1ad8cf42de91d29df19a0317eee7bc712514d546dd66d0e73a47a48be5 Nov 04 05:00:00.828754 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:00.828716 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1874d8-29896140-jrzlz" Nov 04 05:00:00.860388 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:00.860282 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-370078-29896140-9tcm2" Nov 04 05:00:00.869051 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:00.869001 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29896140-kkj8l" Nov 04 05:00:00.874477 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:00.874454 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dab70d-29896140-w2fbg" Nov 04 05:00:00.914202 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:00.913842 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-cedfca-29896140-k8xk6"] Nov 04 05:00:00.915797 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:00.915775 4509 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 04 05:00:01.225962 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:01.225942 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1874d8-29896140-jrzlz"] Nov 04 05:00:01.227109 ip-10-0-133-180 kubenswrapper[4509]: W1104 05:00:01.227080 4509 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod941eb4d4_9d72_412b_b5ac_1bfbd65b439c.slice/crio-3e1cf0dfc04cf257070403826ead76843bbaccf297be2d9d6db3bdcacd34644b WatchSource:0}: Error finding container 3e1cf0dfc04cf257070403826ead76843bbaccf297be2d9d6db3bdcacd34644b: Status 404 returned error can't find the container with id 3e1cf0dfc04cf257070403826ead76843bbaccf297be2d9d6db3bdcacd34644b Nov 04 05:00:01.260484 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:01.260463 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29896140-kkj8l"] Nov 04 05:00:01.263878 ip-10-0-133-180 kubenswrapper[4509]: W1104 05:00:01.263849 4509 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7124f27c_88a3_4467_8793_cf58ae5c06d2.slice/crio-52546f56ce45b1d74f65566078772d8a8dfbc66a18b86c2759024254180db309 WatchSource:0}: Error finding container 52546f56ce45b1d74f65566078772d8a8dfbc66a18b86c2759024254180db309: Status 404 returned error can't find the container with id 52546f56ce45b1d74f65566078772d8a8dfbc66a18b86c2759024254180db309 Nov 04 05:00:01.281939 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:01.281914 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-370078-29896140-9tcm2"] Nov 04 05:00:01.283580 ip-10-0-133-180 kubenswrapper[4509]: W1104 05:00:01.283561 4509 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7114f0d1_b384_4bca_b342_29688c0ec8a5.slice/crio-2a0e48fe3d0e8eaeeb30896969db6ec58ef4e83214952a16a791751565c127d7 WatchSource:0}: Error finding container 2a0e48fe3d0e8eaeeb30896969db6ec58ef4e83214952a16a791751565c127d7: Status 404 returned error can't find the container with id 2a0e48fe3d0e8eaeeb30896969db6ec58ef4e83214952a16a791751565c127d7 Nov 04 05:00:01.310756 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:01.310736 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dab70d-29896140-w2fbg"] Nov 04 05:00:01.312994 ip-10-0-133-180 kubenswrapper[4509]: W1104 05:00:01.312936 4509 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod467298da_84d2_45fa_bf41_c09422256efa.slice/crio-4f9dff1cba7967ec37e9a3c02dbba722f24b25e169057495920cc225b454eab9 WatchSource:0}: Error finding container 4f9dff1cba7967ec37e9a3c02dbba722f24b25e169057495920cc225b454eab9: Status 404 returned error can't find the container with id 4f9dff1cba7967ec37e9a3c02dbba722f24b25e169057495920cc225b454eab9 Nov 04 05:00:01.557396 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:01.557222 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-cedfca-29896140-k8xk6" event={"ID":"4baf9fe5-84d1-48a1-8c10-8080a1e6a5db","Type":"ContainerStarted","Data":"41260b5a4f9d0346b0abe8a8f433932a6d47328536b5ef15f8ac66b3b6d1fa9c"} Nov 04 05:00:01.559888 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:01.559800 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dab70d-29896140-w2fbg" event={"ID":"467298da-84d2-45fa-bf41-c09422256efa","Type":"ContainerStarted","Data":"4f9dff1cba7967ec37e9a3c02dbba722f24b25e169057495920cc225b454eab9"} Nov 04 05:00:01.561728 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:01.561660 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-370078-29896140-9tcm2" event={"ID":"7114f0d1-b384-4bca-b342-29688c0ec8a5","Type":"ContainerStarted","Data":"2a0e48fe3d0e8eaeeb30896969db6ec58ef4e83214952a16a791751565c127d7"} Nov 04 05:00:01.563504 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:01.563476 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1874d8-29896140-jrzlz" event={"ID":"941eb4d4-9d72-412b-b5ac-1bfbd65b439c","Type":"ContainerStarted","Data":"3e1cf0dfc04cf257070403826ead76843bbaccf297be2d9d6db3bdcacd34644b"} Nov 04 05:00:01.564898 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:01.564877 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29896140-kkj8l" event={"ID":"7124f27c-88a3-4467-8793-cf58ae5c06d2","Type":"ContainerStarted","Data":"52546f56ce45b1d74f65566078772d8a8dfbc66a18b86c2759024254180db309"} Nov 04 05:00:01.566313 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:01.566287 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3eb31c/premium-w4-3eb31c-kserve-58bc6d6c55-kh2c4" event={"ID":"9181aed8-6d05-4691-be54-f26391920734","Type":"ContainerStarted","Data":"8d31357fb32fc0a54b637018306887fc1853f6f1c34474340f5b1b806231e998"} Nov 04 05:00:01.566410 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:01.566339 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3eb31c/premium-w4-3eb31c-kserve-58bc6d6c55-kh2c4" event={"ID":"9181aed8-6d05-4691-be54-f26391920734","Type":"ContainerStarted","Data":"e9fb1b1ad8cf42de91d29df19a0317eee7bc712514d546dd66d0e73a47a48be5"} Nov 04 05:00:01.686073 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:01.686030 4509 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-3eb31c/premium-w4-3eb31c-kserve-58bc6d6c55-kh2c4" podStartSLOduration=1.686018331 podStartE2EDuration="1.686018331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 05:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 05:00:01.683255801 +0000 UTC m=+765.560979321" watchObservedRunningTime="2026-11-04 05:00:01.686018331 +0000 UTC m=+765.563741879" Nov 04 05:00:02.397386 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:02.397357 4509 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3eb31c/premium-w4-3eb31c-kserve-58bc6d6c55-kh2c4" Nov 04 05:00:07.611295 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:07.611234 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dab70d-29896140-w2fbg" event={"ID":"467298da-84d2-45fa-bf41-c09422256efa","Type":"ContainerStarted","Data":"476b33eac74acc1feb68b11cd9ba5b3e186f273474ab7ff6894fa12a48bd4dee"} Nov 04 05:00:07.612868 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:07.612840 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1874d8-29896140-jrzlz" event={"ID":"941eb4d4-9d72-412b-b5ac-1bfbd65b439c","Type":"ContainerStarted","Data":"1ff10ee8c5aa571753e38e164e370f37e5e8c6041fc14f951ad38947bc1add84"} Nov 04 05:00:07.614212 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:07.614188 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-cedfca-29896140-k8xk6" event={"ID":"4baf9fe5-84d1-48a1-8c10-8080a1e6a5db","Type":"ContainerStarted","Data":"790ebc461dad72d3340507e712580c0e017d7a00be41c0b71233fa845dfa8c28"} Nov 04 05:00:07.725422 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:07.725380 4509 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dab70d-29896140-w2fbg" podStartSLOduration=1.738114444 podStartE2EDuration="7.725368497s" totalImagesPullingTime="5.987254053s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 05:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 05:00:07.723490133 +0000 UTC m=+771.601213653" watchObservedRunningTime="2026-11-04 05:00:07.725368497 +0000 UTC m=+771.603092010" Nov 04 05:00:07.838305 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:07.838267 4509 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-cedfca-29896140-k8xk6" podStartSLOduration=1.525464634 podStartE2EDuration="7.838251135s" totalImagesPullingTime="6.312786501s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 05:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 05:00:07.837061577 +0000 UTC m=+771.714785097" watchObservedRunningTime="2026-11-04 05:00:07.838251135 +0000 UTC m=+771.715974654" Nov 04 05:00:07.950227 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:07.950153 4509 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1874d8-29896140-jrzlz" podStartSLOduration=2.366563947 podStartE2EDuration="7.950141516s" totalImagesPullingTime="5.583577569s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 05:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 05:00:07.94869142 +0000 UTC m=+771.826414940" watchObservedRunningTime="2026-11-04 05:00:07.950141516 +0000 UTC m=+771.827865035" Nov 04 05:00:08.621366 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:08.621266 4509 generic.go:356] "Generic (PLEG): container finished" podID="4baf9fe5-84d1-48a1-8c10-8080a1e6a5db" containerID="790ebc461dad72d3340507e712580c0e017d7a00be41c0b71233fa845dfa8c28" exitCode=60 Nov 04 05:00:08.621366 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:08.621348 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-cedfca-29896140-k8xk6" event={"ID":"4baf9fe5-84d1-48a1-8c10-8080a1e6a5db","Type":"ContainerDied","Data":"790ebc461dad72d3340507e712580c0e017d7a00be41c0b71233fa845dfa8c28"} Nov 04 05:00:08.621770 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:08.621637 4509 scope.go:122] "RemoveContainer" containerID="790ebc461dad72d3340507e712580c0e017d7a00be41c0b71233fa845dfa8c28" Nov 04 05:00:08.623057 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:08.623035 4509 generic.go:356] "Generic (PLEG): container finished" podID="467298da-84d2-45fa-bf41-c09422256efa" containerID="476b33eac74acc1feb68b11cd9ba5b3e186f273474ab7ff6894fa12a48bd4dee" exitCode=60 Nov 04 05:00:08.623185 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:08.623108 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dab70d-29896140-w2fbg" event={"ID":"467298da-84d2-45fa-bf41-c09422256efa","Type":"ContainerDied","Data":"476b33eac74acc1feb68b11cd9ba5b3e186f273474ab7ff6894fa12a48bd4dee"} Nov 04 05:00:08.623354 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:08.623341 4509 scope.go:122] "RemoveContainer" containerID="476b33eac74acc1feb68b11cd9ba5b3e186f273474ab7ff6894fa12a48bd4dee" Nov 04 05:00:08.624803 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:08.624775 4509 generic.go:356] "Generic (PLEG): container finished" podID="7114f0d1-b384-4bca-b342-29688c0ec8a5" containerID="5a7c9ccd31e158e07104cb0e37f18fa2ec7ed6889ad8574569830e44bf52e6c1" exitCode=60 Nov 04 05:00:08.624874 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:08.624851 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-370078-29896140-9tcm2" event={"ID":"7114f0d1-b384-4bca-b342-29688c0ec8a5","Type":"ContainerDied","Data":"5a7c9ccd31e158e07104cb0e37f18fa2ec7ed6889ad8574569830e44bf52e6c1"} Nov 04 05:00:08.625085 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:08.625066 4509 scope.go:122] "RemoveContainer" containerID="5a7c9ccd31e158e07104cb0e37f18fa2ec7ed6889ad8574569830e44bf52e6c1" Nov 04 05:00:08.626468 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:08.626443 4509 generic.go:356] "Generic (PLEG): container finished" podID="941eb4d4-9d72-412b-b5ac-1bfbd65b439c" containerID="1ff10ee8c5aa571753e38e164e370f37e5e8c6041fc14f951ad38947bc1add84" exitCode=60 Nov 04 05:00:08.626540 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:08.626504 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1874d8-29896140-jrzlz" event={"ID":"941eb4d4-9d72-412b-b5ac-1bfbd65b439c","Type":"ContainerDied","Data":"1ff10ee8c5aa571753e38e164e370f37e5e8c6041fc14f951ad38947bc1add84"} Nov 04 05:00:08.626757 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:08.626744 4509 scope.go:122] "RemoveContainer" containerID="1ff10ee8c5aa571753e38e164e370f37e5e8c6041fc14f951ad38947bc1add84" Nov 04 05:00:08.629150 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:08.629132 4509 generic.go:356] "Generic (PLEG): container finished" podID="7124f27c-88a3-4467-8793-cf58ae5c06d2" containerID="a3ac33d933b98b6acf5a89defc4167c99849634099975219601eae567fd71480" exitCode=60 Nov 04 05:00:08.629235 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:08.629177 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29896140-kkj8l" event={"ID":"7124f27c-88a3-4467-8793-cf58ae5c06d2","Type":"ContainerDied","Data":"a3ac33d933b98b6acf5a89defc4167c99849634099975219601eae567fd71480"} Nov 04 05:00:08.629471 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:08.629456 4509 scope.go:122] "RemoveContainer" containerID="a3ac33d933b98b6acf5a89defc4167c99849634099975219601eae567fd71480" Nov 04 05:00:09.636559 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:09.636527 4509 generic.go:356] "Generic (PLEG): container finished" podID="4baf9fe5-84d1-48a1-8c10-8080a1e6a5db" containerID="7e574908985decb9a1152414e6d313997be1f928e84f0acd06980a420db4bb04" exitCode=60 Nov 04 05:00:09.636911 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:09.636576 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-cedfca-29896140-k8xk6" event={"ID":"4baf9fe5-84d1-48a1-8c10-8080a1e6a5db","Type":"ContainerDied","Data":"7e574908985decb9a1152414e6d313997be1f928e84f0acd06980a420db4bb04"} Nov 04 05:00:09.636911 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:09.636637 4509 scope.go:122] "RemoveContainer" containerID="790ebc461dad72d3340507e712580c0e017d7a00be41c0b71233fa845dfa8c28" Nov 04 05:00:09.637013 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:09.636944 4509 scope.go:122] "RemoveContainer" containerID="7e574908985decb9a1152414e6d313997be1f928e84f0acd06980a420db4bb04" Nov 04 05:00:09.637244 ip-10-0-133-180 kubenswrapper[4509]: E1104 05:00:09.637219 4509 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-cedfca-29896140-k8xk6_odh-ai-gateway-infra(4baf9fe5-84d1-48a1-8c10-8080a1e6a5db)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-cedfca-29896140-k8xk6" podUID="4baf9fe5-84d1-48a1-8c10-8080a1e6a5db" Nov 04 05:00:09.638742 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:09.638722 4509 generic.go:356] "Generic (PLEG): container finished" podID="467298da-84d2-45fa-bf41-c09422256efa" containerID="72ca0c0672e67913b3cc0e5e9047f6728bec5bb4ff859f9a9246a5ce3f8030ea" exitCode=60 Nov 04 05:00:09.638854 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:09.638748 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dab70d-29896140-w2fbg" event={"ID":"467298da-84d2-45fa-bf41-c09422256efa","Type":"ContainerDied","Data":"72ca0c0672e67913b3cc0e5e9047f6728bec5bb4ff859f9a9246a5ce3f8030ea"} Nov 04 05:00:09.639042 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:09.639022 4509 scope.go:122] "RemoveContainer" containerID="72ca0c0672e67913b3cc0e5e9047f6728bec5bb4ff859f9a9246a5ce3f8030ea" Nov 04 05:00:09.639299 ip-10-0-133-180 kubenswrapper[4509]: E1104 05:00:09.639279 4509 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-dab70d-29896140-w2fbg_odh-ai-gateway-infra(467298da-84d2-45fa-bf41-c09422256efa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dab70d-29896140-w2fbg" podUID="467298da-84d2-45fa-bf41-c09422256efa" Nov 04 05:00:09.640729 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:09.640700 4509 generic.go:356] "Generic (PLEG): container finished" podID="7114f0d1-b384-4bca-b342-29688c0ec8a5" containerID="e2cfc2ac4b8dd19ca3b0e9624e242f420393e5569506eef5d0de4f994ad43358" exitCode=60 Nov 04 05:00:09.640823 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:09.640772 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-370078-29896140-9tcm2" event={"ID":"7114f0d1-b384-4bca-b342-29688c0ec8a5","Type":"ContainerDied","Data":"e2cfc2ac4b8dd19ca3b0e9624e242f420393e5569506eef5d0de4f994ad43358"} Nov 04 05:00:09.641039 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:09.641024 4509 scope.go:122] "RemoveContainer" containerID="e2cfc2ac4b8dd19ca3b0e9624e242f420393e5569506eef5d0de4f994ad43358" Nov 04 05:00:09.641264 ip-10-0-133-180 kubenswrapper[4509]: E1104 05:00:09.641245 4509 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-370078-29896140-9tcm2_odh-ai-gateway-infra(7114f0d1-b384-4bca-b342-29688c0ec8a5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-370078-29896140-9tcm2" podUID="7114f0d1-b384-4bca-b342-29688c0ec8a5" Nov 04 05:00:09.642838 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:09.642814 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1874d8-29896140-jrzlz" event={"ID":"941eb4d4-9d72-412b-b5ac-1bfbd65b439c","Type":"ContainerStarted","Data":"d49871543851bede258765d156d4034987e9d6fa6937f66f033dde346f5a1322"} Nov 04 05:00:09.644536 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:09.644517 4509 generic.go:356] "Generic (PLEG): container finished" podID="7124f27c-88a3-4467-8793-cf58ae5c06d2" containerID="45d6337e32994f86dcdfb2bbb1459dd798aec98b68a941c5c14deec62b1dcfbf" exitCode=60 Nov 04 05:00:09.644617 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:09.644590 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29896140-kkj8l" event={"ID":"7124f27c-88a3-4467-8793-cf58ae5c06d2","Type":"ContainerDied","Data":"45d6337e32994f86dcdfb2bbb1459dd798aec98b68a941c5c14deec62b1dcfbf"} Nov 04 05:00:09.644757 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:09.644744 4509 scope.go:122] "RemoveContainer" containerID="45d6337e32994f86dcdfb2bbb1459dd798aec98b68a941c5c14deec62b1dcfbf" Nov 04 05:00:09.644931 ip-10-0-133-180 kubenswrapper[4509]: E1104 05:00:09.644917 4509 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29896140-kkj8l_odh-ai-gateway-infra(7124f27c-88a3-4467-8793-cf58ae5c06d2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29896140-kkj8l" podUID="7124f27c-88a3-4467-8793-cf58ae5c06d2" Nov 04 05:00:09.655459 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:09.655439 4509 scope.go:122] "RemoveContainer" containerID="476b33eac74acc1feb68b11cd9ba5b3e186f273474ab7ff6894fa12a48bd4dee" Nov 04 05:00:09.667896 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:09.667877 4509 scope.go:122] "RemoveContainer" containerID="5a7c9ccd31e158e07104cb0e37f18fa2ec7ed6889ad8574569830e44bf52e6c1" Nov 04 05:00:09.708477 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:09.708452 4509 scope.go:122] "RemoveContainer" containerID="a3ac33d933b98b6acf5a89defc4167c99849634099975219601eae567fd71480" Nov 04 05:00:10.397639 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:10.397612 4509 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-3eb31c/premium-w4-3eb31c-kserve-58bc6d6c55-kh2c4" Nov 04 05:00:10.409443 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:10.409419 4509 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-3eb31c/premium-w4-3eb31c-kserve-58bc6d6c55-kh2c4" Nov 04 05:00:10.652007 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:10.651936 4509 scope.go:122] "RemoveContainer" containerID="72ca0c0672e67913b3cc0e5e9047f6728bec5bb4ff859f9a9246a5ce3f8030ea" Nov 04 05:00:10.652450 ip-10-0-133-180 kubenswrapper[4509]: E1104 05:00:10.652154 4509 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-dab70d-29896140-w2fbg_odh-ai-gateway-infra(467298da-84d2-45fa-bf41-c09422256efa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dab70d-29896140-w2fbg" podUID="467298da-84d2-45fa-bf41-c09422256efa" Nov 04 05:00:10.653404 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:10.653384 4509 scope.go:122] "RemoveContainer" containerID="e2cfc2ac4b8dd19ca3b0e9624e242f420393e5569506eef5d0de4f994ad43358" Nov 04 05:00:10.653608 ip-10-0-133-180 kubenswrapper[4509]: E1104 05:00:10.653586 4509 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-370078-29896140-9tcm2_odh-ai-gateway-infra(7114f0d1-b384-4bca-b342-29688c0ec8a5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-370078-29896140-9tcm2" podUID="7114f0d1-b384-4bca-b342-29688c0ec8a5" Nov 04 05:00:10.654902 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:10.654876 4509 generic.go:356] "Generic (PLEG): container finished" podID="941eb4d4-9d72-412b-b5ac-1bfbd65b439c" containerID="d49871543851bede258765d156d4034987e9d6fa6937f66f033dde346f5a1322" exitCode=60 Nov 04 05:00:10.654985 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:10.654940 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1874d8-29896140-jrzlz" event={"ID":"941eb4d4-9d72-412b-b5ac-1bfbd65b439c","Type":"ContainerDied","Data":"d49871543851bede258765d156d4034987e9d6fa6937f66f033dde346f5a1322"} Nov 04 05:00:10.654985 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:10.654978 4509 scope.go:122] "RemoveContainer" containerID="1ff10ee8c5aa571753e38e164e370f37e5e8c6041fc14f951ad38947bc1add84" Nov 04 05:00:10.655118 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:10.655104 4509 scope.go:122] "RemoveContainer" containerID="d49871543851bede258765d156d4034987e9d6fa6937f66f033dde346f5a1322" Nov 04 05:00:10.655802 ip-10-0-133-180 kubenswrapper[4509]: E1104 05:00:10.655773 4509 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-1874d8-29896140-jrzlz_odh-ai-gateway-infra(941eb4d4-9d72-412b-b5ac-1bfbd65b439c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1874d8-29896140-jrzlz" podUID="941eb4d4-9d72-412b-b5ac-1bfbd65b439c" Nov 04 05:00:10.657115 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:10.657097 4509 scope.go:122] "RemoveContainer" containerID="45d6337e32994f86dcdfb2bbb1459dd798aec98b68a941c5c14deec62b1dcfbf" Nov 04 05:00:10.657309 ip-10-0-133-180 kubenswrapper[4509]: E1104 05:00:10.657284 4509 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29896140-kkj8l_odh-ai-gateway-infra(7124f27c-88a3-4467-8793-cf58ae5c06d2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29896140-kkj8l" podUID="7124f27c-88a3-4467-8793-cf58ae5c06d2" Nov 04 05:00:10.658591 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:10.658573 4509 scope.go:122] "RemoveContainer" containerID="7e574908985decb9a1152414e6d313997be1f928e84f0acd06980a420db4bb04" Nov 04 05:00:10.658777 ip-10-0-133-180 kubenswrapper[4509]: E1104 05:00:10.658756 4509 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-cedfca-29896140-k8xk6_odh-ai-gateway-infra(4baf9fe5-84d1-48a1-8c10-8080a1e6a5db)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-cedfca-29896140-k8xk6" podUID="4baf9fe5-84d1-48a1-8c10-8080a1e6a5db" Nov 04 05:00:10.669200 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:10.669180 4509 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3eb31c/premium-w4-3eb31c-kserve-58bc6d6c55-kh2c4" Nov 04 05:00:11.665145 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:11.665117 4509 scope.go:122] "RemoveContainer" containerID="d49871543851bede258765d156d4034987e9d6fa6937f66f033dde346f5a1322" Nov 04 05:00:11.665605 ip-10-0-133-180 kubenswrapper[4509]: E1104 05:00:11.665362 4509 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-1874d8-29896140-jrzlz_odh-ai-gateway-infra(941eb4d4-9d72-412b-b5ac-1bfbd65b439c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1874d8-29896140-jrzlz" podUID="941eb4d4-9d72-412b-b5ac-1bfbd65b439c" Nov 04 05:00:12.955028 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:12.954996 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-42dnm"] Nov 04 05:00:14.452749 ip-10-0-133-180 kubenswrapper[4509]: E1104 05:00:14.452717 4509 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmdhl" containerName="extractor" Nov 04 05:00:20.255172 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:20.255145 4509 scope.go:122] "RemoveContainer" containerID="45d6337e32994f86dcdfb2bbb1459dd798aec98b68a941c5c14deec62b1dcfbf" Nov 04 05:00:20.713249 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:20.713213 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29896140-kkj8l" event={"ID":"7124f27c-88a3-4467-8793-cf58ae5c06d2","Type":"ContainerStarted","Data":"96729a219ee54d0eb74bf617eec775dcb6fd79dfe27365648afa569acabdd5fe"} Nov 04 05:00:20.827981 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:20.827895 4509 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29896140-kkj8l" podStartSLOduration=14.856511229 podStartE2EDuration="20.827883185s" totalImagesPullingTime="5.971371956s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 05:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 05:00:20.826113594 +0000 UTC m=+784.703837113" watchObservedRunningTime="2026-11-04 05:00:20.827883185 +0000 UTC m=+784.705606704" Nov 04 05:00:21.255022 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:21.255000 4509 scope.go:122] "RemoveContainer" containerID="7e574908985decb9a1152414e6d313997be1f928e84f0acd06980a420db4bb04" Nov 04 05:00:21.560156 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:21.560098 4509 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-1874d8/simulator-w1-1874d8-kserve-85dc6d466-lkxlz"] Nov 04 05:00:21.563837 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:21.563821 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-1874d8/simulator-w1-1874d8-kserve-85dc6d466-lkxlz" Nov 04 05:00:21.616467 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:21.616447 4509 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-1874d8\"/\"kube-root-ca.crt\"" Nov 04 05:00:21.654964 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:21.654935 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcqtt\" (UniqueName: \"kubernetes.io/projected/4655c4c9-ad98-4f20-919c-56905e179bad-kube-api-access-wcqtt\") pod \"simulator-w1-1874d8-kserve-85dc6d466-lkxlz\" (UID: \"4655c4c9-ad98-4f20-919c-56905e179bad\") " pod="e2e-models-e2e-worker-w1-1874d8/simulator-w1-1874d8-kserve-85dc6d466-lkxlz" Nov 04 05:00:21.655096 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:21.654983 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4655c4c9-ad98-4f20-919c-56905e179bad-tmp-dir\") pod \"simulator-w1-1874d8-kserve-85dc6d466-lkxlz\" (UID: \"4655c4c9-ad98-4f20-919c-56905e179bad\") " pod="e2e-models-e2e-worker-w1-1874d8/simulator-w1-1874d8-kserve-85dc6d466-lkxlz" Nov 04 05:00:21.655191 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:21.655177 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4655c4c9-ad98-4f20-919c-56905e179bad-model-cache\") pod \"simulator-w1-1874d8-kserve-85dc6d466-lkxlz\" (UID: \"4655c4c9-ad98-4f20-919c-56905e179bad\") " pod="e2e-models-e2e-worker-w1-1874d8/simulator-w1-1874d8-kserve-85dc6d466-lkxlz" Nov 04 05:00:21.655263 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:21.655209 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4655c4c9-ad98-4f20-919c-56905e179bad-dshm\") pod \"simulator-w1-1874d8-kserve-85dc6d466-lkxlz\" (UID: \"4655c4c9-ad98-4f20-919c-56905e179bad\") " pod="e2e-models-e2e-worker-w1-1874d8/simulator-w1-1874d8-kserve-85dc6d466-lkxlz" Nov 04 05:00:21.655263 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:21.655254 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4655c4c9-ad98-4f20-919c-56905e179bad-home\") pod \"simulator-w1-1874d8-kserve-85dc6d466-lkxlz\" (UID: \"4655c4c9-ad98-4f20-919c-56905e179bad\") " pod="e2e-models-e2e-worker-w1-1874d8/simulator-w1-1874d8-kserve-85dc6d466-lkxlz" Nov 04 05:00:21.655403 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:21.655345 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4655c4c9-ad98-4f20-919c-56905e179bad-tls-certs\") pod \"simulator-w1-1874d8-kserve-85dc6d466-lkxlz\" (UID: \"4655c4c9-ad98-4f20-919c-56905e179bad\") " pod="e2e-models-e2e-worker-w1-1874d8/simulator-w1-1874d8-kserve-85dc6d466-lkxlz" Nov 04 05:00:21.666027 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:21.666010 4509 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-1874d8\"/\"openshift-service-ca.crt\"" Nov 04 05:00:21.666138 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:21.666029 4509 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-1874d8\"/\"default-dockercfg-h2jcg\"" Nov 04 05:00:21.666197 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:21.666135 4509 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-1874d8\"/\"simulator-w1-1874d8-kserve-self-signed-certs\"" Nov 04 05:00:21.668633 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:21.668614 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-1874d8/simulator-w1-1874d8-kserve-85dc6d466-lkxlz"] Nov 04 05:00:21.723401 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:21.723368 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-cedfca-29896140-k8xk6" event={"ID":"4baf9fe5-84d1-48a1-8c10-8080a1e6a5db","Type":"ContainerStarted","Data":"a5c50c3923ee57392d00f4d549ad21fe0067a16b5bca160393dacf35ae83a6b0"} Nov 04 05:00:21.725373 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:21.725348 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29896140-kkj8l" event={"ID":"7124f27c-88a3-4467-8793-cf58ae5c06d2","Type":"ContainerDied","Data":"96729a219ee54d0eb74bf617eec775dcb6fd79dfe27365648afa569acabdd5fe"} Nov 04 05:00:21.725501 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:21.725396 4509 scope.go:122] "RemoveContainer" containerID="45d6337e32994f86dcdfb2bbb1459dd798aec98b68a941c5c14deec62b1dcfbf" Nov 04 05:00:21.725501 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:21.725356 4509 generic.go:356] "Generic (PLEG): container finished" podID="7124f27c-88a3-4467-8793-cf58ae5c06d2" containerID="96729a219ee54d0eb74bf617eec775dcb6fd79dfe27365648afa569acabdd5fe" exitCode=60 Nov 04 05:00:21.725753 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:21.725737 4509 scope.go:122] "RemoveContainer" containerID="96729a219ee54d0eb74bf617eec775dcb6fd79dfe27365648afa569acabdd5fe" Nov 04 05:00:21.726005 ip-10-0-133-180 kubenswrapper[4509]: E1104 05:00:21.725986 4509 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29896140-kkj8l_odh-ai-gateway-infra(7124f27c-88a3-4467-8793-cf58ae5c06d2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29896140-kkj8l" podUID="7124f27c-88a3-4467-8793-cf58ae5c06d2" Nov 04 05:00:21.756656 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:21.756635 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4655c4c9-ad98-4f20-919c-56905e179bad-model-cache\") pod \"simulator-w1-1874d8-kserve-85dc6d466-lkxlz\" (UID: \"4655c4c9-ad98-4f20-919c-56905e179bad\") " pod="e2e-models-e2e-worker-w1-1874d8/simulator-w1-1874d8-kserve-85dc6d466-lkxlz" Nov 04 05:00:21.756758 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:21.756665 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4655c4c9-ad98-4f20-919c-56905e179bad-dshm\") pod \"simulator-w1-1874d8-kserve-85dc6d466-lkxlz\" (UID: \"4655c4c9-ad98-4f20-919c-56905e179bad\") " pod="e2e-models-e2e-worker-w1-1874d8/simulator-w1-1874d8-kserve-85dc6d466-lkxlz" Nov 04 05:00:21.756758 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:21.756692 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4655c4c9-ad98-4f20-919c-56905e179bad-home\") pod \"simulator-w1-1874d8-kserve-85dc6d466-lkxlz\" (UID: \"4655c4c9-ad98-4f20-919c-56905e179bad\") " pod="e2e-models-e2e-worker-w1-1874d8/simulator-w1-1874d8-kserve-85dc6d466-lkxlz" Nov 04 05:00:21.756758 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:21.756714 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4655c4c9-ad98-4f20-919c-56905e179bad-tls-certs\") pod \"simulator-w1-1874d8-kserve-85dc6d466-lkxlz\" (UID: \"4655c4c9-ad98-4f20-919c-56905e179bad\") " pod="e2e-models-e2e-worker-w1-1874d8/simulator-w1-1874d8-kserve-85dc6d466-lkxlz" Nov 04 05:00:21.756758 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:21.756745 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcqtt\" (UniqueName: \"kubernetes.io/projected/4655c4c9-ad98-4f20-919c-56905e179bad-kube-api-access-wcqtt\") pod \"simulator-w1-1874d8-kserve-85dc6d466-lkxlz\" (UID: \"4655c4c9-ad98-4f20-919c-56905e179bad\") " pod="e2e-models-e2e-worker-w1-1874d8/simulator-w1-1874d8-kserve-85dc6d466-lkxlz" Nov 04 05:00:21.756946 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:21.756766 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4655c4c9-ad98-4f20-919c-56905e179bad-tmp-dir\") pod \"simulator-w1-1874d8-kserve-85dc6d466-lkxlz\" (UID: \"4655c4c9-ad98-4f20-919c-56905e179bad\") " pod="e2e-models-e2e-worker-w1-1874d8/simulator-w1-1874d8-kserve-85dc6d466-lkxlz" Nov 04 05:00:21.757061 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:21.757034 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4655c4c9-ad98-4f20-919c-56905e179bad-home\") pod \"simulator-w1-1874d8-kserve-85dc6d466-lkxlz\" (UID: \"4655c4c9-ad98-4f20-919c-56905e179bad\") " pod="e2e-models-e2e-worker-w1-1874d8/simulator-w1-1874d8-kserve-85dc6d466-lkxlz" Nov 04 05:00:21.757131 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:21.757079 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4655c4c9-ad98-4f20-919c-56905e179bad-model-cache\") pod \"simulator-w1-1874d8-kserve-85dc6d466-lkxlz\" (UID: \"4655c4c9-ad98-4f20-919c-56905e179bad\") " pod="e2e-models-e2e-worker-w1-1874d8/simulator-w1-1874d8-kserve-85dc6d466-lkxlz" Nov 04 05:00:21.757131 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:21.757124 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4655c4c9-ad98-4f20-919c-56905e179bad-tmp-dir\") pod \"simulator-w1-1874d8-kserve-85dc6d466-lkxlz\" (UID: \"4655c4c9-ad98-4f20-919c-56905e179bad\") " pod="e2e-models-e2e-worker-w1-1874d8/simulator-w1-1874d8-kserve-85dc6d466-lkxlz" Nov 04 05:00:21.758852 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:21.758827 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4655c4c9-ad98-4f20-919c-56905e179bad-dshm\") pod \"simulator-w1-1874d8-kserve-85dc6d466-lkxlz\" (UID: \"4655c4c9-ad98-4f20-919c-56905e179bad\") " pod="e2e-models-e2e-worker-w1-1874d8/simulator-w1-1874d8-kserve-85dc6d466-lkxlz" Nov 04 05:00:21.759043 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:21.759028 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4655c4c9-ad98-4f20-919c-56905e179bad-tls-certs\") pod \"simulator-w1-1874d8-kserve-85dc6d466-lkxlz\" (UID: \"4655c4c9-ad98-4f20-919c-56905e179bad\") " pod="e2e-models-e2e-worker-w1-1874d8/simulator-w1-1874d8-kserve-85dc6d466-lkxlz" Nov 04 05:00:21.813665 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:21.813618 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcqtt\" (UniqueName: \"kubernetes.io/projected/4655c4c9-ad98-4f20-919c-56905e179bad-kube-api-access-wcqtt\") pod \"simulator-w1-1874d8-kserve-85dc6d466-lkxlz\" (UID: \"4655c4c9-ad98-4f20-919c-56905e179bad\") " pod="e2e-models-e2e-worker-w1-1874d8/simulator-w1-1874d8-kserve-85dc6d466-lkxlz" Nov 04 05:00:21.840752 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:21.840728 4509 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29896140-kkj8l"] Nov 04 05:00:21.875022 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:21.874998 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-1874d8/simulator-w1-1874d8-kserve-85dc6d466-lkxlz" Nov 04 05:00:22.246842 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:22.246819 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-1874d8/simulator-w1-1874d8-kserve-85dc6d466-lkxlz"] Nov 04 05:00:22.247927 ip-10-0-133-180 kubenswrapper[4509]: W1104 05:00:22.247901 4509 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4655c4c9_ad98_4f20_919c_56905e179bad.slice/crio-cc13bec347afb47dc11fef8ec243927089bea9c1a5584f566c85c9cbe4f5a68b WatchSource:0}: Error finding container cc13bec347afb47dc11fef8ec243927089bea9c1a5584f566c85c9cbe4f5a68b: Status 404 returned error can't find the container with id cc13bec347afb47dc11fef8ec243927089bea9c1a5584f566c85c9cbe4f5a68b Nov 04 05:00:22.262961 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:22.262942 4509 scope.go:122] "RemoveContainer" containerID="e2cfc2ac4b8dd19ca3b0e9624e242f420393e5569506eef5d0de4f994ad43358" Nov 04 05:00:22.732388 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:22.732366 4509 generic.go:356] "Generic (PLEG): container finished" podID="4baf9fe5-84d1-48a1-8c10-8080a1e6a5db" containerID="a5c50c3923ee57392d00f4d549ad21fe0067a16b5bca160393dacf35ae83a6b0" exitCode=60 Nov 04 05:00:22.732726 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:22.732451 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-cedfca-29896140-k8xk6" event={"ID":"4baf9fe5-84d1-48a1-8c10-8080a1e6a5db","Type":"ContainerDied","Data":"a5c50c3923ee57392d00f4d549ad21fe0067a16b5bca160393dacf35ae83a6b0"} Nov 04 05:00:22.732726 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:22.732521 4509 scope.go:122] "RemoveContainer" containerID="7e574908985decb9a1152414e6d313997be1f928e84f0acd06980a420db4bb04" Nov 04 05:00:22.732839 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:22.732818 4509 scope.go:122] "RemoveContainer" containerID="a5c50c3923ee57392d00f4d549ad21fe0067a16b5bca160393dacf35ae83a6b0" Nov 04 05:00:22.733027 ip-10-0-133-180 kubenswrapper[4509]: E1104 05:00:22.733003 4509 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-cedfca-29896140-k8xk6_odh-ai-gateway-infra(4baf9fe5-84d1-48a1-8c10-8080a1e6a5db)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-cedfca-29896140-k8xk6" podUID="4baf9fe5-84d1-48a1-8c10-8080a1e6a5db" Nov 04 05:00:22.733970 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:22.733945 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-1874d8/simulator-w1-1874d8-kserve-85dc6d466-lkxlz" event={"ID":"4655c4c9-ad98-4f20-919c-56905e179bad","Type":"ContainerStarted","Data":"c7d9a605c5402d9e1ebb3091313249e4dc823474049b155d63ba61482479d721"} Nov 04 05:00:22.734068 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:22.733984 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-1874d8/simulator-w1-1874d8-kserve-85dc6d466-lkxlz" event={"ID":"4655c4c9-ad98-4f20-919c-56905e179bad","Type":"ContainerStarted","Data":"cc13bec347afb47dc11fef8ec243927089bea9c1a5584f566c85c9cbe4f5a68b"} Nov 04 05:00:22.735548 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:22.735527 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-370078-29896140-9tcm2" event={"ID":"7114f0d1-b384-4bca-b342-29688c0ec8a5","Type":"ContainerStarted","Data":"d58e1f71d3696612a1edaf6450137358c1e6e711e597792213c003ce4dd36ba7"} Nov 04 05:00:22.863505 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:22.863473 4509 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-cedfca-29896140-k8xk6"] Nov 04 05:00:22.874074 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:22.874054 4509 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29896140-kkj8l" Nov 04 05:00:22.875802 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:22.875785 4509 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-1874d8/simulator-w1-1874d8-kserve-85dc6d466-lkxlz" Nov 04 05:00:22.961544 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:22.961501 4509 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-1874d8/simulator-w1-1874d8-kserve-85dc6d466-lkxlz" podStartSLOduration=1.961489928 podStartE2EDuration="1.961489928s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 05:00:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 05:00:22.958816366 +0000 UTC m=+786.836539885" watchObservedRunningTime="2026-11-04 05:00:22.961489928 +0000 UTC m=+786.839213447" Nov 04 05:00:22.968687 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:22.968666 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7124f27c-88a3-4467-8793-cf58ae5c06d2-kube-api-access-sdcgh\" (UniqueName: \"kubernetes.io/projected/7124f27c-88a3-4467-8793-cf58ae5c06d2-kube-api-access-sdcgh\") pod \"7124f27c-88a3-4467-8793-cf58ae5c06d2\" (UID: \"7124f27c-88a3-4467-8793-cf58ae5c06d2\") " Nov 04 05:00:22.968818 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:22.968804 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7124f27c-88a3-4467-8793-cf58ae5c06d2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7124f27c-88a3-4467-8793-cf58ae5c06d2-maas-api-service-ca\") pod \"7124f27c-88a3-4467-8793-cf58ae5c06d2\" (UID: \"7124f27c-88a3-4467-8793-cf58ae5c06d2\") " Nov 04 05:00:22.969112 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:22.969091 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7124f27c-88a3-4467-8793-cf58ae5c06d2-maas-api-service-ca" pod "7124f27c-88a3-4467-8793-cf58ae5c06d2" (UID: "7124f27c-88a3-4467-8793-cf58ae5c06d2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 05:00:22.970758 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:22.970736 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7124f27c-88a3-4467-8793-cf58ae5c06d2-kube-api-access-sdcgh" pod "7124f27c-88a3-4467-8793-cf58ae5c06d2" (UID: "7124f27c-88a3-4467-8793-cf58ae5c06d2"). InnerVolumeSpecName "kube-api-access-sdcgh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 05:00:23.079727 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:23.079202 4509 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7124f27c-88a3-4467-8793-cf58ae5c06d2-maas-api-service-ca\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:00:23.079727 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:23.079253 4509 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdcgh\" (UniqueName: \"kubernetes.io/projected/7124f27c-88a3-4467-8793-cf58ae5c06d2-kube-api-access-sdcgh\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:00:23.080884 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:23.080820 4509 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-370078-29896140-9tcm2" podStartSLOduration=17.061604409 podStartE2EDuration="23.080806124s" totalImagesPullingTime="6.019201715s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 05:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 05:00:23.079668124 +0000 UTC m=+786.957391644" watchObservedRunningTime="2026-11-04 05:00:23.080806124 +0000 UTC m=+786.958529644" Nov 04 05:00:23.255642 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:23.255612 4509 scope.go:122] "RemoveContainer" containerID="72ca0c0672e67913b3cc0e5e9047f6728bec5bb4ff859f9a9246a5ce3f8030ea" Nov 04 05:00:23.746793 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:23.746762 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dab70d-29896140-w2fbg" event={"ID":"467298da-84d2-45fa-bf41-c09422256efa","Type":"ContainerStarted","Data":"77707af9e87c60eac37daf21140b9e656949a7a1370f8801db531aa6042ca01d"} Nov 04 05:00:23.748516 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:23.748493 4509 generic.go:356] "Generic (PLEG): container finished" podID="7114f0d1-b384-4bca-b342-29688c0ec8a5" containerID="d58e1f71d3696612a1edaf6450137358c1e6e711e597792213c003ce4dd36ba7" exitCode=60 Nov 04 05:00:23.748635 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:23.748559 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-370078-29896140-9tcm2" event={"ID":"7114f0d1-b384-4bca-b342-29688c0ec8a5","Type":"ContainerDied","Data":"d58e1f71d3696612a1edaf6450137358c1e6e711e597792213c003ce4dd36ba7"} Nov 04 05:00:23.748635 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:23.748600 4509 scope.go:122] "RemoveContainer" containerID="e2cfc2ac4b8dd19ca3b0e9624e242f420393e5569506eef5d0de4f994ad43358" Nov 04 05:00:23.748923 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:23.748905 4509 scope.go:122] "RemoveContainer" containerID="d58e1f71d3696612a1edaf6450137358c1e6e711e597792213c003ce4dd36ba7" Nov 04 05:00:23.749136 ip-10-0-133-180 kubenswrapper[4509]: E1104 05:00:23.749117 4509 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-370078-29896140-9tcm2_odh-ai-gateway-infra(7114f0d1-b384-4bca-b342-29688c0ec8a5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-370078-29896140-9tcm2" podUID="7114f0d1-b384-4bca-b342-29688c0ec8a5" Nov 04 05:00:23.749869 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:23.749853 4509 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29896140-kkj8l" Nov 04 05:00:23.750003 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:23.749883 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29896140-kkj8l" event={"ID":"7124f27c-88a3-4467-8793-cf58ae5c06d2","Type":"ContainerDied","Data":"52546f56ce45b1d74f65566078772d8a8dfbc66a18b86c2759024254180db309"} Nov 04 05:00:23.760333 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:23.760259 4509 scope.go:122] "RemoveContainer" containerID="96729a219ee54d0eb74bf617eec775dcb6fd79dfe27365648afa569acabdd5fe" Nov 04 05:00:23.875602 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:23.875583 4509 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-cedfca-29896140-k8xk6" Nov 04 05:00:23.988224 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:23.988201 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4baf9fe5-84d1-48a1-8c10-8080a1e6a5db-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4baf9fe5-84d1-48a1-8c10-8080a1e6a5db-maas-api-service-ca\") pod \"4baf9fe5-84d1-48a1-8c10-8080a1e6a5db\" (UID: \"4baf9fe5-84d1-48a1-8c10-8080a1e6a5db\") " Nov 04 05:00:23.988311 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:23.988272 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4baf9fe5-84d1-48a1-8c10-8080a1e6a5db-kube-api-access-2fjfh\" (UniqueName: \"kubernetes.io/projected/4baf9fe5-84d1-48a1-8c10-8080a1e6a5db-kube-api-access-2fjfh\") pod \"4baf9fe5-84d1-48a1-8c10-8080a1e6a5db\" (UID: \"4baf9fe5-84d1-48a1-8c10-8080a1e6a5db\") " Nov 04 05:00:23.988523 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:23.988502 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4baf9fe5-84d1-48a1-8c10-8080a1e6a5db-maas-api-service-ca" pod "4baf9fe5-84d1-48a1-8c10-8080a1e6a5db" (UID: "4baf9fe5-84d1-48a1-8c10-8080a1e6a5db"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 05:00:23.990433 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:23.990412 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4baf9fe5-84d1-48a1-8c10-8080a1e6a5db-kube-api-access-2fjfh" pod "4baf9fe5-84d1-48a1-8c10-8080a1e6a5db" (UID: "4baf9fe5-84d1-48a1-8c10-8080a1e6a5db"). InnerVolumeSpecName "kube-api-access-2fjfh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 05:00:24.083365 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:24.083345 4509 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-370078-29896140-9tcm2"] Nov 04 05:00:24.089337 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:24.089303 4509 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2fjfh\" (UniqueName: \"kubernetes.io/projected/4baf9fe5-84d1-48a1-8c10-8080a1e6a5db-kube-api-access-2fjfh\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:00:24.089406 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:24.089352 4509 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4baf9fe5-84d1-48a1-8c10-8080a1e6a5db-maas-api-service-ca\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:00:24.204946 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:24.204924 4509 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29896140-kkj8l"] Nov 04 05:00:24.209558 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:24.209539 4509 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29896140-kkj8l"] Nov 04 05:00:24.255682 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:24.255662 4509 scope.go:122] "RemoveContainer" containerID="d49871543851bede258765d156d4034987e9d6fa6937f66f033dde346f5a1322" Nov 04 05:00:24.263016 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:24.262995 4509 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7124f27c-88a3-4467-8793-cf58ae5c06d2" path="/var/lib/kubelet/pods/7124f27c-88a3-4467-8793-cf58ae5c06d2/volumes" Nov 04 05:00:24.756394 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:24.756371 4509 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-cedfca-29896140-k8xk6" Nov 04 05:00:24.756766 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:24.756391 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-cedfca-29896140-k8xk6" event={"ID":"4baf9fe5-84d1-48a1-8c10-8080a1e6a5db","Type":"ContainerDied","Data":"41260b5a4f9d0346b0abe8a8f433932a6d47328536b5ef15f8ac66b3b6d1fa9c"} Nov 04 05:00:24.756766 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:24.756468 4509 scope.go:122] "RemoveContainer" containerID="a5c50c3923ee57392d00f4d549ad21fe0067a16b5bca160393dacf35ae83a6b0" Nov 04 05:00:24.758551 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:24.758484 4509 generic.go:356] "Generic (PLEG): container finished" podID="467298da-84d2-45fa-bf41-c09422256efa" containerID="77707af9e87c60eac37daf21140b9e656949a7a1370f8801db531aa6042ca01d" exitCode=60 Nov 04 05:00:24.758551 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:24.758555 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dab70d-29896140-w2fbg" event={"ID":"467298da-84d2-45fa-bf41-c09422256efa","Type":"ContainerDied","Data":"77707af9e87c60eac37daf21140b9e656949a7a1370f8801db531aa6042ca01d"} Nov 04 05:00:24.758869 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:24.758849 4509 scope.go:122] "RemoveContainer" containerID="77707af9e87c60eac37daf21140b9e656949a7a1370f8801db531aa6042ca01d" Nov 04 05:00:24.759063 ip-10-0-133-180 kubenswrapper[4509]: E1104 05:00:24.759037 4509 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-dab70d-29896140-w2fbg_odh-ai-gateway-infra(467298da-84d2-45fa-bf41-c09422256efa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dab70d-29896140-w2fbg" podUID="467298da-84d2-45fa-bf41-c09422256efa" Nov 04 05:00:24.762631 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:24.762610 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1874d8-29896140-jrzlz" event={"ID":"941eb4d4-9d72-412b-b5ac-1bfbd65b439c","Type":"ContainerStarted","Data":"cd05cd8d92186efab19625c2058cbba4c94754f63105cfbd3466799816a458ac"} Nov 04 05:00:24.772952 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:24.772926 4509 scope.go:122] "RemoveContainer" containerID="72ca0c0672e67913b3cc0e5e9047f6728bec5bb4ff859f9a9246a5ce3f8030ea" Nov 04 05:00:24.899386 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:24.899365 4509 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-370078-29896140-9tcm2" Nov 04 05:00:24.919947 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:24.919924 4509 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-cedfca-29896140-k8xk6"] Nov 04 05:00:24.973390 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:24.973369 4509 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-cedfca-29896140-k8xk6"] Nov 04 05:00:24.990849 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:24.990828 4509 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dab70d-29896140-w2fbg"] Nov 04 05:00:24.996957 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:24.996938 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7114f0d1-b384-4bca-b342-29688c0ec8a5-kube-api-access-r6tbv\" (UniqueName: \"kubernetes.io/projected/7114f0d1-b384-4bca-b342-29688c0ec8a5-kube-api-access-r6tbv\") pod \"7114f0d1-b384-4bca-b342-29688c0ec8a5\" (UID: \"7114f0d1-b384-4bca-b342-29688c0ec8a5\") " Nov 04 05:00:24.997049 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:24.997024 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7114f0d1-b384-4bca-b342-29688c0ec8a5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7114f0d1-b384-4bca-b342-29688c0ec8a5-maas-api-service-ca\") pod \"7114f0d1-b384-4bca-b342-29688c0ec8a5\" (UID: \"7114f0d1-b384-4bca-b342-29688c0ec8a5\") " Nov 04 05:00:24.997366 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:24.997342 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7114f0d1-b384-4bca-b342-29688c0ec8a5-maas-api-service-ca" pod "7114f0d1-b384-4bca-b342-29688c0ec8a5" (UID: "7114f0d1-b384-4bca-b342-29688c0ec8a5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 05:00:24.998886 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:24.998867 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7114f0d1-b384-4bca-b342-29688c0ec8a5-kube-api-access-r6tbv" pod "7114f0d1-b384-4bca-b342-29688c0ec8a5" (UID: "7114f0d1-b384-4bca-b342-29688c0ec8a5"). InnerVolumeSpecName "kube-api-access-r6tbv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 05:00:25.098256 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:25.098233 4509 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6tbv\" (UniqueName: \"kubernetes.io/projected/7114f0d1-b384-4bca-b342-29688c0ec8a5-kube-api-access-r6tbv\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:00:25.098256 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:25.098254 4509 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7114f0d1-b384-4bca-b342-29688c0ec8a5-maas-api-service-ca\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:00:25.772897 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:25.772825 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-370078-29896140-9tcm2" event={"ID":"7114f0d1-b384-4bca-b342-29688c0ec8a5","Type":"ContainerDied","Data":"2a0e48fe3d0e8eaeeb30896969db6ec58ef4e83214952a16a791751565c127d7"} Nov 04 05:00:25.772897 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:25.772866 4509 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-370078-29896140-9tcm2" Nov 04 05:00:25.773394 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:25.772927 4509 scope.go:122] "RemoveContainer" containerID="d58e1f71d3696612a1edaf6450137358c1e6e711e597792213c003ce4dd36ba7" Nov 04 05:00:25.775343 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:25.775304 4509 generic.go:356] "Generic (PLEG): container finished" podID="941eb4d4-9d72-412b-b5ac-1bfbd65b439c" containerID="cd05cd8d92186efab19625c2058cbba4c94754f63105cfbd3466799816a458ac" exitCode=60 Nov 04 05:00:25.775484 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:25.775349 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1874d8-29896140-jrzlz" event={"ID":"941eb4d4-9d72-412b-b5ac-1bfbd65b439c","Type":"ContainerDied","Data":"cd05cd8d92186efab19625c2058cbba4c94754f63105cfbd3466799816a458ac"} Nov 04 05:00:25.775832 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:25.775813 4509 scope.go:122] "RemoveContainer" containerID="cd05cd8d92186efab19625c2058cbba4c94754f63105cfbd3466799816a458ac" Nov 04 05:00:25.776066 ip-10-0-133-180 kubenswrapper[4509]: E1104 05:00:25.776045 4509 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-1874d8-29896140-jrzlz_odh-ai-gateway-infra(941eb4d4-9d72-412b-b5ac-1bfbd65b439c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1874d8-29896140-jrzlz" podUID="941eb4d4-9d72-412b-b5ac-1bfbd65b439c" Nov 04 05:00:25.783494 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:25.783474 4509 scope.go:122] "RemoveContainer" containerID="d49871543851bede258765d156d4034987e9d6fa6937f66f033dde346f5a1322" Nov 04 05:00:25.913408 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:25.913388 4509 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dab70d-29896140-w2fbg" Nov 04 05:00:25.995748 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:25.995726 4509 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-370078-29896140-9tcm2"] Nov 04 05:00:26.000045 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:26.000026 4509 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-370078-29896140-9tcm2"] Nov 04 05:00:26.010806 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:26.010784 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/467298da-84d2-45fa-bf41-c09422256efa-kube-api-access-4n9xv\" (UniqueName: \"kubernetes.io/projected/467298da-84d2-45fa-bf41-c09422256efa-kube-api-access-4n9xv\") pod \"467298da-84d2-45fa-bf41-c09422256efa\" (UID: \"467298da-84d2-45fa-bf41-c09422256efa\") " Nov 04 05:00:26.010893 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:26.010882 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/467298da-84d2-45fa-bf41-c09422256efa-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/467298da-84d2-45fa-bf41-c09422256efa-maas-api-service-ca\") pod \"467298da-84d2-45fa-bf41-c09422256efa\" (UID: \"467298da-84d2-45fa-bf41-c09422256efa\") " Nov 04 05:00:26.011185 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:26.011164 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/467298da-84d2-45fa-bf41-c09422256efa-maas-api-service-ca" pod "467298da-84d2-45fa-bf41-c09422256efa" (UID: "467298da-84d2-45fa-bf41-c09422256efa"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 05:00:26.012789 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:26.012768 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/467298da-84d2-45fa-bf41-c09422256efa-kube-api-access-4n9xv" pod "467298da-84d2-45fa-bf41-c09422256efa" (UID: "467298da-84d2-45fa-bf41-c09422256efa"). InnerVolumeSpecName "kube-api-access-4n9xv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 05:00:26.096481 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:26.096410 4509 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1874d8-29896140-jrzlz"] Nov 04 05:00:26.112148 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:26.112120 4509 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/467298da-84d2-45fa-bf41-c09422256efa-maas-api-service-ca\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:00:26.112148 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:26.112140 4509 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4n9xv\" (UniqueName: \"kubernetes.io/projected/467298da-84d2-45fa-bf41-c09422256efa-kube-api-access-4n9xv\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:00:26.261836 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:26.261807 4509 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4baf9fe5-84d1-48a1-8c10-8080a1e6a5db" path="/var/lib/kubelet/pods/4baf9fe5-84d1-48a1-8c10-8080a1e6a5db/volumes" Nov 04 05:00:26.262174 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:26.262162 4509 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7114f0d1-b384-4bca-b342-29688c0ec8a5" path="/var/lib/kubelet/pods/7114f0d1-b384-4bca-b342-29688c0ec8a5/volumes" Nov 04 05:00:26.781616 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:26.781595 4509 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dab70d-29896140-w2fbg" Nov 04 05:00:26.782006 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:26.781591 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dab70d-29896140-w2fbg" event={"ID":"467298da-84d2-45fa-bf41-c09422256efa","Type":"ContainerDied","Data":"4f9dff1cba7967ec37e9a3c02dbba722f24b25e169057495920cc225b454eab9"} Nov 04 05:00:26.782006 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:26.781711 4509 scope.go:122] "RemoveContainer" containerID="77707af9e87c60eac37daf21140b9e656949a7a1370f8801db531aa6042ca01d" Nov 04 05:00:26.922926 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:26.922907 4509 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1874d8-29896140-jrzlz" Nov 04 05:00:27.019893 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:27.019872 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/941eb4d4-9d72-412b-b5ac-1bfbd65b439c-kube-api-access-mrlgk\" (UniqueName: \"kubernetes.io/projected/941eb4d4-9d72-412b-b5ac-1bfbd65b439c-kube-api-access-mrlgk\") pod \"941eb4d4-9d72-412b-b5ac-1bfbd65b439c\" (UID: \"941eb4d4-9d72-412b-b5ac-1bfbd65b439c\") " Nov 04 05:00:27.019989 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:27.019972 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/941eb4d4-9d72-412b-b5ac-1bfbd65b439c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/941eb4d4-9d72-412b-b5ac-1bfbd65b439c-maas-api-service-ca\") pod \"941eb4d4-9d72-412b-b5ac-1bfbd65b439c\" (UID: \"941eb4d4-9d72-412b-b5ac-1bfbd65b439c\") " Nov 04 05:00:27.020278 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:27.020259 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/941eb4d4-9d72-412b-b5ac-1bfbd65b439c-maas-api-service-ca" pod "941eb4d4-9d72-412b-b5ac-1bfbd65b439c" (UID: "941eb4d4-9d72-412b-b5ac-1bfbd65b439c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 05:00:27.020387 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:27.020375 4509 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/941eb4d4-9d72-412b-b5ac-1bfbd65b439c-maas-api-service-ca\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:00:27.021870 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:27.021849 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/941eb4d4-9d72-412b-b5ac-1bfbd65b439c-kube-api-access-mrlgk" pod "941eb4d4-9d72-412b-b5ac-1bfbd65b439c" (UID: "941eb4d4-9d72-412b-b5ac-1bfbd65b439c"). InnerVolumeSpecName "kube-api-access-mrlgk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 05:00:27.057690 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:27.057641 4509 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dab70d-29896140-w2fbg"] Nov 04 05:00:27.061535 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:27.061513 4509 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dab70d-29896140-w2fbg"] Nov 04 05:00:27.121456 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:27.121435 4509 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrlgk\" (UniqueName: \"kubernetes.io/projected/941eb4d4-9d72-412b-b5ac-1bfbd65b439c-kube-api-access-mrlgk\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:00:27.791255 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:27.791222 4509 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1874d8-29896140-jrzlz" Nov 04 05:00:27.791617 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:27.791268 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1874d8-29896140-jrzlz" event={"ID":"941eb4d4-9d72-412b-b5ac-1bfbd65b439c","Type":"ContainerDied","Data":"3e1cf0dfc04cf257070403826ead76843bbaccf297be2d9d6db3bdcacd34644b"} Nov 04 05:00:27.791617 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:27.791334 4509 scope.go:122] "RemoveContainer" containerID="cd05cd8d92186efab19625c2058cbba4c94754f63105cfbd3466799816a458ac" Nov 04 05:00:27.955284 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:27.955265 4509 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1874d8-29896140-jrzlz"] Nov 04 05:00:28.004749 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:28.004729 4509 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1874d8-29896140-jrzlz"] Nov 04 05:00:28.261479 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:28.261452 4509 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="467298da-84d2-45fa-bf41-c09422256efa" path="/var/lib/kubelet/pods/467298da-84d2-45fa-bf41-c09422256efa/volumes" Nov 04 05:00:28.261878 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:28.261864 4509 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="941eb4d4-9d72-412b-b5ac-1bfbd65b439c" path="/var/lib/kubelet/pods/941eb4d4-9d72-412b-b5ac-1bfbd65b439c/volumes" Nov 04 05:00:28.449177 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:28.449140 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-42dnm"] Nov 04 05:00:31.875368 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:31.875313 4509 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-1874d8/simulator-w1-1874d8-kserve-85dc6d466-lkxlz" Nov 04 05:00:31.886878 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:31.886856 4509 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-1874d8/simulator-w1-1874d8-kserve-85dc6d466-lkxlz" Nov 04 05:00:32.826737 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:32.826711 4509 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-1874d8/simulator-w1-1874d8-kserve-85dc6d466-lkxlz" Nov 04 05:00:44.453801 ip-10-0-133-180 kubenswrapper[4509]: E1104 05:00:44.453761 4509 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmdhl" containerName="extractor" Nov 04 05:00:46.742710 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:46.742672 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-42dnm"] Nov 04 05:00:57.387782 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:57.387753 4509 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-4cce33/premium-w5-4cce33-kserve-5875cf9fc6-9k7jz"] Nov 04 05:00:57.388373 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:57.388356 4509 cpu_manager.go:414] "containerMap: removing container" podUID="467298da-84d2-45fa-bf41-c09422256efa" containerName="cleanup" Nov 04 05:00:57.388421 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:57.388374 4509 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="467298da-84d2-45fa-bf41-c09422256efa" containerName="cleanup" Nov 04 05:00:57.388421 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:57.388385 4509 cpu_manager.go:414] "containerMap: removing container" podUID="941eb4d4-9d72-412b-b5ac-1bfbd65b439c" containerName="cleanup" Nov 04 05:00:57.388421 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:57.388390 4509 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="941eb4d4-9d72-412b-b5ac-1bfbd65b439c" containerName="cleanup" Nov 04 05:00:57.388421 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:57.388398 4509 cpu_manager.go:414] "containerMap: removing container" podUID="467298da-84d2-45fa-bf41-c09422256efa" containerName="cleanup" Nov 04 05:00:57.388421 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:57.388403 4509 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="467298da-84d2-45fa-bf41-c09422256efa" containerName="cleanup" Nov 04 05:00:57.388421 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:57.388416 4509 cpu_manager.go:414] "containerMap: removing container" podUID="7124f27c-88a3-4467-8793-cf58ae5c06d2" containerName="cleanup" Nov 04 05:00:57.388421 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:57.388421 4509 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7124f27c-88a3-4467-8793-cf58ae5c06d2" containerName="cleanup" Nov 04 05:00:57.388628 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:57.388427 4509 cpu_manager.go:414] "containerMap: removing container" podUID="7124f27c-88a3-4467-8793-cf58ae5c06d2" containerName="cleanup" Nov 04 05:00:57.388628 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:57.388438 4509 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7124f27c-88a3-4467-8793-cf58ae5c06d2" containerName="cleanup" Nov 04 05:00:57.388628 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:57.388444 4509 cpu_manager.go:414] "containerMap: removing container" podUID="7124f27c-88a3-4467-8793-cf58ae5c06d2" containerName="cleanup" Nov 04 05:00:57.388628 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:57.388450 4509 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7124f27c-88a3-4467-8793-cf58ae5c06d2" containerName="cleanup" Nov 04 05:00:57.388628 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:57.388461 4509 cpu_manager.go:414] "containerMap: removing container" podUID="4baf9fe5-84d1-48a1-8c10-8080a1e6a5db" containerName="cleanup" Nov 04 05:00:57.388628 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:57.388465 4509 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4baf9fe5-84d1-48a1-8c10-8080a1e6a5db" containerName="cleanup" Nov 04 05:00:57.388628 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:57.388472 4509 cpu_manager.go:414] "containerMap: removing container" podUID="941eb4d4-9d72-412b-b5ac-1bfbd65b439c" containerName="cleanup" Nov 04 05:00:57.388628 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:57.388477 4509 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="941eb4d4-9d72-412b-b5ac-1bfbd65b439c" containerName="cleanup" Nov 04 05:00:57.388628 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:57.388489 4509 cpu_manager.go:414] "containerMap: removing container" podUID="7114f0d1-b384-4bca-b342-29688c0ec8a5" containerName="cleanup" Nov 04 05:00:57.388628 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:57.388493 4509 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7114f0d1-b384-4bca-b342-29688c0ec8a5" containerName="cleanup" Nov 04 05:00:57.388628 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:57.388502 4509 cpu_manager.go:414] "containerMap: removing container" podUID="7114f0d1-b384-4bca-b342-29688c0ec8a5" containerName="cleanup" Nov 04 05:00:57.388628 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:57.388507 4509 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7114f0d1-b384-4bca-b342-29688c0ec8a5" containerName="cleanup" Nov 04 05:00:57.388628 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:57.388515 4509 cpu_manager.go:414] "containerMap: removing container" podUID="4baf9fe5-84d1-48a1-8c10-8080a1e6a5db" containerName="cleanup" Nov 04 05:00:57.388628 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:57.388519 4509 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4baf9fe5-84d1-48a1-8c10-8080a1e6a5db" containerName="cleanup" Nov 04 05:00:57.388628 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:57.388524 4509 cpu_manager.go:414] "containerMap: removing container" podUID="7114f0d1-b384-4bca-b342-29688c0ec8a5" containerName="cleanup" Nov 04 05:00:57.388628 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:57.388528 4509 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7114f0d1-b384-4bca-b342-29688c0ec8a5" containerName="cleanup" Nov 04 05:00:57.388628 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:57.388604 4509 memory_manager.go:365] "RemoveStaleState removing state" podUID="7114f0d1-b384-4bca-b342-29688c0ec8a5" containerName="cleanup" Nov 04 05:00:57.388628 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:57.388610 4509 memory_manager.go:365] "RemoveStaleState removing state" podUID="941eb4d4-9d72-412b-b5ac-1bfbd65b439c" containerName="cleanup" Nov 04 05:00:57.388628 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:57.388616 4509 memory_manager.go:365] "RemoveStaleState removing state" podUID="467298da-84d2-45fa-bf41-c09422256efa" containerName="cleanup" Nov 04 05:00:57.388628 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:57.388622 4509 memory_manager.go:365] "RemoveStaleState removing state" podUID="7114f0d1-b384-4bca-b342-29688c0ec8a5" containerName="cleanup" Nov 04 05:00:57.388628 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:57.388626 4509 memory_manager.go:365] "RemoveStaleState removing state" podUID="4baf9fe5-84d1-48a1-8c10-8080a1e6a5db" containerName="cleanup" Nov 04 05:00:57.388628 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:57.388631 4509 memory_manager.go:365] "RemoveStaleState removing state" podUID="7114f0d1-b384-4bca-b342-29688c0ec8a5" containerName="cleanup" Nov 04 05:00:57.388628 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:57.388638 4509 memory_manager.go:365] "RemoveStaleState removing state" podUID="467298da-84d2-45fa-bf41-c09422256efa" containerName="cleanup" Nov 04 05:00:57.389250 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:57.388644 4509 memory_manager.go:365] "RemoveStaleState removing state" podUID="467298da-84d2-45fa-bf41-c09422256efa" containerName="cleanup" Nov 04 05:00:57.389250 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:57.388649 4509 memory_manager.go:365] "RemoveStaleState removing state" podUID="7124f27c-88a3-4467-8793-cf58ae5c06d2" containerName="cleanup" Nov 04 05:00:57.389250 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:57.388654 4509 memory_manager.go:365] "RemoveStaleState removing state" podUID="7124f27c-88a3-4467-8793-cf58ae5c06d2" containerName="cleanup" Nov 04 05:00:57.389250 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:57.388659 4509 memory_manager.go:365] "RemoveStaleState removing state" podUID="7124f27c-88a3-4467-8793-cf58ae5c06d2" containerName="cleanup" Nov 04 05:00:57.389250 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:57.388666 4509 memory_manager.go:365] "RemoveStaleState removing state" podUID="4baf9fe5-84d1-48a1-8c10-8080a1e6a5db" containerName="cleanup" Nov 04 05:00:57.389250 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:57.388671 4509 memory_manager.go:365] "RemoveStaleState removing state" podUID="941eb4d4-9d72-412b-b5ac-1bfbd65b439c" containerName="cleanup" Nov 04 05:00:57.395196 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:57.395181 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-4cce33/premium-w5-4cce33-kserve-5875cf9fc6-9k7jz" Nov 04 05:00:57.411736 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:57.411714 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d9118e1-aff5-4fb2-8a56-78e3ee9fa3e4-home\") pod \"premium-w5-4cce33-kserve-5875cf9fc6-9k7jz\" (UID: \"0d9118e1-aff5-4fb2-8a56-78e3ee9fa3e4\") " pod="e2e-models-e2e-worker-w5-4cce33/premium-w5-4cce33-kserve-5875cf9fc6-9k7jz" Nov 04 05:00:57.411864 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:57.411742 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85wqk\" (UniqueName: \"kubernetes.io/projected/0d9118e1-aff5-4fb2-8a56-78e3ee9fa3e4-kube-api-access-85wqk\") pod \"premium-w5-4cce33-kserve-5875cf9fc6-9k7jz\" (UID: \"0d9118e1-aff5-4fb2-8a56-78e3ee9fa3e4\") " pod="e2e-models-e2e-worker-w5-4cce33/premium-w5-4cce33-kserve-5875cf9fc6-9k7jz" Nov 04 05:00:57.411864 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:57.411769 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d9118e1-aff5-4fb2-8a56-78e3ee9fa3e4-dshm\") pod \"premium-w5-4cce33-kserve-5875cf9fc6-9k7jz\" (UID: \"0d9118e1-aff5-4fb2-8a56-78e3ee9fa3e4\") " pod="e2e-models-e2e-worker-w5-4cce33/premium-w5-4cce33-kserve-5875cf9fc6-9k7jz" Nov 04 05:00:57.411864 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:57.411833 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d9118e1-aff5-4fb2-8a56-78e3ee9fa3e4-model-cache\") pod \"premium-w5-4cce33-kserve-5875cf9fc6-9k7jz\" (UID: \"0d9118e1-aff5-4fb2-8a56-78e3ee9fa3e4\") " pod="e2e-models-e2e-worker-w5-4cce33/premium-w5-4cce33-kserve-5875cf9fc6-9k7jz" Nov 04 05:00:57.412032 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:57.411954 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d9118e1-aff5-4fb2-8a56-78e3ee9fa3e4-tmp-dir\") pod \"premium-w5-4cce33-kserve-5875cf9fc6-9k7jz\" (UID: \"0d9118e1-aff5-4fb2-8a56-78e3ee9fa3e4\") " pod="e2e-models-e2e-worker-w5-4cce33/premium-w5-4cce33-kserve-5875cf9fc6-9k7jz" Nov 04 05:00:57.412032 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:57.411988 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d9118e1-aff5-4fb2-8a56-78e3ee9fa3e4-tls-certs\") pod \"premium-w5-4cce33-kserve-5875cf9fc6-9k7jz\" (UID: \"0d9118e1-aff5-4fb2-8a56-78e3ee9fa3e4\") " pod="e2e-models-e2e-worker-w5-4cce33/premium-w5-4cce33-kserve-5875cf9fc6-9k7jz" Nov 04 05:00:57.447607 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:57.447578 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-4cce33/premium-w5-4cce33-kserve-5875cf9fc6-9k7jz"] Nov 04 05:00:57.494754 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:57.494725 4509 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-4cce33\"/\"openshift-service-ca.crt\"" Nov 04 05:00:57.494754 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:57.494749 4509 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-4cce33\"/\"kube-root-ca.crt\"" Nov 04 05:00:57.494901 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:57.494813 4509 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-4cce33\"/\"default-dockercfg-6frkd\"" Nov 04 05:00:57.497223 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:57.497207 4509 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-4cce33\"/\"premium-w5-4cce33-kserve-self-signed-certs\"" Nov 04 05:00:57.513544 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:57.513526 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d9118e1-aff5-4fb2-8a56-78e3ee9fa3e4-home\") pod \"premium-w5-4cce33-kserve-5875cf9fc6-9k7jz\" (UID: \"0d9118e1-aff5-4fb2-8a56-78e3ee9fa3e4\") " pod="e2e-models-e2e-worker-w5-4cce33/premium-w5-4cce33-kserve-5875cf9fc6-9k7jz" Nov 04 05:00:57.513647 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:57.513551 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85wqk\" (UniqueName: \"kubernetes.io/projected/0d9118e1-aff5-4fb2-8a56-78e3ee9fa3e4-kube-api-access-85wqk\") pod \"premium-w5-4cce33-kserve-5875cf9fc6-9k7jz\" (UID: \"0d9118e1-aff5-4fb2-8a56-78e3ee9fa3e4\") " pod="e2e-models-e2e-worker-w5-4cce33/premium-w5-4cce33-kserve-5875cf9fc6-9k7jz" Nov 04 05:00:57.513647 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:57.513581 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d9118e1-aff5-4fb2-8a56-78e3ee9fa3e4-dshm\") pod \"premium-w5-4cce33-kserve-5875cf9fc6-9k7jz\" (UID: \"0d9118e1-aff5-4fb2-8a56-78e3ee9fa3e4\") " pod="e2e-models-e2e-worker-w5-4cce33/premium-w5-4cce33-kserve-5875cf9fc6-9k7jz" Nov 04 05:00:57.513647 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:57.513607 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d9118e1-aff5-4fb2-8a56-78e3ee9fa3e4-model-cache\") pod \"premium-w5-4cce33-kserve-5875cf9fc6-9k7jz\" (UID: \"0d9118e1-aff5-4fb2-8a56-78e3ee9fa3e4\") " pod="e2e-models-e2e-worker-w5-4cce33/premium-w5-4cce33-kserve-5875cf9fc6-9k7jz" Nov 04 05:00:57.513789 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:57.513660 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d9118e1-aff5-4fb2-8a56-78e3ee9fa3e4-tmp-dir\") pod \"premium-w5-4cce33-kserve-5875cf9fc6-9k7jz\" (UID: \"0d9118e1-aff5-4fb2-8a56-78e3ee9fa3e4\") " pod="e2e-models-e2e-worker-w5-4cce33/premium-w5-4cce33-kserve-5875cf9fc6-9k7jz" Nov 04 05:00:57.513789 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:57.513683 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d9118e1-aff5-4fb2-8a56-78e3ee9fa3e4-tls-certs\") pod \"premium-w5-4cce33-kserve-5875cf9fc6-9k7jz\" (UID: \"0d9118e1-aff5-4fb2-8a56-78e3ee9fa3e4\") " pod="e2e-models-e2e-worker-w5-4cce33/premium-w5-4cce33-kserve-5875cf9fc6-9k7jz" Nov 04 05:00:57.513985 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:57.513966 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d9118e1-aff5-4fb2-8a56-78e3ee9fa3e4-home\") pod \"premium-w5-4cce33-kserve-5875cf9fc6-9k7jz\" (UID: \"0d9118e1-aff5-4fb2-8a56-78e3ee9fa3e4\") " pod="e2e-models-e2e-worker-w5-4cce33/premium-w5-4cce33-kserve-5875cf9fc6-9k7jz" Nov 04 05:00:57.514056 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:57.514020 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d9118e1-aff5-4fb2-8a56-78e3ee9fa3e4-tmp-dir\") pod \"premium-w5-4cce33-kserve-5875cf9fc6-9k7jz\" (UID: \"0d9118e1-aff5-4fb2-8a56-78e3ee9fa3e4\") " pod="e2e-models-e2e-worker-w5-4cce33/premium-w5-4cce33-kserve-5875cf9fc6-9k7jz" Nov 04 05:00:57.514112 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:57.514090 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d9118e1-aff5-4fb2-8a56-78e3ee9fa3e4-model-cache\") pod \"premium-w5-4cce33-kserve-5875cf9fc6-9k7jz\" (UID: \"0d9118e1-aff5-4fb2-8a56-78e3ee9fa3e4\") " pod="e2e-models-e2e-worker-w5-4cce33/premium-w5-4cce33-kserve-5875cf9fc6-9k7jz" Nov 04 05:00:57.515872 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:57.515854 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d9118e1-aff5-4fb2-8a56-78e3ee9fa3e4-dshm\") pod \"premium-w5-4cce33-kserve-5875cf9fc6-9k7jz\" (UID: \"0d9118e1-aff5-4fb2-8a56-78e3ee9fa3e4\") " pod="e2e-models-e2e-worker-w5-4cce33/premium-w5-4cce33-kserve-5875cf9fc6-9k7jz" Nov 04 05:00:57.516186 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:57.516169 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d9118e1-aff5-4fb2-8a56-78e3ee9fa3e4-tls-certs\") pod \"premium-w5-4cce33-kserve-5875cf9fc6-9k7jz\" (UID: \"0d9118e1-aff5-4fb2-8a56-78e3ee9fa3e4\") " pod="e2e-models-e2e-worker-w5-4cce33/premium-w5-4cce33-kserve-5875cf9fc6-9k7jz" Nov 04 05:00:57.571759 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:57.571727 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85wqk\" (UniqueName: \"kubernetes.io/projected/0d9118e1-aff5-4fb2-8a56-78e3ee9fa3e4-kube-api-access-85wqk\") pod \"premium-w5-4cce33-kserve-5875cf9fc6-9k7jz\" (UID: \"0d9118e1-aff5-4fb2-8a56-78e3ee9fa3e4\") " pod="e2e-models-e2e-worker-w5-4cce33/premium-w5-4cce33-kserve-5875cf9fc6-9k7jz" Nov 04 05:00:57.706473 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:57.706391 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-4cce33/premium-w5-4cce33-kserve-5875cf9fc6-9k7jz" Nov 04 05:00:58.079353 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:58.079312 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-4cce33/premium-w5-4cce33-kserve-5875cf9fc6-9k7jz"] Nov 04 05:00:58.081232 ip-10-0-133-180 kubenswrapper[4509]: W1104 05:00:58.081206 4509 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0d9118e1_aff5_4fb2_8a56_78e3ee9fa3e4.slice/crio-9e9a3584deb1937e8b8b067aa39fbb87aa8e8fb5acfd53b42921d6e223d823c2 WatchSource:0}: Error finding container 9e9a3584deb1937e8b8b067aa39fbb87aa8e8fb5acfd53b42921d6e223d823c2: Status 404 returned error can't find the container with id 9e9a3584deb1937e8b8b067aa39fbb87aa8e8fb5acfd53b42921d6e223d823c2 Nov 04 05:00:58.977002 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:58.976965 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-4cce33/premium-w5-4cce33-kserve-5875cf9fc6-9k7jz" event={"ID":"0d9118e1-aff5-4fb2-8a56-78e3ee9fa3e4","Type":"ContainerStarted","Data":"9aabeaf1e4937576824210ad25e559b17f36aaf258476162d5bfb21b1f1c65cc"} Nov 04 05:00:58.977395 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:58.977017 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-4cce33/premium-w5-4cce33-kserve-5875cf9fc6-9k7jz" event={"ID":"0d9118e1-aff5-4fb2-8a56-78e3ee9fa3e4","Type":"ContainerStarted","Data":"9e9a3584deb1937e8b8b067aa39fbb87aa8e8fb5acfd53b42921d6e223d823c2"} Nov 04 05:00:59.091751 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:59.091708 4509 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-4cce33/premium-w5-4cce33-kserve-5875cf9fc6-9k7jz" podStartSLOduration=2.091695662 podStartE2EDuration="2.091695662s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 05:00:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 05:00:59.090942987 +0000 UTC m=+822.968666509" watchObservedRunningTime="2026-11-04 05:00:59.091695662 +0000 UTC m=+822.969419183" Nov 04 05:00:59.707001 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:00:59.706966 4509 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-4cce33/premium-w5-4cce33-kserve-5875cf9fc6-9k7jz" Nov 04 05:01:07.706916 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:01:07.706874 4509 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-4cce33/premium-w5-4cce33-kserve-5875cf9fc6-9k7jz" Nov 04 05:01:07.719273 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:01:07.719246 4509 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-4cce33/premium-w5-4cce33-kserve-5875cf9fc6-9k7jz" Nov 04 05:01:08.062094 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:01:08.062063 4509 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-4cce33/premium-w5-4cce33-kserve-5875cf9fc6-9k7jz" Nov 04 05:01:14.454431 ip-10-0-133-180 kubenswrapper[4509]: E1104 05:01:14.454397 4509 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmdhl" containerName="extractor" Nov 04 05:01:15.400241 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:01:15.400208 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-42dnm"] Nov 04 05:01:44.456749 ip-10-0-133-180 kubenswrapper[4509]: E1104 05:01:44.456659 4509 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmdhl" containerName="extractor" Nov 04 05:01:47.550439 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:01:47.550394 4509 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-370078/premium-w2-370078-kserve-54d45b6bc-q4xfk"] Nov 04 05:01:47.550965 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:01:47.550938 4509 cpu_manager.go:414] "containerMap: removing container" podUID="4baf9fe5-84d1-48a1-8c10-8080a1e6a5db" containerName="cleanup" Nov 04 05:01:47.550965 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:01:47.550953 4509 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4baf9fe5-84d1-48a1-8c10-8080a1e6a5db" containerName="cleanup" Nov 04 05:01:47.551079 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:01:47.550972 4509 cpu_manager.go:414] "containerMap: removing container" podUID="467298da-84d2-45fa-bf41-c09422256efa" containerName="cleanup" Nov 04 05:01:47.551079 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:01:47.550978 4509 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="467298da-84d2-45fa-bf41-c09422256efa" containerName="cleanup" Nov 04 05:01:47.551079 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:01:47.550996 4509 cpu_manager.go:414] "containerMap: removing container" podUID="941eb4d4-9d72-412b-b5ac-1bfbd65b439c" containerName="cleanup" Nov 04 05:01:47.551079 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:01:47.551000 4509 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="941eb4d4-9d72-412b-b5ac-1bfbd65b439c" containerName="cleanup" Nov 04 05:01:47.551079 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:01:47.551071 4509 memory_manager.go:365] "RemoveStaleState removing state" podUID="941eb4d4-9d72-412b-b5ac-1bfbd65b439c" containerName="cleanup" Nov 04 05:01:47.551079 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:01:47.551079 4509 memory_manager.go:365] "RemoveStaleState removing state" podUID="4baf9fe5-84d1-48a1-8c10-8080a1e6a5db" containerName="cleanup" Nov 04 05:01:47.554213 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:01:47.554188 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-42dnm"] Nov 04 05:01:47.554413 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:01:47.554350 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-370078/premium-w2-370078-kserve-54d45b6bc-q4xfk" Nov 04 05:01:47.608835 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:01:47.608788 4509 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-370078\"/\"default-dockercfg-ksfsj\"" Nov 04 05:01:47.610971 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:01:47.610940 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-370078/premium-w2-370078-kserve-54d45b6bc-q4xfk"] Nov 04 05:01:47.634482 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:01:47.634445 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qj7fh\" (UniqueName: \"kubernetes.io/projected/19bf8535-557c-4598-a6fd-e937faeee299-kube-api-access-qj7fh\") pod \"premium-w2-370078-kserve-54d45b6bc-q4xfk\" (UID: \"19bf8535-557c-4598-a6fd-e937faeee299\") " pod="e2e-models-e2e-worker-w2-370078/premium-w2-370078-kserve-54d45b6bc-q4xfk" Nov 04 05:01:47.634696 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:01:47.634493 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19bf8535-557c-4598-a6fd-e937faeee299-model-cache\") pod \"premium-w2-370078-kserve-54d45b6bc-q4xfk\" (UID: \"19bf8535-557c-4598-a6fd-e937faeee299\") " pod="e2e-models-e2e-worker-w2-370078/premium-w2-370078-kserve-54d45b6bc-q4xfk" Nov 04 05:01:47.634696 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:01:47.634664 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19bf8535-557c-4598-a6fd-e937faeee299-dshm\") pod \"premium-w2-370078-kserve-54d45b6bc-q4xfk\" (UID: \"19bf8535-557c-4598-a6fd-e937faeee299\") " pod="e2e-models-e2e-worker-w2-370078/premium-w2-370078-kserve-54d45b6bc-q4xfk" Nov 04 05:01:47.634823 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:01:47.634698 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19bf8535-557c-4598-a6fd-e937faeee299-home\") pod \"premium-w2-370078-kserve-54d45b6bc-q4xfk\" (UID: \"19bf8535-557c-4598-a6fd-e937faeee299\") " pod="e2e-models-e2e-worker-w2-370078/premium-w2-370078-kserve-54d45b6bc-q4xfk" Nov 04 05:01:47.634823 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:01:47.634729 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19bf8535-557c-4598-a6fd-e937faeee299-tmp-dir\") pod \"premium-w2-370078-kserve-54d45b6bc-q4xfk\" (UID: \"19bf8535-557c-4598-a6fd-e937faeee299\") " pod="e2e-models-e2e-worker-w2-370078/premium-w2-370078-kserve-54d45b6bc-q4xfk" Nov 04 05:01:47.634823 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:01:47.634766 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19bf8535-557c-4598-a6fd-e937faeee299-tls-certs\") pod \"premium-w2-370078-kserve-54d45b6bc-q4xfk\" (UID: \"19bf8535-557c-4598-a6fd-e937faeee299\") " pod="e2e-models-e2e-worker-w2-370078/premium-w2-370078-kserve-54d45b6bc-q4xfk" Nov 04 05:01:47.657522 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:01:47.657480 4509 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-370078\"/\"openshift-service-ca.crt\"" Nov 04 05:01:47.657699 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:01:47.657524 4509 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-370078\"/\"kube-root-ca.crt\"" Nov 04 05:01:47.657699 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:01:47.657545 4509 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-370078\"/\"premium-w2-370078-kserve-self-signed-certs\"" Nov 04 05:01:47.735971 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:01:47.735928 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19bf8535-557c-4598-a6fd-e937faeee299-model-cache\") pod \"premium-w2-370078-kserve-54d45b6bc-q4xfk\" (UID: \"19bf8535-557c-4598-a6fd-e937faeee299\") " pod="e2e-models-e2e-worker-w2-370078/premium-w2-370078-kserve-54d45b6bc-q4xfk" Nov 04 05:01:47.736163 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:01:47.736098 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19bf8535-557c-4598-a6fd-e937faeee299-dshm\") pod \"premium-w2-370078-kserve-54d45b6bc-q4xfk\" (UID: \"19bf8535-557c-4598-a6fd-e937faeee299\") " pod="e2e-models-e2e-worker-w2-370078/premium-w2-370078-kserve-54d45b6bc-q4xfk" Nov 04 05:01:47.736163 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:01:47.736143 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19bf8535-557c-4598-a6fd-e937faeee299-home\") pod \"premium-w2-370078-kserve-54d45b6bc-q4xfk\" (UID: \"19bf8535-557c-4598-a6fd-e937faeee299\") " pod="e2e-models-e2e-worker-w2-370078/premium-w2-370078-kserve-54d45b6bc-q4xfk" Nov 04 05:01:47.736278 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:01:47.736189 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19bf8535-557c-4598-a6fd-e937faeee299-tmp-dir\") pod \"premium-w2-370078-kserve-54d45b6bc-q4xfk\" (UID: \"19bf8535-557c-4598-a6fd-e937faeee299\") " pod="e2e-models-e2e-worker-w2-370078/premium-w2-370078-kserve-54d45b6bc-q4xfk" Nov 04 05:01:47.736278 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:01:47.736247 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19bf8535-557c-4598-a6fd-e937faeee299-tls-certs\") pod \"premium-w2-370078-kserve-54d45b6bc-q4xfk\" (UID: \"19bf8535-557c-4598-a6fd-e937faeee299\") " pod="e2e-models-e2e-worker-w2-370078/premium-w2-370078-kserve-54d45b6bc-q4xfk" Nov 04 05:01:47.736410 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:01:47.736278 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qj7fh\" (UniqueName: \"kubernetes.io/projected/19bf8535-557c-4598-a6fd-e937faeee299-kube-api-access-qj7fh\") pod \"premium-w2-370078-kserve-54d45b6bc-q4xfk\" (UID: \"19bf8535-557c-4598-a6fd-e937faeee299\") " pod="e2e-models-e2e-worker-w2-370078/premium-w2-370078-kserve-54d45b6bc-q4xfk" Nov 04 05:01:47.736410 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:01:47.736390 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19bf8535-557c-4598-a6fd-e937faeee299-model-cache\") pod \"premium-w2-370078-kserve-54d45b6bc-q4xfk\" (UID: \"19bf8535-557c-4598-a6fd-e937faeee299\") " pod="e2e-models-e2e-worker-w2-370078/premium-w2-370078-kserve-54d45b6bc-q4xfk" Nov 04 05:01:47.736555 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:01:47.736537 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19bf8535-557c-4598-a6fd-e937faeee299-home\") pod \"premium-w2-370078-kserve-54d45b6bc-q4xfk\" (UID: \"19bf8535-557c-4598-a6fd-e937faeee299\") " pod="e2e-models-e2e-worker-w2-370078/premium-w2-370078-kserve-54d45b6bc-q4xfk" Nov 04 05:01:47.736629 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:01:47.736608 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19bf8535-557c-4598-a6fd-e937faeee299-tmp-dir\") pod \"premium-w2-370078-kserve-54d45b6bc-q4xfk\" (UID: \"19bf8535-557c-4598-a6fd-e937faeee299\") " pod="e2e-models-e2e-worker-w2-370078/premium-w2-370078-kserve-54d45b6bc-q4xfk" Nov 04 05:01:47.738657 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:01:47.738633 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19bf8535-557c-4598-a6fd-e937faeee299-dshm\") pod \"premium-w2-370078-kserve-54d45b6bc-q4xfk\" (UID: \"19bf8535-557c-4598-a6fd-e937faeee299\") " pod="e2e-models-e2e-worker-w2-370078/premium-w2-370078-kserve-54d45b6bc-q4xfk" Nov 04 05:01:47.738994 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:01:47.738977 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19bf8535-557c-4598-a6fd-e937faeee299-tls-certs\") pod \"premium-w2-370078-kserve-54d45b6bc-q4xfk\" (UID: \"19bf8535-557c-4598-a6fd-e937faeee299\") " pod="e2e-models-e2e-worker-w2-370078/premium-w2-370078-kserve-54d45b6bc-q4xfk" Nov 04 05:01:47.797960 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:01:47.797924 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qj7fh\" (UniqueName: \"kubernetes.io/projected/19bf8535-557c-4598-a6fd-e937faeee299-kube-api-access-qj7fh\") pod \"premium-w2-370078-kserve-54d45b6bc-q4xfk\" (UID: \"19bf8535-557c-4598-a6fd-e937faeee299\") " pod="e2e-models-e2e-worker-w2-370078/premium-w2-370078-kserve-54d45b6bc-q4xfk" Nov 04 05:01:47.875699 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:01:47.875596 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-370078/premium-w2-370078-kserve-54d45b6bc-q4xfk" Nov 04 05:01:48.349357 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:01:48.344762 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-370078/premium-w2-370078-kserve-54d45b6bc-q4xfk"] Nov 04 05:01:49.303918 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:01:49.303877 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-370078/premium-w2-370078-kserve-54d45b6bc-q4xfk" event={"ID":"19bf8535-557c-4598-a6fd-e937faeee299","Type":"ContainerStarted","Data":"387b3c4e818e00fb7a52728f2397e371625f2a8231ae09cd4932f44ee46bf254"} Nov 04 05:01:49.304350 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:01:49.303930 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-370078/premium-w2-370078-kserve-54d45b6bc-q4xfk" event={"ID":"19bf8535-557c-4598-a6fd-e937faeee299","Type":"ContainerStarted","Data":"68569abd741a71b9fd3cb2de2d9fe5943ee8779de9cd0193e284a3d9ddb942a5"} Nov 04 05:01:49.422348 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:01:49.420047 4509 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-370078/premium-w2-370078-kserve-54d45b6bc-q4xfk" podStartSLOduration=2.420029656 podStartE2EDuration="2.420029656s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 05:01:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 05:01:49.419487034 +0000 UTC m=+873.297210556" watchObservedRunningTime="2026-11-04 05:01:49.420029656 +0000 UTC m=+873.297753179" Nov 04 05:01:49.876936 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:01:49.876891 4509 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-370078/premium-w2-370078-kserve-54d45b6bc-q4xfk" Nov 04 05:01:57.876861 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:01:57.876823 4509 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-370078/premium-w2-370078-kserve-54d45b6bc-q4xfk" Nov 04 05:01:57.889488 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:01:57.889456 4509 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-370078/premium-w2-370078-kserve-54d45b6bc-q4xfk" Nov 04 05:01:58.368083 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:01:58.368042 4509 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-370078/premium-w2-370078-kserve-54d45b6bc-q4xfk" Nov 04 05:02:14.455139 ip-10-0-133-180 kubenswrapper[4509]: E1104 05:02:14.455094 4509 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmdhl" containerName="extractor" Nov 04 05:02:21.665494 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:02:21.665455 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-42dnm"] Nov 04 05:02:22.630647 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:02:22.630609 4509 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-dab70d/premium-w3-dab70d-kserve-575d94569f-9q687"] Nov 04 05:02:22.634737 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:02:22.634719 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-dab70d/premium-w3-dab70d-kserve-575d94569f-9q687" Nov 04 05:02:22.688780 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:02:22.688743 4509 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-dab70d\"/\"openshift-service-ca.crt\"" Nov 04 05:02:22.691820 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:02:22.691795 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-dab70d/premium-w3-dab70d-kserve-575d94569f-9q687"] Nov 04 05:02:22.738029 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:02:22.738000 4509 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-dab70d\"/\"kube-root-ca.crt\"" Nov 04 05:02:22.738235 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:02:22.738080 4509 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-dab70d\"/\"premium-w3-dab70d-kserve-self-signed-certs\"" Nov 04 05:02:22.738235 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:02:22.738171 4509 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-dab70d\"/\"default-dockercfg-g6kzk\"" Nov 04 05:02:22.795486 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:02:22.795419 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f12cb461-1adf-4a95-8af6-9cf0fe1c3bbb-model-cache\") pod \"premium-w3-dab70d-kserve-575d94569f-9q687\" (UID: \"f12cb461-1adf-4a95-8af6-9cf0fe1c3bbb\") " pod="e2e-models-e2e-worker-w3-dab70d/premium-w3-dab70d-kserve-575d94569f-9q687" Nov 04 05:02:22.795687 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:02:22.795594 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wk7jj\" (UniqueName: \"kubernetes.io/projected/f12cb461-1adf-4a95-8af6-9cf0fe1c3bbb-kube-api-access-wk7jj\") pod \"premium-w3-dab70d-kserve-575d94569f-9q687\" (UID: \"f12cb461-1adf-4a95-8af6-9cf0fe1c3bbb\") " pod="e2e-models-e2e-worker-w3-dab70d/premium-w3-dab70d-kserve-575d94569f-9q687" Nov 04 05:02:22.796533 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:02:22.796506 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f12cb461-1adf-4a95-8af6-9cf0fe1c3bbb-tmp-dir\") pod \"premium-w3-dab70d-kserve-575d94569f-9q687\" (UID: \"f12cb461-1adf-4a95-8af6-9cf0fe1c3bbb\") " pod="e2e-models-e2e-worker-w3-dab70d/premium-w3-dab70d-kserve-575d94569f-9q687" Nov 04 05:02:22.796670 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:02:22.796566 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f12cb461-1adf-4a95-8af6-9cf0fe1c3bbb-tls-certs\") pod \"premium-w3-dab70d-kserve-575d94569f-9q687\" (UID: \"f12cb461-1adf-4a95-8af6-9cf0fe1c3bbb\") " pod="e2e-models-e2e-worker-w3-dab70d/premium-w3-dab70d-kserve-575d94569f-9q687" Nov 04 05:02:22.796670 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:02:22.796604 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f12cb461-1adf-4a95-8af6-9cf0fe1c3bbb-dshm\") pod \"premium-w3-dab70d-kserve-575d94569f-9q687\" (UID: \"f12cb461-1adf-4a95-8af6-9cf0fe1c3bbb\") " pod="e2e-models-e2e-worker-w3-dab70d/premium-w3-dab70d-kserve-575d94569f-9q687" Nov 04 05:02:22.796670 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:02:22.796650 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f12cb461-1adf-4a95-8af6-9cf0fe1c3bbb-home\") pod \"premium-w3-dab70d-kserve-575d94569f-9q687\" (UID: \"f12cb461-1adf-4a95-8af6-9cf0fe1c3bbb\") " pod="e2e-models-e2e-worker-w3-dab70d/premium-w3-dab70d-kserve-575d94569f-9q687" Nov 04 05:02:22.897711 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:02:22.897623 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wk7jj\" (UniqueName: \"kubernetes.io/projected/f12cb461-1adf-4a95-8af6-9cf0fe1c3bbb-kube-api-access-wk7jj\") pod \"premium-w3-dab70d-kserve-575d94569f-9q687\" (UID: \"f12cb461-1adf-4a95-8af6-9cf0fe1c3bbb\") " pod="e2e-models-e2e-worker-w3-dab70d/premium-w3-dab70d-kserve-575d94569f-9q687" Nov 04 05:02:22.897711 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:02:22.897683 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f12cb461-1adf-4a95-8af6-9cf0fe1c3bbb-tmp-dir\") pod \"premium-w3-dab70d-kserve-575d94569f-9q687\" (UID: \"f12cb461-1adf-4a95-8af6-9cf0fe1c3bbb\") " pod="e2e-models-e2e-worker-w3-dab70d/premium-w3-dab70d-kserve-575d94569f-9q687" Nov 04 05:02:22.897711 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:02:22.897712 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f12cb461-1adf-4a95-8af6-9cf0fe1c3bbb-tls-certs\") pod \"premium-w3-dab70d-kserve-575d94569f-9q687\" (UID: \"f12cb461-1adf-4a95-8af6-9cf0fe1c3bbb\") " pod="e2e-models-e2e-worker-w3-dab70d/premium-w3-dab70d-kserve-575d94569f-9q687" Nov 04 05:02:22.897991 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:02:22.897740 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f12cb461-1adf-4a95-8af6-9cf0fe1c3bbb-dshm\") pod \"premium-w3-dab70d-kserve-575d94569f-9q687\" (UID: \"f12cb461-1adf-4a95-8af6-9cf0fe1c3bbb\") " pod="e2e-models-e2e-worker-w3-dab70d/premium-w3-dab70d-kserve-575d94569f-9q687" Nov 04 05:02:22.897991 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:02:22.897778 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f12cb461-1adf-4a95-8af6-9cf0fe1c3bbb-home\") pod \"premium-w3-dab70d-kserve-575d94569f-9q687\" (UID: \"f12cb461-1adf-4a95-8af6-9cf0fe1c3bbb\") " pod="e2e-models-e2e-worker-w3-dab70d/premium-w3-dab70d-kserve-575d94569f-9q687" Nov 04 05:02:22.897991 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:02:22.897845 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f12cb461-1adf-4a95-8af6-9cf0fe1c3bbb-model-cache\") pod \"premium-w3-dab70d-kserve-575d94569f-9q687\" (UID: \"f12cb461-1adf-4a95-8af6-9cf0fe1c3bbb\") " pod="e2e-models-e2e-worker-w3-dab70d/premium-w3-dab70d-kserve-575d94569f-9q687" Nov 04 05:02:22.898174 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:02:22.898149 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f12cb461-1adf-4a95-8af6-9cf0fe1c3bbb-tmp-dir\") pod \"premium-w3-dab70d-kserve-575d94569f-9q687\" (UID: \"f12cb461-1adf-4a95-8af6-9cf0fe1c3bbb\") " pod="e2e-models-e2e-worker-w3-dab70d/premium-w3-dab70d-kserve-575d94569f-9q687" Nov 04 05:02:22.898239 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:02:22.898222 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f12cb461-1adf-4a95-8af6-9cf0fe1c3bbb-model-cache\") pod \"premium-w3-dab70d-kserve-575d94569f-9q687\" (UID: \"f12cb461-1adf-4a95-8af6-9cf0fe1c3bbb\") " pod="e2e-models-e2e-worker-w3-dab70d/premium-w3-dab70d-kserve-575d94569f-9q687" Nov 04 05:02:22.898291 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:02:22.898253 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f12cb461-1adf-4a95-8af6-9cf0fe1c3bbb-home\") pod \"premium-w3-dab70d-kserve-575d94569f-9q687\" (UID: \"f12cb461-1adf-4a95-8af6-9cf0fe1c3bbb\") " pod="e2e-models-e2e-worker-w3-dab70d/premium-w3-dab70d-kserve-575d94569f-9q687" Nov 04 05:02:22.900339 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:02:22.900300 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f12cb461-1adf-4a95-8af6-9cf0fe1c3bbb-dshm\") pod \"premium-w3-dab70d-kserve-575d94569f-9q687\" (UID: \"f12cb461-1adf-4a95-8af6-9cf0fe1c3bbb\") " pod="e2e-models-e2e-worker-w3-dab70d/premium-w3-dab70d-kserve-575d94569f-9q687" Nov 04 05:02:22.900811 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:02:22.900791 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f12cb461-1adf-4a95-8af6-9cf0fe1c3bbb-tls-certs\") pod \"premium-w3-dab70d-kserve-575d94569f-9q687\" (UID: \"f12cb461-1adf-4a95-8af6-9cf0fe1c3bbb\") " pod="e2e-models-e2e-worker-w3-dab70d/premium-w3-dab70d-kserve-575d94569f-9q687" Nov 04 05:02:22.955398 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:02:22.955366 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wk7jj\" (UniqueName: \"kubernetes.io/projected/f12cb461-1adf-4a95-8af6-9cf0fe1c3bbb-kube-api-access-wk7jj\") pod \"premium-w3-dab70d-kserve-575d94569f-9q687\" (UID: \"f12cb461-1adf-4a95-8af6-9cf0fe1c3bbb\") " pod="e2e-models-e2e-worker-w3-dab70d/premium-w3-dab70d-kserve-575d94569f-9q687" Nov 04 05:02:23.246461 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:02:23.246421 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-dab70d/premium-w3-dab70d-kserve-575d94569f-9q687" Nov 04 05:02:23.621044 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:02:23.621017 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-dab70d/premium-w3-dab70d-kserve-575d94569f-9q687"] Nov 04 05:02:23.622267 ip-10-0-133-180 kubenswrapper[4509]: W1104 05:02:23.622235 4509 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf12cb461_1adf_4a95_8af6_9cf0fe1c3bbb.slice/crio-f31112ad28b60012afbe9908b9087ba4cc4ce8ae0308871555be5110659ad875 WatchSource:0}: Error finding container f31112ad28b60012afbe9908b9087ba4cc4ce8ae0308871555be5110659ad875: Status 404 returned error can't find the container with id f31112ad28b60012afbe9908b9087ba4cc4ce8ae0308871555be5110659ad875 Nov 04 05:02:24.514048 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:02:24.514010 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-dab70d/premium-w3-dab70d-kserve-575d94569f-9q687" event={"ID":"f12cb461-1adf-4a95-8af6-9cf0fe1c3bbb","Type":"ContainerStarted","Data":"8d5a7de60a68c9d03132c998a7be9a336104add480322792a9a2565bb7c9f8bc"} Nov 04 05:02:24.514523 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:02:24.514057 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-dab70d/premium-w3-dab70d-kserve-575d94569f-9q687" event={"ID":"f12cb461-1adf-4a95-8af6-9cf0fe1c3bbb","Type":"ContainerStarted","Data":"f31112ad28b60012afbe9908b9087ba4cc4ce8ae0308871555be5110659ad875"} Nov 04 05:02:24.631057 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:02:24.631005 4509 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-dab70d/premium-w3-dab70d-kserve-575d94569f-9q687" podStartSLOduration=2.630991232 podStartE2EDuration="2.630991232s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 05:02:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 05:02:24.629309007 +0000 UTC m=+908.507032529" watchObservedRunningTime="2026-11-04 05:02:24.630991232 +0000 UTC m=+908.508714753" Nov 04 05:02:25.247464 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:02:25.247425 4509 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-dab70d/premium-w3-dab70d-kserve-575d94569f-9q687" Nov 04 05:02:32.180049 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:02:32.180012 4509 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-j8ns4"] Nov 04 05:02:32.180636 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:02:32.180251 4509 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-j8ns4" podUID="3914303c-ffca-4fa0-bf97-d15430976da6" containerName="istio-proxy" containerID="cri-o://e85fa7471bbfc80c1eafbaace18a80c16a30420142fe26f0bd8d48ce9842caae" gracePeriod=30 Nov 04 05:02:33.247156 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:02:33.247120 4509 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-dab70d/premium-w3-dab70d-kserve-575d94569f-9q687" Nov 04 05:02:33.263713 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:02:33.263687 4509 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-dab70d/premium-w3-dab70d-kserve-575d94569f-9q687" Nov 04 05:02:33.575641 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:02:33.575571 4509 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-dab70d/premium-w3-dab70d-kserve-575d94569f-9q687" Nov 04 05:02:34.843251 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:02:34.843215 4509 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-370078-openshift-default-584c45775-h6rwx"] Nov 04 05:02:34.846942 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:02:34.846927 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-370078-openshift-default-584c45775-h6rwx" Nov 04 05:02:34.902584 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:02:34.902553 4509 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-370078-openshift-default-dockercfg-r9m9c\"" Nov 04 05:02:34.954866 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:02:34.954830 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-370078-openshift-default-584c45775-h6rwx"] Nov 04 05:02:35.028197 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:02:35.028158 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cznd\" (UniqueName: \"kubernetes.io/projected/0dfa90b7-fb2a-439c-a9ca-d92c30a7f088-kube-api-access-8cznd\") pod \"e2e-worker-w2-370078-openshift-default-584c45775-h6rwx\" (UID: \"0dfa90b7-fb2a-439c-a9ca-d92c30a7f088\") " pod="openshift-ingress/e2e-worker-w2-370078-openshift-default-584c45775-h6rwx" Nov 04 05:02:35.028419 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:02:35.028223 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0dfa90b7-fb2a-439c-a9ca-d92c30a7f088-istiod-ca-cert\") pod \"e2e-worker-w2-370078-openshift-default-584c45775-h6rwx\" (UID: \"0dfa90b7-fb2a-439c-a9ca-d92c30a7f088\") " pod="openshift-ingress/e2e-worker-w2-370078-openshift-default-584c45775-h6rwx" Nov 04 05:02:35.028419 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:02:35.028356 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0dfa90b7-fb2a-439c-a9ca-d92c30a7f088-workload-certs\") pod \"e2e-worker-w2-370078-openshift-default-584c45775-h6rwx\" (UID: \"0dfa90b7-fb2a-439c-a9ca-d92c30a7f088\") " pod="openshift-ingress/e2e-worker-w2-370078-openshift-default-584c45775-h6rwx" Nov 04 05:02:35.028419 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:02:35.028399 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0dfa90b7-fb2a-439c-a9ca-d92c30a7f088-istio-podinfo\") pod \"e2e-worker-w2-370078-openshift-default-584c45775-h6rwx\" (UID: \"0dfa90b7-fb2a-439c-a9ca-d92c30a7f088\") " pod="openshift-ingress/e2e-worker-w2-370078-openshift-default-584c45775-h6rwx" Nov 04 05:02:35.028571 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:02:35.028475 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0dfa90b7-fb2a-439c-a9ca-d92c30a7f088-istio-token\") pod \"e2e-worker-w2-370078-openshift-default-584c45775-h6rwx\" (UID: \"0dfa90b7-fb2a-439c-a9ca-d92c30a7f088\") " pod="openshift-ingress/e2e-worker-w2-370078-openshift-default-584c45775-h6rwx" Nov 04 05:02:35.028571 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:02:35.028498 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dfa90b7-fb2a-439c-a9ca-d92c30a7f088-credential-socket\") pod \"e2e-worker-w2-370078-openshift-default-584c45775-h6rwx\" (UID: \"0dfa90b7-fb2a-439c-a9ca-d92c30a7f088\") " pod="openshift-ingress/e2e-worker-w2-370078-openshift-default-584c45775-h6rwx" Nov 04 05:02:35.028571 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:02:35.028543 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dfa90b7-fb2a-439c-a9ca-d92c30a7f088-workload-socket\") pod \"e2e-worker-w2-370078-openshift-default-584c45775-h6rwx\" (UID: \"0dfa90b7-fb2a-439c-a9ca-d92c30a7f088\") " pod="openshift-ingress/e2e-worker-w2-370078-openshift-default-584c45775-h6rwx" Nov 04 05:02:35.028571 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:02:35.028560 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0dfa90b7-fb2a-439c-a9ca-d92c30a7f088-istio-data\") pod \"e2e-worker-w2-370078-openshift-default-584c45775-h6rwx\" (UID: \"0dfa90b7-fb2a-439c-a9ca-d92c30a7f088\") " pod="openshift-ingress/e2e-worker-w2-370078-openshift-default-584c45775-h6rwx" Nov 04 05:02:35.028719 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:02:35.028578 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0dfa90b7-fb2a-439c-a9ca-d92c30a7f088-istio-envoy\") pod \"e2e-worker-w2-370078-openshift-default-584c45775-h6rwx\" (UID: \"0dfa90b7-fb2a-439c-a9ca-d92c30a7f088\") " pod="openshift-ingress/e2e-worker-w2-370078-openshift-default-584c45775-h6rwx" Nov 04 05:02:35.129297 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:02:35.129215 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0dfa90b7-fb2a-439c-a9ca-d92c30a7f088-workload-certs\") pod \"e2e-worker-w2-370078-openshift-default-584c45775-h6rwx\" (UID: \"0dfa90b7-fb2a-439c-a9ca-d92c30a7f088\") " pod="openshift-ingress/e2e-worker-w2-370078-openshift-default-584c45775-h6rwx" Nov 04 05:02:35.129297 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:02:35.129257 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0dfa90b7-fb2a-439c-a9ca-d92c30a7f088-istio-podinfo\") pod \"e2e-worker-w2-370078-openshift-default-584c45775-h6rwx\" (UID: \"0dfa90b7-fb2a-439c-a9ca-d92c30a7f088\") " pod="openshift-ingress/e2e-worker-w2-370078-openshift-default-584c45775-h6rwx" Nov 04 05:02:35.129297 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:02:35.129297 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0dfa90b7-fb2a-439c-a9ca-d92c30a7f088-istio-token\") pod \"e2e-worker-w2-370078-openshift-default-584c45775-h6rwx\" (UID: \"0dfa90b7-fb2a-439c-a9ca-d92c30a7f088\") " pod="openshift-ingress/e2e-worker-w2-370078-openshift-default-584c45775-h6rwx" Nov 04 05:02:35.129588 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:02:35.129342 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dfa90b7-fb2a-439c-a9ca-d92c30a7f088-credential-socket\") pod \"e2e-worker-w2-370078-openshift-default-584c45775-h6rwx\" (UID: \"0dfa90b7-fb2a-439c-a9ca-d92c30a7f088\") " pod="openshift-ingress/e2e-worker-w2-370078-openshift-default-584c45775-h6rwx" Nov 04 05:02:35.129588 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:02:35.129364 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dfa90b7-fb2a-439c-a9ca-d92c30a7f088-workload-socket\") pod \"e2e-worker-w2-370078-openshift-default-584c45775-h6rwx\" (UID: \"0dfa90b7-fb2a-439c-a9ca-d92c30a7f088\") " pod="openshift-ingress/e2e-worker-w2-370078-openshift-default-584c45775-h6rwx" Nov 04 05:02:35.129588 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:02:35.129381 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0dfa90b7-fb2a-439c-a9ca-d92c30a7f088-istio-data\") pod \"e2e-worker-w2-370078-openshift-default-584c45775-h6rwx\" (UID: \"0dfa90b7-fb2a-439c-a9ca-d92c30a7f088\") " pod="openshift-ingress/e2e-worker-w2-370078-openshift-default-584c45775-h6rwx" Nov 04 05:02:35.129588 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:02:35.129400 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0dfa90b7-fb2a-439c-a9ca-d92c30a7f088-istio-envoy\") pod \"e2e-worker-w2-370078-openshift-default-584c45775-h6rwx\" (UID: \"0dfa90b7-fb2a-439c-a9ca-d92c30a7f088\") " pod="openshift-ingress/e2e-worker-w2-370078-openshift-default-584c45775-h6rwx" Nov 04 05:02:35.129588 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:02:35.129474 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cznd\" (UniqueName: \"kubernetes.io/projected/0dfa90b7-fb2a-439c-a9ca-d92c30a7f088-kube-api-access-8cznd\") pod \"e2e-worker-w2-370078-openshift-default-584c45775-h6rwx\" (UID: \"0dfa90b7-fb2a-439c-a9ca-d92c30a7f088\") " pod="openshift-ingress/e2e-worker-w2-370078-openshift-default-584c45775-h6rwx" Nov 04 05:02:35.129588 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:02:35.129588 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0dfa90b7-fb2a-439c-a9ca-d92c30a7f088-istiod-ca-cert\") pod \"e2e-worker-w2-370078-openshift-default-584c45775-h6rwx\" (UID: \"0dfa90b7-fb2a-439c-a9ca-d92c30a7f088\") " pod="openshift-ingress/e2e-worker-w2-370078-openshift-default-584c45775-h6rwx" Nov 04 05:02:35.129899 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:02:35.129722 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0dfa90b7-fb2a-439c-a9ca-d92c30a7f088-workload-certs\") pod \"e2e-worker-w2-370078-openshift-default-584c45775-h6rwx\" (UID: \"0dfa90b7-fb2a-439c-a9ca-d92c30a7f088\") " pod="openshift-ingress/e2e-worker-w2-370078-openshift-default-584c45775-h6rwx" Nov 04 05:02:35.129899 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:02:35.129833 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dfa90b7-fb2a-439c-a9ca-d92c30a7f088-workload-socket\") pod \"e2e-worker-w2-370078-openshift-default-584c45775-h6rwx\" (UID: \"0dfa90b7-fb2a-439c-a9ca-d92c30a7f088\") " pod="openshift-ingress/e2e-worker-w2-370078-openshift-default-584c45775-h6rwx" Nov 04 05:02:35.129899 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:02:35.129844 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dfa90b7-fb2a-439c-a9ca-d92c30a7f088-credential-socket\") pod \"e2e-worker-w2-370078-openshift-default-584c45775-h6rwx\" (UID: \"0dfa90b7-fb2a-439c-a9ca-d92c30a7f088\") " pod="openshift-ingress/e2e-worker-w2-370078-openshift-default-584c45775-h6rwx" Nov 04 05:02:35.130021 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:02:35.129926 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0dfa90b7-fb2a-439c-a9ca-d92c30a7f088-istio-data\") pod \"e2e-worker-w2-370078-openshift-default-584c45775-h6rwx\" (UID: \"0dfa90b7-fb2a-439c-a9ca-d92c30a7f088\") " pod="openshift-ingress/e2e-worker-w2-370078-openshift-default-584c45775-h6rwx" Nov 04 05:02:35.130233 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:02:35.130211 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0dfa90b7-fb2a-439c-a9ca-d92c30a7f088-istiod-ca-cert\") pod \"e2e-worker-w2-370078-openshift-default-584c45775-h6rwx\" (UID: \"0dfa90b7-fb2a-439c-a9ca-d92c30a7f088\") " pod="openshift-ingress/e2e-worker-w2-370078-openshift-default-584c45775-h6rwx" Nov 04 05:02:35.131889 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:02:35.131869 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0dfa90b7-fb2a-439c-a9ca-d92c30a7f088-istio-envoy\") pod \"e2e-worker-w2-370078-openshift-default-584c45775-h6rwx\" (UID: \"0dfa90b7-fb2a-439c-a9ca-d92c30a7f088\") " pod="openshift-ingress/e2e-worker-w2-370078-openshift-default-584c45775-h6rwx" Nov 04 05:02:35.131938 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:02:35.131927 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0dfa90b7-fb2a-439c-a9ca-d92c30a7f088-istio-podinfo\") pod \"e2e-worker-w2-370078-openshift-default-584c45775-h6rwx\" (UID: \"0dfa90b7-fb2a-439c-a9ca-d92c30a7f088\") " pod="openshift-ingress/e2e-worker-w2-370078-openshift-default-584c45775-h6rwx" Nov 04 05:02:35.187301 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:02:35.187273 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0dfa90b7-fb2a-439c-a9ca-d92c30a7f088-istio-token\") pod \"e2e-worker-w2-370078-openshift-default-584c45775-h6rwx\" (UID: \"0dfa90b7-fb2a-439c-a9ca-d92c30a7f088\") " pod="openshift-ingress/e2e-worker-w2-370078-openshift-default-584c45775-h6rwx" Nov 04 05:02:35.187460 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:02:35.187276 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cznd\" (UniqueName: \"kubernetes.io/projected/0dfa90b7-fb2a-439c-a9ca-d92c30a7f088-kube-api-access-8cznd\") pod \"e2e-worker-w2-370078-openshift-default-584c45775-h6rwx\" (UID: \"0dfa90b7-fb2a-439c-a9ca-d92c30a7f088\") " pod="openshift-ingress/e2e-worker-w2-370078-openshift-default-584c45775-h6rwx" Nov 04 05:02:35.460557 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:02:35.460530 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-370078-openshift-default-584c45775-h6rwx" Nov 04 05:02:35.887435 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:02:35.887402 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-370078-openshift-default-584c45775-h6rwx"] Nov 04 05:02:35.889002 ip-10-0-133-180 kubenswrapper[4509]: W1104 05:02:35.888977 4509 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dfa90b7_fb2a_439c_a9ca_d92c30a7f088.slice/crio-e0949ef3def5b6fa3fa42b6e7eb2632e0bc2125f7b8e8fb0a2e4c9b2339daee4 WatchSource:0}: Error finding container e0949ef3def5b6fa3fa42b6e7eb2632e0bc2125f7b8e8fb0a2e4c9b2339daee4: Status 404 returned error can't find the container with id e0949ef3def5b6fa3fa42b6e7eb2632e0bc2125f7b8e8fb0a2e4c9b2339daee4 Nov 04 05:02:35.891374 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:02:35.891339 4509 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 04 05:02:35.891472 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:02:35.891421 4509 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 04 05:02:35.891540 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:02:35.891468 4509 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 04 05:02:36.593396 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:02:36.593347 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-370078-openshift-default-584c45775-h6rwx" event={"ID":"0dfa90b7-fb2a-439c-a9ca-d92c30a7f088","Type":"ContainerStarted","Data":"26445351906e2b176a05446942950cace621ccd4d3fbfafc591b6cee04bf1d05"} Nov 04 05:02:36.593575 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:02:36.593412 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-370078-openshift-default-584c45775-h6rwx" event={"ID":"0dfa90b7-fb2a-439c-a9ca-d92c30a7f088","Type":"ContainerStarted","Data":"e0949ef3def5b6fa3fa42b6e7eb2632e0bc2125f7b8e8fb0a2e4c9b2339daee4"} Nov 04 05:02:36.811645 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:02:36.811574 4509 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-370078-openshift-default-584c45775-h6rwx" podStartSLOduration=2.811558371 podStartE2EDuration="2.811558371s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 05:02:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 05:02:36.78078301 +0000 UTC m=+920.658506537" watchObservedRunningTime="2026-11-04 05:02:36.811558371 +0000 UTC m=+920.689281891" Nov 04 05:02:37.462816 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:02:37.462430 4509 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-370078-openshift-default-584c45775-h6rwx" Nov 04 05:02:37.462816 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:02:37.462541 4509 patch_prober.go:29] interesting pod/e2e-worker-w2-370078-openshift-default-584c45775-h6rwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Nov 04 05:02:37.462816 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:02:37.462582 4509 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-370078-openshift-default-584c45775-h6rwx" podUID="0dfa90b7-fb2a-439c-a9ca-d92c30a7f088" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Nov 04 05:02:37.466507 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:02:37.466206 4509 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-j8ns4" Nov 04 05:02:37.556050 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:02:37.556012 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3914303c-ffca-4fa0-bf97-d15430976da6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3914303c-ffca-4fa0-bf97-d15430976da6-credential-socket\") pod \"3914303c-ffca-4fa0-bf97-d15430976da6\" (UID: \"3914303c-ffca-4fa0-bf97-d15430976da6\") " Nov 04 05:02:37.556242 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:02:37.556061 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3914303c-ffca-4fa0-bf97-d15430976da6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3914303c-ffca-4fa0-bf97-d15430976da6-istio-envoy\") pod \"3914303c-ffca-4fa0-bf97-d15430976da6\" (UID: \"3914303c-ffca-4fa0-bf97-d15430976da6\") " Nov 04 05:02:37.556242 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:02:37.556099 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3914303c-ffca-4fa0-bf97-d15430976da6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3914303c-ffca-4fa0-bf97-d15430976da6-istiod-ca-cert\") pod \"3914303c-ffca-4fa0-bf97-d15430976da6\" (UID: \"3914303c-ffca-4fa0-bf97-d15430976da6\") " Nov 04 05:02:37.556242 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:02:37.556121 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3914303c-ffca-4fa0-bf97-d15430976da6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3914303c-ffca-4fa0-bf97-d15430976da6-workload-socket\") pod \"3914303c-ffca-4fa0-bf97-d15430976da6\" (UID: \"3914303c-ffca-4fa0-bf97-d15430976da6\") " Nov 04 05:02:37.556242 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:02:37.556183 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3914303c-ffca-4fa0-bf97-d15430976da6-istio-token\" (UniqueName: \"kubernetes.io/projected/3914303c-ffca-4fa0-bf97-d15430976da6-istio-token\") pod \"3914303c-ffca-4fa0-bf97-d15430976da6\" (UID: \"3914303c-ffca-4fa0-bf97-d15430976da6\") " Nov 04 05:02:37.556242 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:02:37.556220 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3914303c-ffca-4fa0-bf97-d15430976da6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3914303c-ffca-4fa0-bf97-d15430976da6-istio-podinfo\") pod \"3914303c-ffca-4fa0-bf97-d15430976da6\" (UID: \"3914303c-ffca-4fa0-bf97-d15430976da6\") " Nov 04 05:02:37.556543 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:02:37.556353 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3914303c-ffca-4fa0-bf97-d15430976da6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3914303c-ffca-4fa0-bf97-d15430976da6-workload-certs\") pod \"3914303c-ffca-4fa0-bf97-d15430976da6\" (UID: \"3914303c-ffca-4fa0-bf97-d15430976da6\") " Nov 04 05:02:37.556543 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:02:37.556377 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3914303c-ffca-4fa0-bf97-d15430976da6-credential-socket" pod "3914303c-ffca-4fa0-bf97-d15430976da6" (UID: "3914303c-ffca-4fa0-bf97-d15430976da6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:02:37.556543 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:02:37.556388 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3914303c-ffca-4fa0-bf97-d15430976da6-kube-api-access-lkgkw\" (UniqueName: \"kubernetes.io/projected/3914303c-ffca-4fa0-bf97-d15430976da6-kube-api-access-lkgkw\") pod \"3914303c-ffca-4fa0-bf97-d15430976da6\" (UID: \"3914303c-ffca-4fa0-bf97-d15430976da6\") " Nov 04 05:02:37.556543 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:02:37.556476 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3914303c-ffca-4fa0-bf97-d15430976da6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3914303c-ffca-4fa0-bf97-d15430976da6-istio-data\") pod \"3914303c-ffca-4fa0-bf97-d15430976da6\" (UID: \"3914303c-ffca-4fa0-bf97-d15430976da6\") " Nov 04 05:02:37.556751 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:02:37.556574 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3914303c-ffca-4fa0-bf97-d15430976da6-workload-socket" pod "3914303c-ffca-4fa0-bf97-d15430976da6" (UID: "3914303c-ffca-4fa0-bf97-d15430976da6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:02:37.556751 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:02:37.556647 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3914303c-ffca-4fa0-bf97-d15430976da6-istiod-ca-cert" pod "3914303c-ffca-4fa0-bf97-d15430976da6" (UID: "3914303c-ffca-4fa0-bf97-d15430976da6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 05:02:37.556871 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:02:37.556837 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3914303c-ffca-4fa0-bf97-d15430976da6-workload-certs" pod "3914303c-ffca-4fa0-bf97-d15430976da6" (UID: "3914303c-ffca-4fa0-bf97-d15430976da6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:02:37.557098 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:02:37.556866 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3914303c-ffca-4fa0-bf97-d15430976da6-istio-data" pod "3914303c-ffca-4fa0-bf97-d15430976da6" (UID: "3914303c-ffca-4fa0-bf97-d15430976da6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:02:37.557098 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:02:37.556970 4509 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3914303c-ffca-4fa0-bf97-d15430976da6-istio-data\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:02:37.557098 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:02:37.556990 4509 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3914303c-ffca-4fa0-bf97-d15430976da6-credential-socket\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:02:37.557098 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:02:37.557007 4509 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3914303c-ffca-4fa0-bf97-d15430976da6-istiod-ca-cert\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:02:37.557098 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:02:37.557020 4509 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3914303c-ffca-4fa0-bf97-d15430976da6-workload-socket\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:02:37.557098 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:02:37.557034 4509 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3914303c-ffca-4fa0-bf97-d15430976da6-workload-certs\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:02:37.558718 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:02:37.558693 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3914303c-ffca-4fa0-bf97-d15430976da6-istio-envoy" pod "3914303c-ffca-4fa0-bf97-d15430976da6" (UID: "3914303c-ffca-4fa0-bf97-d15430976da6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:02:37.558831 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:02:37.558733 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3914303c-ffca-4fa0-bf97-d15430976da6-istio-podinfo" pod "3914303c-ffca-4fa0-bf97-d15430976da6" (UID: "3914303c-ffca-4fa0-bf97-d15430976da6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 05:02:37.558831 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:02:37.558786 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3914303c-ffca-4fa0-bf97-d15430976da6-istio-token" pod "3914303c-ffca-4fa0-bf97-d15430976da6" (UID: "3914303c-ffca-4fa0-bf97-d15430976da6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 05:02:37.558898 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:02:37.558864 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3914303c-ffca-4fa0-bf97-d15430976da6-kube-api-access-lkgkw" pod "3914303c-ffca-4fa0-bf97-d15430976da6" (UID: "3914303c-ffca-4fa0-bf97-d15430976da6"). InnerVolumeSpecName "kube-api-access-lkgkw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 05:02:37.599763 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:02:37.599727 4509 generic.go:356] "Generic (PLEG): container finished" podID="3914303c-ffca-4fa0-bf97-d15430976da6" containerID="e85fa7471bbfc80c1eafbaace18a80c16a30420142fe26f0bd8d48ce9842caae" exitCode=0 Nov 04 05:02:37.599932 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:02:37.599781 4509 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-j8ns4" Nov 04 05:02:37.599932 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:02:37.599807 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-j8ns4" event={"ID":"3914303c-ffca-4fa0-bf97-d15430976da6","Type":"ContainerDied","Data":"e85fa7471bbfc80c1eafbaace18a80c16a30420142fe26f0bd8d48ce9842caae"} Nov 04 05:02:37.599932 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:02:37.599859 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-j8ns4" event={"ID":"3914303c-ffca-4fa0-bf97-d15430976da6","Type":"ContainerDied","Data":"df1c7a88a578f8018c5d2028bd196f2134449580388eb6e9a7198e8baae5cce3"} Nov 04 05:02:37.599932 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:02:37.599882 4509 scope.go:122] "RemoveContainer" containerID="e85fa7471bbfc80c1eafbaace18a80c16a30420142fe26f0bd8d48ce9842caae" Nov 04 05:02:37.611762 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:02:37.611737 4509 scope.go:122] "RemoveContainer" containerID="e85fa7471bbfc80c1eafbaace18a80c16a30420142fe26f0bd8d48ce9842caae" Nov 04 05:02:37.612586 ip-10-0-133-180 kubenswrapper[4509]: E1104 05:02:37.612558 4509 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e85fa7471bbfc80c1eafbaace18a80c16a30420142fe26f0bd8d48ce9842caae\": container with ID starting with e85fa7471bbfc80c1eafbaace18a80c16a30420142fe26f0bd8d48ce9842caae not found: ID does not exist" containerID="e85fa7471bbfc80c1eafbaace18a80c16a30420142fe26f0bd8d48ce9842caae" Nov 04 05:02:37.612688 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:02:37.612597 4509 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e85fa7471bbfc80c1eafbaace18a80c16a30420142fe26f0bd8d48ce9842caae"} err="failed to get container status \"e85fa7471bbfc80c1eafbaace18a80c16a30420142fe26f0bd8d48ce9842caae\": rpc error: code = NotFound desc = could not find container \"e85fa7471bbfc80c1eafbaace18a80c16a30420142fe26f0bd8d48ce9842caae\": container with ID starting with e85fa7471bbfc80c1eafbaace18a80c16a30420142fe26f0bd8d48ce9842caae not found: ID does not exist" Nov 04 05:02:37.657900 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:02:37.657872 4509 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkgkw\" (UniqueName: \"kubernetes.io/projected/3914303c-ffca-4fa0-bf97-d15430976da6-kube-api-access-lkgkw\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:02:37.657900 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:02:37.657897 4509 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3914303c-ffca-4fa0-bf97-d15430976da6-istio-envoy\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:02:37.658136 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:02:37.657910 4509 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3914303c-ffca-4fa0-bf97-d15430976da6-istio-token\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:02:37.658136 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:02:37.657924 4509 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3914303c-ffca-4fa0-bf97-d15430976da6-istio-podinfo\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:02:37.757539 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:02:37.757470 4509 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-j8ns4"] Nov 04 05:02:37.782348 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:02:37.760705 4509 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-j8ns4"] Nov 04 05:02:38.284762 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:02:38.284728 4509 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3914303c-ffca-4fa0-bf97-d15430976da6" path="/var/lib/kubelet/pods/3914303c-ffca-4fa0-bf97-d15430976da6/volumes" Nov 04 05:02:38.482351 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:02:38.479749 4509 patch_prober.go:29] interesting pod/e2e-worker-w2-370078-openshift-default-584c45775-h6rwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Nov 04 05:02:38.482351 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:02:38.479810 4509 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-370078-openshift-default-584c45775-h6rwx" podUID="0dfa90b7-fb2a-439c-a9ca-d92c30a7f088" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Nov 04 05:02:39.482349 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:02:39.474882 4509 patch_prober.go:29] interesting pod/e2e-worker-w2-370078-openshift-default-584c45775-h6rwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Nov 04 05:02:39.482349 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:02:39.475013 4509 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-370078-openshift-default-584c45775-h6rwx" podUID="0dfa90b7-fb2a-439c-a9ca-d92c30a7f088" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Nov 04 05:02:40.484859 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:02:40.484822 4509 patch_prober.go:29] interesting pod/e2e-worker-w2-370078-openshift-default-584c45775-h6rwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Nov 04 05:02:40.485880 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:02:40.485467 4509 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-370078-openshift-default-584c45775-h6rwx" podUID="0dfa90b7-fb2a-439c-a9ca-d92c30a7f088" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Nov 04 05:02:41.483349 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:02:41.482566 4509 patch_prober.go:29] interesting pod/e2e-worker-w2-370078-openshift-default-584c45775-h6rwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Nov 04 05:02:41.483349 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:02:41.482643 4509 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-370078-openshift-default-584c45775-h6rwx" podUID="0dfa90b7-fb2a-439c-a9ca-d92c30a7f088" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Nov 04 05:02:42.651072 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:02:42.651022 4509 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-370078-openshift-default-584c45775-h6rwx" Nov 04 05:02:42.651594 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:02:42.651451 4509 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-370078-openshift-default-584c45775-h6rwx" Nov 04 05:02:42.652208 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:02:42.652188 4509 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-370078-openshift-default-584c45775-h6rwx" Nov 04 05:02:44.046430 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:02:44.046394 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-42dnm"] Nov 04 05:02:44.482399 ip-10-0-133-180 kubenswrapper[4509]: E1104 05:02:44.482362 4509 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmdhl" containerName="extractor" Nov 04 05:03:09.327978 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:09.327932 4509 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-4cce33/distinct-w5-4cce33-kserve-68bf9d9bb7-x62f2"] Nov 04 05:03:09.328807 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:09.328784 4509 cpu_manager.go:414] "containerMap: removing container" podUID="3914303c-ffca-4fa0-bf97-d15430976da6" containerName="istio-proxy" Nov 04 05:03:09.328906 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:09.328808 4509 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3914303c-ffca-4fa0-bf97-d15430976da6" containerName="istio-proxy" Nov 04 05:03:09.328972 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:09.328957 4509 memory_manager.go:365] "RemoveStaleState removing state" podUID="3914303c-ffca-4fa0-bf97-d15430976da6" containerName="istio-proxy" Nov 04 05:03:09.332388 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:09.332367 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-4cce33/distinct-w5-4cce33-kserve-68bf9d9bb7-x62f2" Nov 04 05:03:09.435018 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:09.434988 4509 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-4cce33\"/\"distinct-w5-4cce33-kserve-self-signed-certs\"" Nov 04 05:03:09.437457 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:09.437437 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-4cce33/distinct-w5-4cce33-kserve-68bf9d9bb7-x62f2"] Nov 04 05:03:09.459364 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:09.459314 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1afb6cd-a9c5-43b9-ae3b-9929840fe019-tmp-dir\") pod \"distinct-w5-4cce33-kserve-68bf9d9bb7-x62f2\" (UID: \"c1afb6cd-a9c5-43b9-ae3b-9929840fe019\") " pod="e2e-models-e2e-worker-w5-4cce33/distinct-w5-4cce33-kserve-68bf9d9bb7-x62f2" Nov 04 05:03:09.459512 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:09.459434 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1afb6cd-a9c5-43b9-ae3b-9929840fe019-model-cache\") pod \"distinct-w5-4cce33-kserve-68bf9d9bb7-x62f2\" (UID: \"c1afb6cd-a9c5-43b9-ae3b-9929840fe019\") " pod="e2e-models-e2e-worker-w5-4cce33/distinct-w5-4cce33-kserve-68bf9d9bb7-x62f2" Nov 04 05:03:09.459512 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:09.459478 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnrsv\" (UniqueName: \"kubernetes.io/projected/c1afb6cd-a9c5-43b9-ae3b-9929840fe019-kube-api-access-gnrsv\") pod \"distinct-w5-4cce33-kserve-68bf9d9bb7-x62f2\" (UID: \"c1afb6cd-a9c5-43b9-ae3b-9929840fe019\") " pod="e2e-models-e2e-worker-w5-4cce33/distinct-w5-4cce33-kserve-68bf9d9bb7-x62f2" Nov 04 05:03:09.459512 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:09.459506 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1afb6cd-a9c5-43b9-ae3b-9929840fe019-home\") pod \"distinct-w5-4cce33-kserve-68bf9d9bb7-x62f2\" (UID: \"c1afb6cd-a9c5-43b9-ae3b-9929840fe019\") " pod="e2e-models-e2e-worker-w5-4cce33/distinct-w5-4cce33-kserve-68bf9d9bb7-x62f2" Nov 04 05:03:09.459621 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:09.459589 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1afb6cd-a9c5-43b9-ae3b-9929840fe019-dshm\") pod \"distinct-w5-4cce33-kserve-68bf9d9bb7-x62f2\" (UID: \"c1afb6cd-a9c5-43b9-ae3b-9929840fe019\") " pod="e2e-models-e2e-worker-w5-4cce33/distinct-w5-4cce33-kserve-68bf9d9bb7-x62f2" Nov 04 05:03:09.459733 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:09.459717 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1afb6cd-a9c5-43b9-ae3b-9929840fe019-tls-certs\") pod \"distinct-w5-4cce33-kserve-68bf9d9bb7-x62f2\" (UID: \"c1afb6cd-a9c5-43b9-ae3b-9929840fe019\") " pod="e2e-models-e2e-worker-w5-4cce33/distinct-w5-4cce33-kserve-68bf9d9bb7-x62f2" Nov 04 05:03:09.560771 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:09.560738 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1afb6cd-a9c5-43b9-ae3b-9929840fe019-home\") pod \"distinct-w5-4cce33-kserve-68bf9d9bb7-x62f2\" (UID: \"c1afb6cd-a9c5-43b9-ae3b-9929840fe019\") " pod="e2e-models-e2e-worker-w5-4cce33/distinct-w5-4cce33-kserve-68bf9d9bb7-x62f2" Nov 04 05:03:09.560771 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:09.560779 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1afb6cd-a9c5-43b9-ae3b-9929840fe019-dshm\") pod \"distinct-w5-4cce33-kserve-68bf9d9bb7-x62f2\" (UID: \"c1afb6cd-a9c5-43b9-ae3b-9929840fe019\") " pod="e2e-models-e2e-worker-w5-4cce33/distinct-w5-4cce33-kserve-68bf9d9bb7-x62f2" Nov 04 05:03:09.561026 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:09.560856 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1afb6cd-a9c5-43b9-ae3b-9929840fe019-tls-certs\") pod \"distinct-w5-4cce33-kserve-68bf9d9bb7-x62f2\" (UID: \"c1afb6cd-a9c5-43b9-ae3b-9929840fe019\") " pod="e2e-models-e2e-worker-w5-4cce33/distinct-w5-4cce33-kserve-68bf9d9bb7-x62f2" Nov 04 05:03:09.561026 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:09.560918 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1afb6cd-a9c5-43b9-ae3b-9929840fe019-tmp-dir\") pod \"distinct-w5-4cce33-kserve-68bf9d9bb7-x62f2\" (UID: \"c1afb6cd-a9c5-43b9-ae3b-9929840fe019\") " pod="e2e-models-e2e-worker-w5-4cce33/distinct-w5-4cce33-kserve-68bf9d9bb7-x62f2" Nov 04 05:03:09.561026 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:09.560948 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1afb6cd-a9c5-43b9-ae3b-9929840fe019-model-cache\") pod \"distinct-w5-4cce33-kserve-68bf9d9bb7-x62f2\" (UID: \"c1afb6cd-a9c5-43b9-ae3b-9929840fe019\") " pod="e2e-models-e2e-worker-w5-4cce33/distinct-w5-4cce33-kserve-68bf9d9bb7-x62f2" Nov 04 05:03:09.561026 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:09.560981 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnrsv\" (UniqueName: \"kubernetes.io/projected/c1afb6cd-a9c5-43b9-ae3b-9929840fe019-kube-api-access-gnrsv\") pod \"distinct-w5-4cce33-kserve-68bf9d9bb7-x62f2\" (UID: \"c1afb6cd-a9c5-43b9-ae3b-9929840fe019\") " pod="e2e-models-e2e-worker-w5-4cce33/distinct-w5-4cce33-kserve-68bf9d9bb7-x62f2" Nov 04 05:03:09.561243 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:09.561228 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1afb6cd-a9c5-43b9-ae3b-9929840fe019-home\") pod \"distinct-w5-4cce33-kserve-68bf9d9bb7-x62f2\" (UID: \"c1afb6cd-a9c5-43b9-ae3b-9929840fe019\") " pod="e2e-models-e2e-worker-w5-4cce33/distinct-w5-4cce33-kserve-68bf9d9bb7-x62f2" Nov 04 05:03:09.561303 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:09.561276 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1afb6cd-a9c5-43b9-ae3b-9929840fe019-tmp-dir\") pod \"distinct-w5-4cce33-kserve-68bf9d9bb7-x62f2\" (UID: \"c1afb6cd-a9c5-43b9-ae3b-9929840fe019\") " pod="e2e-models-e2e-worker-w5-4cce33/distinct-w5-4cce33-kserve-68bf9d9bb7-x62f2" Nov 04 05:03:09.561398 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:09.561340 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1afb6cd-a9c5-43b9-ae3b-9929840fe019-model-cache\") pod \"distinct-w5-4cce33-kserve-68bf9d9bb7-x62f2\" (UID: \"c1afb6cd-a9c5-43b9-ae3b-9929840fe019\") " pod="e2e-models-e2e-worker-w5-4cce33/distinct-w5-4cce33-kserve-68bf9d9bb7-x62f2" Nov 04 05:03:09.563231 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:09.563211 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1afb6cd-a9c5-43b9-ae3b-9929840fe019-dshm\") pod \"distinct-w5-4cce33-kserve-68bf9d9bb7-x62f2\" (UID: \"c1afb6cd-a9c5-43b9-ae3b-9929840fe019\") " pod="e2e-models-e2e-worker-w5-4cce33/distinct-w5-4cce33-kserve-68bf9d9bb7-x62f2" Nov 04 05:03:09.563559 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:09.563543 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1afb6cd-a9c5-43b9-ae3b-9929840fe019-tls-certs\") pod \"distinct-w5-4cce33-kserve-68bf9d9bb7-x62f2\" (UID: \"c1afb6cd-a9c5-43b9-ae3b-9929840fe019\") " pod="e2e-models-e2e-worker-w5-4cce33/distinct-w5-4cce33-kserve-68bf9d9bb7-x62f2" Nov 04 05:03:09.618209 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:09.618143 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnrsv\" (UniqueName: \"kubernetes.io/projected/c1afb6cd-a9c5-43b9-ae3b-9929840fe019-kube-api-access-gnrsv\") pod \"distinct-w5-4cce33-kserve-68bf9d9bb7-x62f2\" (UID: \"c1afb6cd-a9c5-43b9-ae3b-9929840fe019\") " pod="e2e-models-e2e-worker-w5-4cce33/distinct-w5-4cce33-kserve-68bf9d9bb7-x62f2" Nov 04 05:03:09.644469 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:09.644437 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-4cce33/distinct-w5-4cce33-kserve-68bf9d9bb7-x62f2" Nov 04 05:03:10.071091 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:10.071064 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-4cce33/distinct-w5-4cce33-kserve-68bf9d9bb7-x62f2"] Nov 04 05:03:10.072664 ip-10-0-133-180 kubenswrapper[4509]: W1104 05:03:10.072641 4509 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc1afb6cd_a9c5_43b9_ae3b_9929840fe019.slice/crio-b9892b4933cd991ced00faac6931f1eef09b4bd95cf49c87755325a54f3bf218 WatchSource:0}: Error finding container b9892b4933cd991ced00faac6931f1eef09b4bd95cf49c87755325a54f3bf218: Status 404 returned error can't find the container with id b9892b4933cd991ced00faac6931f1eef09b4bd95cf49c87755325a54f3bf218 Nov 04 05:03:10.857258 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:10.857219 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-4cce33/distinct-w5-4cce33-kserve-68bf9d9bb7-x62f2" event={"ID":"c1afb6cd-a9c5-43b9-ae3b-9929840fe019","Type":"ContainerStarted","Data":"23d7e72b6b5c50984a8aee1d10405a99f4d045dfb5933ebc93ea44560de5a49f"} Nov 04 05:03:10.857258 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:10.857260 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-4cce33/distinct-w5-4cce33-kserve-68bf9d9bb7-x62f2" event={"ID":"c1afb6cd-a9c5-43b9-ae3b-9929840fe019","Type":"ContainerStarted","Data":"b9892b4933cd991ced00faac6931f1eef09b4bd95cf49c87755325a54f3bf218"} Nov 04 05:03:10.974486 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:10.974423 4509 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-4cce33/distinct-w5-4cce33-kserve-68bf9d9bb7-x62f2" podStartSLOduration=1.974409321 podStartE2EDuration="1.974409321s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 05:03:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 05:03:10.972690862 +0000 UTC m=+954.850414386" watchObservedRunningTime="2026-11-04 05:03:10.974409321 +0000 UTC m=+954.852132843" Nov 04 05:03:11.644871 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:11.644829 4509 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-4cce33/distinct-w5-4cce33-kserve-68bf9d9bb7-x62f2" Nov 04 05:03:14.454636 ip-10-0-133-180 kubenswrapper[4509]: E1104 05:03:14.454595 4509 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmdhl" containerName="extractor" Nov 04 05:03:18.102432 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:18.102399 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-42dnm"] Nov 04 05:03:19.645389 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:19.645351 4509 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-4cce33/distinct-w5-4cce33-kserve-68bf9d9bb7-x62f2" Nov 04 05:03:19.657932 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:19.657909 4509 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-4cce33/distinct-w5-4cce33-kserve-68bf9d9bb7-x62f2" Nov 04 05:03:19.918205 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:19.918126 4509 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-4cce33/distinct-w5-4cce33-kserve-68bf9d9bb7-x62f2" Nov 04 05:03:25.447546 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:25.447510 4509 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-cedfca/distinct-w6-cedfca-kserve-54897db797-fsx45"] Nov 04 05:03:25.452744 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:25.452724 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-cedfca/distinct-w6-cedfca-kserve-54897db797-fsx45" Nov 04 05:03:25.508085 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:25.508053 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-cedfca/distinct-w6-cedfca-kserve-54897db797-fsx45"] Nov 04 05:03:25.537532 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:25.537503 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5cf9f31-04dc-4d03-a8e2-bf38bcaf3830-tmp-dir\") pod \"distinct-w6-cedfca-kserve-54897db797-fsx45\" (UID: \"e5cf9f31-04dc-4d03-a8e2-bf38bcaf3830\") " pod="e2e-models-e2e-worker-w6-cedfca/distinct-w6-cedfca-kserve-54897db797-fsx45" Nov 04 05:03:25.537689 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:25.537557 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5cf9f31-04dc-4d03-a8e2-bf38bcaf3830-dshm\") pod \"distinct-w6-cedfca-kserve-54897db797-fsx45\" (UID: \"e5cf9f31-04dc-4d03-a8e2-bf38bcaf3830\") " pod="e2e-models-e2e-worker-w6-cedfca/distinct-w6-cedfca-kserve-54897db797-fsx45" Nov 04 05:03:25.537765 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:25.537736 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5cf9f31-04dc-4d03-a8e2-bf38bcaf3830-tls-certs\") pod \"distinct-w6-cedfca-kserve-54897db797-fsx45\" (UID: \"e5cf9f31-04dc-4d03-a8e2-bf38bcaf3830\") " pod="e2e-models-e2e-worker-w6-cedfca/distinct-w6-cedfca-kserve-54897db797-fsx45" Nov 04 05:03:25.537833 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:25.537816 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5cf9f31-04dc-4d03-a8e2-bf38bcaf3830-home\") pod \"distinct-w6-cedfca-kserve-54897db797-fsx45\" (UID: \"e5cf9f31-04dc-4d03-a8e2-bf38bcaf3830\") " pod="e2e-models-e2e-worker-w6-cedfca/distinct-w6-cedfca-kserve-54897db797-fsx45" Nov 04 05:03:25.537878 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:25.537850 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phcsq\" (UniqueName: \"kubernetes.io/projected/e5cf9f31-04dc-4d03-a8e2-bf38bcaf3830-kube-api-access-phcsq\") pod \"distinct-w6-cedfca-kserve-54897db797-fsx45\" (UID: \"e5cf9f31-04dc-4d03-a8e2-bf38bcaf3830\") " pod="e2e-models-e2e-worker-w6-cedfca/distinct-w6-cedfca-kserve-54897db797-fsx45" Nov 04 05:03:25.537924 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:25.537907 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5cf9f31-04dc-4d03-a8e2-bf38bcaf3830-model-cache\") pod \"distinct-w6-cedfca-kserve-54897db797-fsx45\" (UID: \"e5cf9f31-04dc-4d03-a8e2-bf38bcaf3830\") " pod="e2e-models-e2e-worker-w6-cedfca/distinct-w6-cedfca-kserve-54897db797-fsx45" Nov 04 05:03:25.554298 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:25.554273 4509 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-cedfca\"/\"distinct-w6-cedfca-kserve-self-signed-certs\"" Nov 04 05:03:25.639706 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:25.639675 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5cf9f31-04dc-4d03-a8e2-bf38bcaf3830-tls-certs\") pod \"distinct-w6-cedfca-kserve-54897db797-fsx45\" (UID: \"e5cf9f31-04dc-4d03-a8e2-bf38bcaf3830\") " pod="e2e-models-e2e-worker-w6-cedfca/distinct-w6-cedfca-kserve-54897db797-fsx45" Nov 04 05:03:25.639886 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:25.639733 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5cf9f31-04dc-4d03-a8e2-bf38bcaf3830-home\") pod \"distinct-w6-cedfca-kserve-54897db797-fsx45\" (UID: \"e5cf9f31-04dc-4d03-a8e2-bf38bcaf3830\") " pod="e2e-models-e2e-worker-w6-cedfca/distinct-w6-cedfca-kserve-54897db797-fsx45" Nov 04 05:03:25.639886 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:25.639759 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phcsq\" (UniqueName: \"kubernetes.io/projected/e5cf9f31-04dc-4d03-a8e2-bf38bcaf3830-kube-api-access-phcsq\") pod \"distinct-w6-cedfca-kserve-54897db797-fsx45\" (UID: \"e5cf9f31-04dc-4d03-a8e2-bf38bcaf3830\") " pod="e2e-models-e2e-worker-w6-cedfca/distinct-w6-cedfca-kserve-54897db797-fsx45" Nov 04 05:03:25.639886 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:25.639812 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5cf9f31-04dc-4d03-a8e2-bf38bcaf3830-model-cache\") pod \"distinct-w6-cedfca-kserve-54897db797-fsx45\" (UID: \"e5cf9f31-04dc-4d03-a8e2-bf38bcaf3830\") " pod="e2e-models-e2e-worker-w6-cedfca/distinct-w6-cedfca-kserve-54897db797-fsx45" Nov 04 05:03:25.640043 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:25.639926 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5cf9f31-04dc-4d03-a8e2-bf38bcaf3830-tmp-dir\") pod \"distinct-w6-cedfca-kserve-54897db797-fsx45\" (UID: \"e5cf9f31-04dc-4d03-a8e2-bf38bcaf3830\") " pod="e2e-models-e2e-worker-w6-cedfca/distinct-w6-cedfca-kserve-54897db797-fsx45" Nov 04 05:03:25.640043 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:25.639968 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5cf9f31-04dc-4d03-a8e2-bf38bcaf3830-dshm\") pod \"distinct-w6-cedfca-kserve-54897db797-fsx45\" (UID: \"e5cf9f31-04dc-4d03-a8e2-bf38bcaf3830\") " pod="e2e-models-e2e-worker-w6-cedfca/distinct-w6-cedfca-kserve-54897db797-fsx45" Nov 04 05:03:25.640273 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:25.640243 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5cf9f31-04dc-4d03-a8e2-bf38bcaf3830-home\") pod \"distinct-w6-cedfca-kserve-54897db797-fsx45\" (UID: \"e5cf9f31-04dc-4d03-a8e2-bf38bcaf3830\") " pod="e2e-models-e2e-worker-w6-cedfca/distinct-w6-cedfca-kserve-54897db797-fsx45" Nov 04 05:03:25.640438 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:25.640349 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5cf9f31-04dc-4d03-a8e2-bf38bcaf3830-tmp-dir\") pod \"distinct-w6-cedfca-kserve-54897db797-fsx45\" (UID: \"e5cf9f31-04dc-4d03-a8e2-bf38bcaf3830\") " pod="e2e-models-e2e-worker-w6-cedfca/distinct-w6-cedfca-kserve-54897db797-fsx45" Nov 04 05:03:25.640438 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:25.640317 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5cf9f31-04dc-4d03-a8e2-bf38bcaf3830-model-cache\") pod \"distinct-w6-cedfca-kserve-54897db797-fsx45\" (UID: \"e5cf9f31-04dc-4d03-a8e2-bf38bcaf3830\") " pod="e2e-models-e2e-worker-w6-cedfca/distinct-w6-cedfca-kserve-54897db797-fsx45" Nov 04 05:03:25.642851 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:25.642818 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5cf9f31-04dc-4d03-a8e2-bf38bcaf3830-dshm\") pod \"distinct-w6-cedfca-kserve-54897db797-fsx45\" (UID: \"e5cf9f31-04dc-4d03-a8e2-bf38bcaf3830\") " pod="e2e-models-e2e-worker-w6-cedfca/distinct-w6-cedfca-kserve-54897db797-fsx45" Nov 04 05:03:25.643071 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:25.643056 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5cf9f31-04dc-4d03-a8e2-bf38bcaf3830-tls-certs\") pod \"distinct-w6-cedfca-kserve-54897db797-fsx45\" (UID: \"e5cf9f31-04dc-4d03-a8e2-bf38bcaf3830\") " pod="e2e-models-e2e-worker-w6-cedfca/distinct-w6-cedfca-kserve-54897db797-fsx45" Nov 04 05:03:25.696680 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:25.696655 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phcsq\" (UniqueName: \"kubernetes.io/projected/e5cf9f31-04dc-4d03-a8e2-bf38bcaf3830-kube-api-access-phcsq\") pod \"distinct-w6-cedfca-kserve-54897db797-fsx45\" (UID: \"e5cf9f31-04dc-4d03-a8e2-bf38bcaf3830\") " pod="e2e-models-e2e-worker-w6-cedfca/distinct-w6-cedfca-kserve-54897db797-fsx45" Nov 04 05:03:25.764781 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:25.764752 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-cedfca/distinct-w6-cedfca-kserve-54897db797-fsx45" Nov 04 05:03:26.180306 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:26.180274 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-cedfca/distinct-w6-cedfca-kserve-54897db797-fsx45"] Nov 04 05:03:26.181588 ip-10-0-133-180 kubenswrapper[4509]: W1104 05:03:26.181560 4509 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode5cf9f31_04dc_4d03_a8e2_bf38bcaf3830.slice/crio-38c511b43d4f6efc2d20b1ffa4032ab1dd522f221958f0822dd37f2e78c2809d WatchSource:0}: Error finding container 38c511b43d4f6efc2d20b1ffa4032ab1dd522f221958f0822dd37f2e78c2809d: Status 404 returned error can't find the container with id 38c511b43d4f6efc2d20b1ffa4032ab1dd522f221958f0822dd37f2e78c2809d Nov 04 05:03:26.948357 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:26.948301 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-cedfca/distinct-w6-cedfca-kserve-54897db797-fsx45" event={"ID":"e5cf9f31-04dc-4d03-a8e2-bf38bcaf3830","Type":"ContainerStarted","Data":"73f05a18ee37cace663307b405a11529613e33d011290d4755ef03d1d3cd6216"} Nov 04 05:03:26.948776 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:26.948366 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-cedfca/distinct-w6-cedfca-kserve-54897db797-fsx45" event={"ID":"e5cf9f31-04dc-4d03-a8e2-bf38bcaf3830","Type":"ContainerStarted","Data":"38c511b43d4f6efc2d20b1ffa4032ab1dd522f221958f0822dd37f2e78c2809d"} Nov 04 05:03:27.067577 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:27.067525 4509 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-cedfca/distinct-w6-cedfca-kserve-54897db797-fsx45" podStartSLOduration=2.067506814 podStartE2EDuration="2.067506814s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 05:03:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 05:03:27.06744053 +0000 UTC m=+970.945164053" watchObservedRunningTime="2026-11-04 05:03:27.067506814 +0000 UTC m=+970.945230336" Nov 04 05:03:27.765387 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:27.765356 4509 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-cedfca/distinct-w6-cedfca-kserve-54897db797-fsx45" Nov 04 05:03:30.233427 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:30.233398 4509 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-0aae2f-openshift-default-5f99b6799-x69j7"] Nov 04 05:03:30.237538 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:30.237519 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0aae2f-openshift-default-5f99b6799-x69j7" Nov 04 05:03:30.289623 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:30.289596 4509 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-0aae2f-openshift-default-dockercfg-9n62j\"" Nov 04 05:03:30.298149 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:30.298119 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b878b44-bc26-4e68-94c0-7297413ecd02-istio-podinfo\") pod \"e2e-worker-w0-0aae2f-openshift-default-5f99b6799-x69j7\" (UID: \"0b878b44-bc26-4e68-94c0-7297413ecd02\") " pod="openshift-ingress/e2e-worker-w0-0aae2f-openshift-default-5f99b6799-x69j7" Nov 04 05:03:30.298279 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:30.298164 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b878b44-bc26-4e68-94c0-7297413ecd02-istio-token\") pod \"e2e-worker-w0-0aae2f-openshift-default-5f99b6799-x69j7\" (UID: \"0b878b44-bc26-4e68-94c0-7297413ecd02\") " pod="openshift-ingress/e2e-worker-w0-0aae2f-openshift-default-5f99b6799-x69j7" Nov 04 05:03:30.298279 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:30.298183 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b878b44-bc26-4e68-94c0-7297413ecd02-istiod-ca-cert\") pod \"e2e-worker-w0-0aae2f-openshift-default-5f99b6799-x69j7\" (UID: \"0b878b44-bc26-4e68-94c0-7297413ecd02\") " pod="openshift-ingress/e2e-worker-w0-0aae2f-openshift-default-5f99b6799-x69j7" Nov 04 05:03:30.298378 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:30.298286 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b878b44-bc26-4e68-94c0-7297413ecd02-istio-envoy\") pod \"e2e-worker-w0-0aae2f-openshift-default-5f99b6799-x69j7\" (UID: \"0b878b44-bc26-4e68-94c0-7297413ecd02\") " pod="openshift-ingress/e2e-worker-w0-0aae2f-openshift-default-5f99b6799-x69j7" Nov 04 05:03:30.298432 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:30.298413 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b878b44-bc26-4e68-94c0-7297413ecd02-istio-data\") pod \"e2e-worker-w0-0aae2f-openshift-default-5f99b6799-x69j7\" (UID: \"0b878b44-bc26-4e68-94c0-7297413ecd02\") " pod="openshift-ingress/e2e-worker-w0-0aae2f-openshift-default-5f99b6799-x69j7" Nov 04 05:03:30.298563 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:30.298548 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b878b44-bc26-4e68-94c0-7297413ecd02-credential-socket\") pod \"e2e-worker-w0-0aae2f-openshift-default-5f99b6799-x69j7\" (UID: \"0b878b44-bc26-4e68-94c0-7297413ecd02\") " pod="openshift-ingress/e2e-worker-w0-0aae2f-openshift-default-5f99b6799-x69j7" Nov 04 05:03:30.298655 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:30.298638 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b878b44-bc26-4e68-94c0-7297413ecd02-workload-socket\") pod \"e2e-worker-w0-0aae2f-openshift-default-5f99b6799-x69j7\" (UID: \"0b878b44-bc26-4e68-94c0-7297413ecd02\") " pod="openshift-ingress/e2e-worker-w0-0aae2f-openshift-default-5f99b6799-x69j7" Nov 04 05:03:30.298708 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:30.298668 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b878b44-bc26-4e68-94c0-7297413ecd02-workload-certs\") pod \"e2e-worker-w0-0aae2f-openshift-default-5f99b6799-x69j7\" (UID: \"0b878b44-bc26-4e68-94c0-7297413ecd02\") " pod="openshift-ingress/e2e-worker-w0-0aae2f-openshift-default-5f99b6799-x69j7" Nov 04 05:03:30.298770 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:30.298756 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlfcz\" (UniqueName: \"kubernetes.io/projected/0b878b44-bc26-4e68-94c0-7297413ecd02-kube-api-access-mlfcz\") pod \"e2e-worker-w0-0aae2f-openshift-default-5f99b6799-x69j7\" (UID: \"0b878b44-bc26-4e68-94c0-7297413ecd02\") " pod="openshift-ingress/e2e-worker-w0-0aae2f-openshift-default-5f99b6799-x69j7" Nov 04 05:03:30.343880 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:30.343840 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-0aae2f-openshift-default-5f99b6799-x69j7"] Nov 04 05:03:30.400034 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:30.399997 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b878b44-bc26-4e68-94c0-7297413ecd02-credential-socket\") pod \"e2e-worker-w0-0aae2f-openshift-default-5f99b6799-x69j7\" (UID: \"0b878b44-bc26-4e68-94c0-7297413ecd02\") " pod="openshift-ingress/e2e-worker-w0-0aae2f-openshift-default-5f99b6799-x69j7" Nov 04 05:03:30.400214 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:30.400058 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b878b44-bc26-4e68-94c0-7297413ecd02-workload-socket\") pod \"e2e-worker-w0-0aae2f-openshift-default-5f99b6799-x69j7\" (UID: \"0b878b44-bc26-4e68-94c0-7297413ecd02\") " pod="openshift-ingress/e2e-worker-w0-0aae2f-openshift-default-5f99b6799-x69j7" Nov 04 05:03:30.400214 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:30.400082 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b878b44-bc26-4e68-94c0-7297413ecd02-workload-certs\") pod \"e2e-worker-w0-0aae2f-openshift-default-5f99b6799-x69j7\" (UID: \"0b878b44-bc26-4e68-94c0-7297413ecd02\") " pod="openshift-ingress/e2e-worker-w0-0aae2f-openshift-default-5f99b6799-x69j7" Nov 04 05:03:30.400214 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:30.400116 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlfcz\" (UniqueName: \"kubernetes.io/projected/0b878b44-bc26-4e68-94c0-7297413ecd02-kube-api-access-mlfcz\") pod \"e2e-worker-w0-0aae2f-openshift-default-5f99b6799-x69j7\" (UID: \"0b878b44-bc26-4e68-94c0-7297413ecd02\") " pod="openshift-ingress/e2e-worker-w0-0aae2f-openshift-default-5f99b6799-x69j7" Nov 04 05:03:30.400214 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:30.400169 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b878b44-bc26-4e68-94c0-7297413ecd02-istio-podinfo\") pod \"e2e-worker-w0-0aae2f-openshift-default-5f99b6799-x69j7\" (UID: \"0b878b44-bc26-4e68-94c0-7297413ecd02\") " pod="openshift-ingress/e2e-worker-w0-0aae2f-openshift-default-5f99b6799-x69j7" Nov 04 05:03:30.400420 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:30.400214 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b878b44-bc26-4e68-94c0-7297413ecd02-istio-token\") pod \"e2e-worker-w0-0aae2f-openshift-default-5f99b6799-x69j7\" (UID: \"0b878b44-bc26-4e68-94c0-7297413ecd02\") " pod="openshift-ingress/e2e-worker-w0-0aae2f-openshift-default-5f99b6799-x69j7" Nov 04 05:03:30.400420 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:30.400236 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b878b44-bc26-4e68-94c0-7297413ecd02-istiod-ca-cert\") pod \"e2e-worker-w0-0aae2f-openshift-default-5f99b6799-x69j7\" (UID: \"0b878b44-bc26-4e68-94c0-7297413ecd02\") " pod="openshift-ingress/e2e-worker-w0-0aae2f-openshift-default-5f99b6799-x69j7" Nov 04 05:03:30.400420 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:30.400280 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b878b44-bc26-4e68-94c0-7297413ecd02-istio-envoy\") pod \"e2e-worker-w0-0aae2f-openshift-default-5f99b6799-x69j7\" (UID: \"0b878b44-bc26-4e68-94c0-7297413ecd02\") " pod="openshift-ingress/e2e-worker-w0-0aae2f-openshift-default-5f99b6799-x69j7" Nov 04 05:03:30.400420 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:30.400352 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b878b44-bc26-4e68-94c0-7297413ecd02-istio-data\") pod \"e2e-worker-w0-0aae2f-openshift-default-5f99b6799-x69j7\" (UID: \"0b878b44-bc26-4e68-94c0-7297413ecd02\") " pod="openshift-ingress/e2e-worker-w0-0aae2f-openshift-default-5f99b6799-x69j7" Nov 04 05:03:30.400572 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:30.400468 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b878b44-bc26-4e68-94c0-7297413ecd02-credential-socket\") pod \"e2e-worker-w0-0aae2f-openshift-default-5f99b6799-x69j7\" (UID: \"0b878b44-bc26-4e68-94c0-7297413ecd02\") " pod="openshift-ingress/e2e-worker-w0-0aae2f-openshift-default-5f99b6799-x69j7" Nov 04 05:03:30.400572 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:30.400494 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b878b44-bc26-4e68-94c0-7297413ecd02-workload-socket\") pod \"e2e-worker-w0-0aae2f-openshift-default-5f99b6799-x69j7\" (UID: \"0b878b44-bc26-4e68-94c0-7297413ecd02\") " pod="openshift-ingress/e2e-worker-w0-0aae2f-openshift-default-5f99b6799-x69j7" Nov 04 05:03:30.400572 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:30.400546 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b878b44-bc26-4e68-94c0-7297413ecd02-workload-certs\") pod \"e2e-worker-w0-0aae2f-openshift-default-5f99b6799-x69j7\" (UID: \"0b878b44-bc26-4e68-94c0-7297413ecd02\") " pod="openshift-ingress/e2e-worker-w0-0aae2f-openshift-default-5f99b6799-x69j7" Nov 04 05:03:30.400719 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:30.400685 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b878b44-bc26-4e68-94c0-7297413ecd02-istio-data\") pod \"e2e-worker-w0-0aae2f-openshift-default-5f99b6799-x69j7\" (UID: \"0b878b44-bc26-4e68-94c0-7297413ecd02\") " pod="openshift-ingress/e2e-worker-w0-0aae2f-openshift-default-5f99b6799-x69j7" Nov 04 05:03:30.401114 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:30.401087 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b878b44-bc26-4e68-94c0-7297413ecd02-istiod-ca-cert\") pod \"e2e-worker-w0-0aae2f-openshift-default-5f99b6799-x69j7\" (UID: \"0b878b44-bc26-4e68-94c0-7297413ecd02\") " pod="openshift-ingress/e2e-worker-w0-0aae2f-openshift-default-5f99b6799-x69j7" Nov 04 05:03:30.403257 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:30.402752 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b878b44-bc26-4e68-94c0-7297413ecd02-istio-envoy\") pod \"e2e-worker-w0-0aae2f-openshift-default-5f99b6799-x69j7\" (UID: \"0b878b44-bc26-4e68-94c0-7297413ecd02\") " pod="openshift-ingress/e2e-worker-w0-0aae2f-openshift-default-5f99b6799-x69j7" Nov 04 05:03:30.403369 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:30.403269 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b878b44-bc26-4e68-94c0-7297413ecd02-istio-podinfo\") pod \"e2e-worker-w0-0aae2f-openshift-default-5f99b6799-x69j7\" (UID: \"0b878b44-bc26-4e68-94c0-7297413ecd02\") " pod="openshift-ingress/e2e-worker-w0-0aae2f-openshift-default-5f99b6799-x69j7" Nov 04 05:03:30.457121 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:30.457090 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b878b44-bc26-4e68-94c0-7297413ecd02-istio-token\") pod \"e2e-worker-w0-0aae2f-openshift-default-5f99b6799-x69j7\" (UID: \"0b878b44-bc26-4e68-94c0-7297413ecd02\") " pod="openshift-ingress/e2e-worker-w0-0aae2f-openshift-default-5f99b6799-x69j7" Nov 04 05:03:30.457448 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:30.457388 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlfcz\" (UniqueName: \"kubernetes.io/projected/0b878b44-bc26-4e68-94c0-7297413ecd02-kube-api-access-mlfcz\") pod \"e2e-worker-w0-0aae2f-openshift-default-5f99b6799-x69j7\" (UID: \"0b878b44-bc26-4e68-94c0-7297413ecd02\") " pod="openshift-ingress/e2e-worker-w0-0aae2f-openshift-default-5f99b6799-x69j7" Nov 04 05:03:30.550966 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:30.550893 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0aae2f-openshift-default-5f99b6799-x69j7" Nov 04 05:03:30.977468 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:30.977444 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-0aae2f-openshift-default-5f99b6799-x69j7"] Nov 04 05:03:30.979408 ip-10-0-133-180 kubenswrapper[4509]: W1104 05:03:30.979378 4509 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b878b44_bc26_4e68_94c0_7297413ecd02.slice/crio-756907cd8bca0214d67e6b71c66f72db10dd331054eac548520927d74eaad1fd WatchSource:0}: Error finding container 756907cd8bca0214d67e6b71c66f72db10dd331054eac548520927d74eaad1fd: Status 404 returned error can't find the container with id 756907cd8bca0214d67e6b71c66f72db10dd331054eac548520927d74eaad1fd Nov 04 05:03:30.981920 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:30.981890 4509 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 04 05:03:30.981996 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:30.981969 4509 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 04 05:03:30.982035 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:30.982017 4509 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 04 05:03:31.990479 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:31.990435 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0aae2f-openshift-default-5f99b6799-x69j7" event={"ID":"0b878b44-bc26-4e68-94c0-7297413ecd02","Type":"ContainerStarted","Data":"734f16dad94dc264a31288e392f10f79fd05883259f3782b080dd733d0b9f468"} Nov 04 05:03:31.990982 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:31.990494 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0aae2f-openshift-default-5f99b6799-x69j7" event={"ID":"0b878b44-bc26-4e68-94c0-7297413ecd02","Type":"ContainerStarted","Data":"756907cd8bca0214d67e6b71c66f72db10dd331054eac548520927d74eaad1fd"} Nov 04 05:03:32.163350 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:32.159616 4509 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-0aae2f-openshift-default-5f99b6799-x69j7" podStartSLOduration=2.1595938869999998 podStartE2EDuration="2.159593887s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 05:03:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 05:03:32.159133756 +0000 UTC m=+976.036857270" watchObservedRunningTime="2026-11-04 05:03:32.159593887 +0000 UTC m=+976.037317410" Nov 04 05:03:32.551619 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:32.551581 4509 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-0aae2f-openshift-default-5f99b6799-x69j7" Nov 04 05:03:32.553600 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:32.553574 4509 patch_prober.go:29] interesting pod/e2e-worker-w0-0aae2f-openshift-default-5f99b6799-x69j7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Nov 04 05:03:32.553727 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:32.553635 4509 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0aae2f-openshift-default-5f99b6799-x69j7" podUID="0b878b44-bc26-4e68-94c0-7297413ecd02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Nov 04 05:03:33.551889 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:33.551858 4509 patch_prober.go:29] interesting pod/e2e-worker-w0-0aae2f-openshift-default-5f99b6799-x69j7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Nov 04 05:03:33.552254 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:33.551908 4509 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0aae2f-openshift-default-5f99b6799-x69j7" podUID="0b878b44-bc26-4e68-94c0-7297413ecd02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Nov 04 05:03:34.552304 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:34.552270 4509 patch_prober.go:29] interesting pod/e2e-worker-w0-0aae2f-openshift-default-5f99b6799-x69j7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Nov 04 05:03:34.552733 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:34.552355 4509 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0aae2f-openshift-default-5f99b6799-x69j7" podUID="0b878b44-bc26-4e68-94c0-7297413ecd02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Nov 04 05:03:35.552070 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:35.552028 4509 patch_prober.go:29] interesting pod/e2e-worker-w0-0aae2f-openshift-default-5f99b6799-x69j7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Nov 04 05:03:35.552266 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:35.552089 4509 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0aae2f-openshift-default-5f99b6799-x69j7" podUID="0b878b44-bc26-4e68-94c0-7297413ecd02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Nov 04 05:03:35.780348 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:35.777723 4509 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-cedfca/distinct-w6-cedfca-kserve-54897db797-fsx45" Nov 04 05:03:35.795679 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:35.795649 4509 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-cedfca/distinct-w6-cedfca-kserve-54897db797-fsx45" Nov 04 05:03:36.022257 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:36.022230 4509 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-cedfca/distinct-w6-cedfca-kserve-54897db797-fsx45" Nov 04 05:03:36.551651 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:36.551612 4509 patch_prober.go:29] interesting pod/e2e-worker-w0-0aae2f-openshift-default-5f99b6799-x69j7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Nov 04 05:03:36.551867 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:36.551670 4509 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0aae2f-openshift-default-5f99b6799-x69j7" podUID="0b878b44-bc26-4e68-94c0-7297413ecd02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Nov 04 05:03:37.555977 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:37.555950 4509 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-0aae2f-openshift-default-5f99b6799-x69j7" Nov 04 05:03:37.556420 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:37.556356 4509 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-0aae2f-openshift-default-5f99b6799-x69j7" Nov 04 05:03:37.556991 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:37.556974 4509 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-0aae2f-openshift-default-5f99b6799-x69j7" Nov 04 05:03:39.244855 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:39.244817 4509 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-836dd023-qv9dl"] Nov 04 05:03:39.248816 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:39.248792 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-836dd023-qv9dl" Nov 04 05:03:39.306104 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:39.306064 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-836dd023-qv9dl"] Nov 04 05:03:39.324706 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:39.324680 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e213f5c3-01ac-485d-a99b-59afd30ab150-maas-api-xks-ca\") pod \"maas-api-revoke-sub-836dd023-qv9dl\" (UID: \"e213f5c3-01ac-485d-a99b-59afd30ab150\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-836dd023-qv9dl" Nov 04 05:03:39.324938 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:39.324915 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58678\" (UniqueName: \"kubernetes.io/projected/e213f5c3-01ac-485d-a99b-59afd30ab150-kube-api-access-58678\") pod \"maas-api-revoke-sub-836dd023-qv9dl\" (UID: \"e213f5c3-01ac-485d-a99b-59afd30ab150\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-836dd023-qv9dl" Nov 04 05:03:39.325102 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:39.325077 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e213f5c3-01ac-485d-a99b-59afd30ab150-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-836dd023-qv9dl\" (UID: \"e213f5c3-01ac-485d-a99b-59afd30ab150\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-836dd023-qv9dl" Nov 04 05:03:39.352295 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:39.352238 4509 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 05:03:39.352295 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:39.352255 4509 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 05:03:39.352295 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:39.352274 4509 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 05:03:39.354403 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:39.354387 4509 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-krgvj\"" Nov 04 05:03:39.426218 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:39.426174 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e213f5c3-01ac-485d-a99b-59afd30ab150-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-836dd023-qv9dl\" (UID: \"e213f5c3-01ac-485d-a99b-59afd30ab150\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-836dd023-qv9dl" Nov 04 05:03:39.426415 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:39.426287 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e213f5c3-01ac-485d-a99b-59afd30ab150-maas-api-xks-ca\") pod \"maas-api-revoke-sub-836dd023-qv9dl\" (UID: \"e213f5c3-01ac-485d-a99b-59afd30ab150\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-836dd023-qv9dl" Nov 04 05:03:39.426415 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:39.426398 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58678\" (UniqueName: \"kubernetes.io/projected/e213f5c3-01ac-485d-a99b-59afd30ab150-kube-api-access-58678\") pod \"maas-api-revoke-sub-836dd023-qv9dl\" (UID: \"e213f5c3-01ac-485d-a99b-59afd30ab150\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-836dd023-qv9dl" Nov 04 05:03:39.426954 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:39.426935 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e213f5c3-01ac-485d-a99b-59afd30ab150-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-836dd023-qv9dl\" (UID: \"e213f5c3-01ac-485d-a99b-59afd30ab150\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-836dd023-qv9dl" Nov 04 05:03:39.429027 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:39.428994 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e213f5c3-01ac-485d-a99b-59afd30ab150-maas-api-xks-ca\") pod \"maas-api-revoke-sub-836dd023-qv9dl\" (UID: \"e213f5c3-01ac-485d-a99b-59afd30ab150\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-836dd023-qv9dl" Nov 04 05:03:39.483195 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:39.483158 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58678\" (UniqueName: \"kubernetes.io/projected/e213f5c3-01ac-485d-a99b-59afd30ab150-kube-api-access-58678\") pod \"maas-api-revoke-sub-836dd023-qv9dl\" (UID: \"e213f5c3-01ac-485d-a99b-59afd30ab150\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-836dd023-qv9dl" Nov 04 05:03:39.563392 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:39.563292 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-836dd023-qv9dl" Nov 04 05:03:39.950160 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:39.950133 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-836dd023-qv9dl"] Nov 04 05:03:39.952001 ip-10-0-133-180 kubenswrapper[4509]: W1104 05:03:39.951974 4509 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode213f5c3_01ac_485d_a99b_59afd30ab150.slice/crio-82ef5b8dcf64bce050ec2829981f338db8fdd7bcdbd71e7b44f30dc75b94a0fd WatchSource:0}: Error finding container 82ef5b8dcf64bce050ec2829981f338db8fdd7bcdbd71e7b44f30dc75b94a0fd: Status 404 returned error can't find the container with id 82ef5b8dcf64bce050ec2829981f338db8fdd7bcdbd71e7b44f30dc75b94a0fd Nov 04 05:03:40.038457 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:40.038421 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-836dd023-qv9dl" event={"ID":"e213f5c3-01ac-485d-a99b-59afd30ab150","Type":"ContainerStarted","Data":"82ef5b8dcf64bce050ec2829981f338db8fdd7bcdbd71e7b44f30dc75b94a0fd"} Nov 04 05:03:40.877917 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:40.875036 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-42dnm"] Nov 04 05:03:41.067343 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:41.059085 4509 generic.go:356] "Generic (PLEG): container finished" podID="e213f5c3-01ac-485d-a99b-59afd30ab150" containerID="3ea9f98aa62c7295cd9e0db052eb0ed68db4851d10ac97bf62614efa8ba7775d" exitCode=0 Nov 04 05:03:41.067343 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:41.059308 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-836dd023-qv9dl" event={"ID":"e213f5c3-01ac-485d-a99b-59afd30ab150","Type":"ContainerDied","Data":"3ea9f98aa62c7295cd9e0db052eb0ed68db4851d10ac97bf62614efa8ba7775d"} Nov 04 05:03:42.255731 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:42.255696 4509 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-836dd023-qv9dl" Nov 04 05:03:42.387844 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:42.386782 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e213f5c3-01ac-485d-a99b-59afd30ab150-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e213f5c3-01ac-485d-a99b-59afd30ab150-maas-api-xks-ca\") pod \"e213f5c3-01ac-485d-a99b-59afd30ab150\" (UID: \"e213f5c3-01ac-485d-a99b-59afd30ab150\") " Nov 04 05:03:42.387844 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:42.386844 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e213f5c3-01ac-485d-a99b-59afd30ab150-kube-api-access-58678\" (UniqueName: \"kubernetes.io/projected/e213f5c3-01ac-485d-a99b-59afd30ab150-kube-api-access-58678\") pod \"e213f5c3-01ac-485d-a99b-59afd30ab150\" (UID: \"e213f5c3-01ac-485d-a99b-59afd30ab150\") " Nov 04 05:03:42.387844 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:42.386906 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e213f5c3-01ac-485d-a99b-59afd30ab150-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e213f5c3-01ac-485d-a99b-59afd30ab150-maas-api-openshift-ca\") pod \"e213f5c3-01ac-485d-a99b-59afd30ab150\" (UID: \"e213f5c3-01ac-485d-a99b-59afd30ab150\") " Nov 04 05:03:42.388638 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:42.388298 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e213f5c3-01ac-485d-a99b-59afd30ab150-maas-api-openshift-ca" pod "e213f5c3-01ac-485d-a99b-59afd30ab150" (UID: "e213f5c3-01ac-485d-a99b-59afd30ab150"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 05:03:42.394632 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:42.394589 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e213f5c3-01ac-485d-a99b-59afd30ab150-kube-api-access-58678" pod "e213f5c3-01ac-485d-a99b-59afd30ab150" (UID: "e213f5c3-01ac-485d-a99b-59afd30ab150"). InnerVolumeSpecName "kube-api-access-58678". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 05:03:42.398526 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:42.398497 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e213f5c3-01ac-485d-a99b-59afd30ab150-maas-api-xks-ca" pod "e213f5c3-01ac-485d-a99b-59afd30ab150" (UID: "e213f5c3-01ac-485d-a99b-59afd30ab150"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 05:03:42.489498 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:42.489453 4509 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e213f5c3-01ac-485d-a99b-59afd30ab150-maas-api-openshift-ca\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:03:42.489498 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:42.489491 4509 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e213f5c3-01ac-485d-a99b-59afd30ab150-maas-api-xks-ca\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:03:42.489498 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:42.489506 4509 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58678\" (UniqueName: \"kubernetes.io/projected/e213f5c3-01ac-485d-a99b-59afd30ab150-kube-api-access-58678\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:03:43.075276 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:43.075245 4509 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-836dd023-qv9dl" Nov 04 05:03:43.075276 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:43.075258 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-836dd023-qv9dl" event={"ID":"e213f5c3-01ac-485d-a99b-59afd30ab150","Type":"ContainerDied","Data":"82ef5b8dcf64bce050ec2829981f338db8fdd7bcdbd71e7b44f30dc75b94a0fd"} Nov 04 05:03:43.075520 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:43.075305 4509 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="82ef5b8dcf64bce050ec2829981f338db8fdd7bcdbd71e7b44f30dc75b94a0fd" Nov 04 05:03:44.455751 ip-10-0-133-180 kubenswrapper[4509]: E1104 05:03:44.455715 4509 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmdhl" containerName="extractor" Nov 04 05:03:49.540971 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:49.540933 4509 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fdaae210-nptmk"] Nov 04 05:03:49.541485 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:49.541472 4509 cpu_manager.go:414] "containerMap: removing container" podUID="e213f5c3-01ac-485d-a99b-59afd30ab150" containerName="revoke-keys" Nov 04 05:03:49.541543 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:49.541487 4509 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e213f5c3-01ac-485d-a99b-59afd30ab150" containerName="revoke-keys" Nov 04 05:03:49.541585 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:49.541576 4509 memory_manager.go:365] "RemoveStaleState removing state" podUID="e213f5c3-01ac-485d-a99b-59afd30ab150" containerName="revoke-keys" Nov 04 05:03:49.544396 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:49.544379 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdaae210-nptmk" Nov 04 05:03:49.570245 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:49.570207 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1179b74-131f-49ae-9e4b-da6c1ac617be-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fdaae210-nptmk\" (UID: \"b1179b74-131f-49ae-9e4b-da6c1ac617be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdaae210-nptmk" Nov 04 05:03:49.570430 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:49.570272 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xm4rm\" (UniqueName: \"kubernetes.io/projected/b1179b74-131f-49ae-9e4b-da6c1ac617be-kube-api-access-xm4rm\") pod \"maas-api-revoke-sub-fdaae210-nptmk\" (UID: \"b1179b74-131f-49ae-9e4b-da6c1ac617be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdaae210-nptmk" Nov 04 05:03:49.570478 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:49.570432 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1179b74-131f-49ae-9e4b-da6c1ac617be-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fdaae210-nptmk\" (UID: \"b1179b74-131f-49ae-9e4b-da6c1ac617be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdaae210-nptmk" Nov 04 05:03:49.597883 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:49.597794 4509 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 05:03:49.647883 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:49.647853 4509 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 05:03:49.647883 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:49.647866 4509 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 05:03:49.648102 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:49.647932 4509 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-krgvj\"" Nov 04 05:03:49.650774 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:49.650748 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fdaae210-nptmk"] Nov 04 05:03:49.671847 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:49.671811 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1179b74-131f-49ae-9e4b-da6c1ac617be-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fdaae210-nptmk\" (UID: \"b1179b74-131f-49ae-9e4b-da6c1ac617be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdaae210-nptmk" Nov 04 05:03:49.672049 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:49.671888 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xm4rm\" (UniqueName: \"kubernetes.io/projected/b1179b74-131f-49ae-9e4b-da6c1ac617be-kube-api-access-xm4rm\") pod \"maas-api-revoke-sub-fdaae210-nptmk\" (UID: \"b1179b74-131f-49ae-9e4b-da6c1ac617be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdaae210-nptmk" Nov 04 05:03:49.672049 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:49.671956 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1179b74-131f-49ae-9e4b-da6c1ac617be-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fdaae210-nptmk\" (UID: \"b1179b74-131f-49ae-9e4b-da6c1ac617be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdaae210-nptmk" Nov 04 05:03:49.672692 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:49.672668 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1179b74-131f-49ae-9e4b-da6c1ac617be-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fdaae210-nptmk\" (UID: \"b1179b74-131f-49ae-9e4b-da6c1ac617be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdaae210-nptmk" Nov 04 05:03:49.674621 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:49.674597 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1179b74-131f-49ae-9e4b-da6c1ac617be-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fdaae210-nptmk\" (UID: \"b1179b74-131f-49ae-9e4b-da6c1ac617be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdaae210-nptmk" Nov 04 05:03:49.730163 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:49.730135 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xm4rm\" (UniqueName: \"kubernetes.io/projected/b1179b74-131f-49ae-9e4b-da6c1ac617be-kube-api-access-xm4rm\") pod \"maas-api-revoke-sub-fdaae210-nptmk\" (UID: \"b1179b74-131f-49ae-9e4b-da6c1ac617be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdaae210-nptmk" Nov 04 05:03:49.857874 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:49.857789 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdaae210-nptmk" Nov 04 05:03:50.235001 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:50.234968 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fdaae210-nptmk"] Nov 04 05:03:50.236762 ip-10-0-133-180 kubenswrapper[4509]: W1104 05:03:50.236732 4509 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1179b74_131f_49ae_9e4b_da6c1ac617be.slice/crio-73dedf94071ded00cff6fa6bd0c6ff1d45a03ee50af5cf66ba6542184fa4fbb6 WatchSource:0}: Error finding container 73dedf94071ded00cff6fa6bd0c6ff1d45a03ee50af5cf66ba6542184fa4fbb6: Status 404 returned error can't find the container with id 73dedf94071ded00cff6fa6bd0c6ff1d45a03ee50af5cf66ba6542184fa4fbb6 Nov 04 05:03:50.513005 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:50.512924 4509 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85f49706-9jtsg"] Nov 04 05:03:50.518158 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:50.518135 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85f49706-9jtsg" Nov 04 05:03:50.573440 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:50.573394 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85f49706-9jtsg"] Nov 04 05:03:50.584248 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:50.584209 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdzff\" (UniqueName: \"kubernetes.io/projected/fdc76a64-cc07-4e07-b02d-3007a7f1e83f-kube-api-access-hdzff\") pod \"maas-api-revoke-sub-85f49706-9jtsg\" (UID: \"fdc76a64-cc07-4e07-b02d-3007a7f1e83f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85f49706-9jtsg" Nov 04 05:03:50.584550 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:50.584348 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdc76a64-cc07-4e07-b02d-3007a7f1e83f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85f49706-9jtsg\" (UID: \"fdc76a64-cc07-4e07-b02d-3007a7f1e83f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85f49706-9jtsg" Nov 04 05:03:50.584550 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:50.584487 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdc76a64-cc07-4e07-b02d-3007a7f1e83f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85f49706-9jtsg\" (UID: \"fdc76a64-cc07-4e07-b02d-3007a7f1e83f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85f49706-9jtsg" Nov 04 05:03:50.685970 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:50.685938 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdzff\" (UniqueName: \"kubernetes.io/projected/fdc76a64-cc07-4e07-b02d-3007a7f1e83f-kube-api-access-hdzff\") pod \"maas-api-revoke-sub-85f49706-9jtsg\" (UID: \"fdc76a64-cc07-4e07-b02d-3007a7f1e83f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85f49706-9jtsg" Nov 04 05:03:50.686141 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:50.686027 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdc76a64-cc07-4e07-b02d-3007a7f1e83f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85f49706-9jtsg\" (UID: \"fdc76a64-cc07-4e07-b02d-3007a7f1e83f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85f49706-9jtsg" Nov 04 05:03:50.686212 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:50.686147 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdc76a64-cc07-4e07-b02d-3007a7f1e83f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85f49706-9jtsg\" (UID: \"fdc76a64-cc07-4e07-b02d-3007a7f1e83f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85f49706-9jtsg" Nov 04 05:03:50.686831 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:50.686809 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdc76a64-cc07-4e07-b02d-3007a7f1e83f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85f49706-9jtsg\" (UID: \"fdc76a64-cc07-4e07-b02d-3007a7f1e83f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85f49706-9jtsg" Nov 04 05:03:50.688560 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:50.688539 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdc76a64-cc07-4e07-b02d-3007a7f1e83f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85f49706-9jtsg\" (UID: \"fdc76a64-cc07-4e07-b02d-3007a7f1e83f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85f49706-9jtsg" Nov 04 05:03:50.742989 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:50.742955 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdzff\" (UniqueName: \"kubernetes.io/projected/fdc76a64-cc07-4e07-b02d-3007a7f1e83f-kube-api-access-hdzff\") pod \"maas-api-revoke-sub-85f49706-9jtsg\" (UID: \"fdc76a64-cc07-4e07-b02d-3007a7f1e83f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85f49706-9jtsg" Nov 04 05:03:50.835104 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:50.835011 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85f49706-9jtsg" Nov 04 05:03:51.124580 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:51.124554 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdaae210-nptmk" event={"ID":"b1179b74-131f-49ae-9e4b-da6c1ac617be","Type":"ContainerStarted","Data":"39ee30ed5c2ddd85728e13268b7e011a7b8de794701c731d989e7c78e6e8f94b"} Nov 04 05:03:51.124712 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:51.124600 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdaae210-nptmk" event={"ID":"b1179b74-131f-49ae-9e4b-da6c1ac617be","Type":"ContainerStarted","Data":"73dedf94071ded00cff6fa6bd0c6ff1d45a03ee50af5cf66ba6542184fa4fbb6"} Nov 04 05:03:51.214275 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:51.214245 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85f49706-9jtsg"] Nov 04 05:03:51.231288 ip-10-0-133-180 kubenswrapper[4509]: W1104 05:03:51.231251 4509 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdc76a64_cc07_4e07_b02d_3007a7f1e83f.slice/crio-dda35e30ee6eb33a0a68c58049ffed0c01d78ceb72682d3061d5e7413643b3bc WatchSource:0}: Error finding container dda35e30ee6eb33a0a68c58049ffed0c01d78ceb72682d3061d5e7413643b3bc: Status 404 returned error can't find the container with id dda35e30ee6eb33a0a68c58049ffed0c01d78ceb72682d3061d5e7413643b3bc Nov 04 05:03:51.238367 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:51.238288 4509 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdaae210-nptmk" podStartSLOduration=2.238269918 podStartE2EDuration="2.238269918s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 05:03:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 05:03:51.236689975 +0000 UTC m=+995.114413496" watchObservedRunningTime="2026-11-04 05:03:51.238269918 +0000 UTC m=+995.115993454" Nov 04 05:03:52.133071 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:52.133026 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85f49706-9jtsg" event={"ID":"fdc76a64-cc07-4e07-b02d-3007a7f1e83f","Type":"ContainerStarted","Data":"9b0c3c9eb3d51eaeb33000115753efedda6be235980196e90013f85ac82561e0"} Nov 04 05:03:52.133566 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:52.133107 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85f49706-9jtsg" event={"ID":"fdc76a64-cc07-4e07-b02d-3007a7f1e83f","Type":"ContainerStarted","Data":"dda35e30ee6eb33a0a68c58049ffed0c01d78ceb72682d3061d5e7413643b3bc"} Nov 04 05:03:52.134838 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:52.134807 4509 generic.go:356] "Generic (PLEG): container finished" podID="b1179b74-131f-49ae-9e4b-da6c1ac617be" containerID="39ee30ed5c2ddd85728e13268b7e011a7b8de794701c731d989e7c78e6e8f94b" exitCode=0 Nov 04 05:03:52.134960 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:52.134881 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdaae210-nptmk" event={"ID":"b1179b74-131f-49ae-9e4b-da6c1ac617be","Type":"ContainerDied","Data":"39ee30ed5c2ddd85728e13268b7e011a7b8de794701c731d989e7c78e6e8f94b"} Nov 04 05:03:52.247748 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:52.247696 4509 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85f49706-9jtsg" podStartSLOduration=2.247680828 podStartE2EDuration="2.247680828s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 05:03:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 05:03:52.245449903 +0000 UTC m=+996.123173424" watchObservedRunningTime="2026-11-04 05:03:52.247680828 +0000 UTC m=+996.125404348" Nov 04 05:03:52.450239 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:52.450133 4509 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-0aae2f-74664f8f76-rzd5v"] Nov 04 05:03:52.562052 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:52.562021 4509 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-0aae2f-5ff7778f8c-4tbhr"] Nov 04 05:03:53.142379 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:53.142314 4509 generic.go:356] "Generic (PLEG): container finished" podID="fdc76a64-cc07-4e07-b02d-3007a7f1e83f" containerID="9b0c3c9eb3d51eaeb33000115753efedda6be235980196e90013f85ac82561e0" exitCode=0 Nov 04 05:03:53.142817 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:53.142428 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85f49706-9jtsg" event={"ID":"fdc76a64-cc07-4e07-b02d-3007a7f1e83f","Type":"ContainerDied","Data":"9b0c3c9eb3d51eaeb33000115753efedda6be235980196e90013f85ac82561e0"} Nov 04 05:03:53.298953 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:53.298925 4509 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdaae210-nptmk" Nov 04 05:03:53.441484 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:53.441390 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1179b74-131f-49ae-9e4b-da6c1ac617be-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1179b74-131f-49ae-9e4b-da6c1ac617be-maas-api-openshift-ca\") pod \"b1179b74-131f-49ae-9e4b-da6c1ac617be\" (UID: \"b1179b74-131f-49ae-9e4b-da6c1ac617be\") " Nov 04 05:03:53.441648 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:53.441509 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1179b74-131f-49ae-9e4b-da6c1ac617be-kube-api-access-xm4rm\" (UniqueName: \"kubernetes.io/projected/b1179b74-131f-49ae-9e4b-da6c1ac617be-kube-api-access-xm4rm\") pod \"b1179b74-131f-49ae-9e4b-da6c1ac617be\" (UID: \"b1179b74-131f-49ae-9e4b-da6c1ac617be\") " Nov 04 05:03:53.441712 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:53.441691 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b1179b74-131f-49ae-9e4b-da6c1ac617be-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1179b74-131f-49ae-9e4b-da6c1ac617be-maas-api-xks-ca\") pod \"b1179b74-131f-49ae-9e4b-da6c1ac617be\" (UID: \"b1179b74-131f-49ae-9e4b-da6c1ac617be\") " Nov 04 05:03:53.441807 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:53.441760 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1179b74-131f-49ae-9e4b-da6c1ac617be-maas-api-openshift-ca" pod "b1179b74-131f-49ae-9e4b-da6c1ac617be" (UID: "b1179b74-131f-49ae-9e4b-da6c1ac617be"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 05:03:53.442271 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:53.442245 4509 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1179b74-131f-49ae-9e4b-da6c1ac617be-maas-api-openshift-ca\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:03:53.444136 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:53.444107 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1179b74-131f-49ae-9e4b-da6c1ac617be-kube-api-access-xm4rm" pod "b1179b74-131f-49ae-9e4b-da6c1ac617be" (UID: "b1179b74-131f-49ae-9e4b-da6c1ac617be"). InnerVolumeSpecName "kube-api-access-xm4rm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 05:03:53.444482 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:53.444439 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1179b74-131f-49ae-9e4b-da6c1ac617be-maas-api-xks-ca" pod "b1179b74-131f-49ae-9e4b-da6c1ac617be" (UID: "b1179b74-131f-49ae-9e4b-da6c1ac617be"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 05:03:53.543799 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:53.543763 4509 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xm4rm\" (UniqueName: \"kubernetes.io/projected/b1179b74-131f-49ae-9e4b-da6c1ac617be-kube-api-access-xm4rm\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:03:53.543799 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:53.543792 4509 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1179b74-131f-49ae-9e4b-da6c1ac617be-maas-api-xks-ca\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:03:54.225524 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:54.225484 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdaae210-nptmk" event={"ID":"b1179b74-131f-49ae-9e4b-da6c1ac617be","Type":"ContainerDied","Data":"73dedf94071ded00cff6fa6bd0c6ff1d45a03ee50af5cf66ba6542184fa4fbb6"} Nov 04 05:03:54.225973 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:54.225698 4509 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="73dedf94071ded00cff6fa6bd0c6ff1d45a03ee50af5cf66ba6542184fa4fbb6" Nov 04 05:03:54.225973 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:54.225783 4509 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdaae210-nptmk" Nov 04 05:03:54.400249 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:54.400222 4509 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85f49706-9jtsg" Nov 04 05:03:54.490802 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:54.490693 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdc76a64-cc07-4e07-b02d-3007a7f1e83f-kube-api-access-hdzff\" (UniqueName: \"kubernetes.io/projected/fdc76a64-cc07-4e07-b02d-3007a7f1e83f-kube-api-access-hdzff\") pod \"fdc76a64-cc07-4e07-b02d-3007a7f1e83f\" (UID: \"fdc76a64-cc07-4e07-b02d-3007a7f1e83f\") " Nov 04 05:03:54.490972 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:54.490846 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fdc76a64-cc07-4e07-b02d-3007a7f1e83f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdc76a64-cc07-4e07-b02d-3007a7f1e83f-maas-api-xks-ca\") pod \"fdc76a64-cc07-4e07-b02d-3007a7f1e83f\" (UID: \"fdc76a64-cc07-4e07-b02d-3007a7f1e83f\") " Nov 04 05:03:54.490972 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:54.490963 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fdc76a64-cc07-4e07-b02d-3007a7f1e83f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdc76a64-cc07-4e07-b02d-3007a7f1e83f-maas-api-openshift-ca\") pod \"fdc76a64-cc07-4e07-b02d-3007a7f1e83f\" (UID: \"fdc76a64-cc07-4e07-b02d-3007a7f1e83f\") " Nov 04 05:03:54.491444 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:54.491416 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdc76a64-cc07-4e07-b02d-3007a7f1e83f-maas-api-openshift-ca" pod "fdc76a64-cc07-4e07-b02d-3007a7f1e83f" (UID: "fdc76a64-cc07-4e07-b02d-3007a7f1e83f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 05:03:54.493636 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:54.493605 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdc76a64-cc07-4e07-b02d-3007a7f1e83f-kube-api-access-hdzff" pod "fdc76a64-cc07-4e07-b02d-3007a7f1e83f" (UID: "fdc76a64-cc07-4e07-b02d-3007a7f1e83f"). InnerVolumeSpecName "kube-api-access-hdzff". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 05:03:54.493756 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:54.493672 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fdc76a64-cc07-4e07-b02d-3007a7f1e83f-maas-api-xks-ca" pod "fdc76a64-cc07-4e07-b02d-3007a7f1e83f" (UID: "fdc76a64-cc07-4e07-b02d-3007a7f1e83f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 05:03:54.517921 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:54.517883 4509 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-1874d8/simulator-w1-1874d8-kserve-85dc6d466-lkxlz"] Nov 04 05:03:54.592888 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:54.592835 4509 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdc76a64-cc07-4e07-b02d-3007a7f1e83f-maas-api-xks-ca\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:03:54.592888 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:54.592880 4509 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdc76a64-cc07-4e07-b02d-3007a7f1e83f-maas-api-openshift-ca\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:03:54.592888 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:54.592898 4509 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdzff\" (UniqueName: \"kubernetes.io/projected/fdc76a64-cc07-4e07-b02d-3007a7f1e83f-kube-api-access-hdzff\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:03:55.232932 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:55.232902 4509 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85f49706-9jtsg" Nov 04 05:03:55.232932 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:55.232924 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85f49706-9jtsg" event={"ID":"fdc76a64-cc07-4e07-b02d-3007a7f1e83f","Type":"ContainerDied","Data":"dda35e30ee6eb33a0a68c58049ffed0c01d78ceb72682d3061d5e7413643b3bc"} Nov 04 05:03:55.233384 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:55.232977 4509 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dda35e30ee6eb33a0a68c58049ffed0c01d78ceb72682d3061d5e7413643b3bc" Nov 04 05:03:57.450903 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:57.450851 4509 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-0aae2f-74664f8f76-rzd5v" podUID="2f1d0cf5-2aa4-49c7-9153-4bc44df28fac" containerName="payload-processing" containerID="cri-o://bc2a4e535a03a824ce7352aadb71e3507d7d92f70e2b9156e11966ff5d27f647" gracePeriod=25 Nov 04 05:03:57.562861 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:57.562820 4509 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-0aae2f-5ff7778f8c-4tbhr" podUID="e1f20ac4-af33-4c51-8ae3-a286a4c9bc2c" containerName="payload-pre-processing" containerID="cri-o://08f4b0e09701b0cc67798a27385f32cf7c5aa1f27a06a63ed3afbd73c6c74079" gracePeriod=25 Nov 04 05:03:57.712749 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:57.712727 4509 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-0aae2f-74664f8f76-rzd5v" Nov 04 05:03:57.831768 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:57.831736 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f1d0cf5-2aa4-49c7-9153-4bc44df28fac-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f1d0cf5-2aa4-49c7-9153-4bc44df28fac-plugins-config-volume\") pod \"2f1d0cf5-2aa4-49c7-9153-4bc44df28fac\" (UID: \"2f1d0cf5-2aa4-49c7-9153-4bc44df28fac\") " Nov 04 05:03:57.831940 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:57.831880 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f1d0cf5-2aa4-49c7-9153-4bc44df28fac-kube-api-access-sjhzd\" (UniqueName: \"kubernetes.io/projected/2f1d0cf5-2aa4-49c7-9153-4bc44df28fac-kube-api-access-sjhzd\") pod \"2f1d0cf5-2aa4-49c7-9153-4bc44df28fac\" (UID: \"2f1d0cf5-2aa4-49c7-9153-4bc44df28fac\") " Nov 04 05:03:57.832138 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:57.832110 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f1d0cf5-2aa4-49c7-9153-4bc44df28fac-plugins-config-volume" pod "2f1d0cf5-2aa4-49c7-9153-4bc44df28fac" (UID: "2f1d0cf5-2aa4-49c7-9153-4bc44df28fac"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 05:03:57.832432 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:57.832395 4509 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f1d0cf5-2aa4-49c7-9153-4bc44df28fac-plugins-config-volume\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:03:57.834222 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:57.834195 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f1d0cf5-2aa4-49c7-9153-4bc44df28fac-kube-api-access-sjhzd" pod "2f1d0cf5-2aa4-49c7-9153-4bc44df28fac" (UID: "2f1d0cf5-2aa4-49c7-9153-4bc44df28fac"). InnerVolumeSpecName "kube-api-access-sjhzd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 05:03:57.933919 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:57.933892 4509 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjhzd\" (UniqueName: \"kubernetes.io/projected/2f1d0cf5-2aa4-49c7-9153-4bc44df28fac-kube-api-access-sjhzd\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:03:58.104939 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:58.104846 4509 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-0aae2f-5ff7778f8c-4tbhr" podUID="e1f20ac4-af33-4c51-8ae3-a286a4c9bc2c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 05:03:58.251944 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:58.251909 4509 generic.go:356] "Generic (PLEG): container finished" podID="2f1d0cf5-2aa4-49c7-9153-4bc44df28fac" containerID="bc2a4e535a03a824ce7352aadb71e3507d7d92f70e2b9156e11966ff5d27f647" exitCode=0 Nov 04 05:03:58.252140 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:58.251970 4509 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-0aae2f-74664f8f76-rzd5v" Nov 04 05:03:58.252140 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:58.251994 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-0aae2f-74664f8f76-rzd5v" event={"ID":"2f1d0cf5-2aa4-49c7-9153-4bc44df28fac","Type":"ContainerDied","Data":"bc2a4e535a03a824ce7352aadb71e3507d7d92f70e2b9156e11966ff5d27f647"} Nov 04 05:03:58.252140 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:58.252048 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-0aae2f-74664f8f76-rzd5v" event={"ID":"2f1d0cf5-2aa4-49c7-9153-4bc44df28fac","Type":"ContainerDied","Data":"8c122dd7e58a8662c323cbdcab4f10d430f1c381fa8a1287960f70746a29386b"} Nov 04 05:03:58.252140 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:58.252070 4509 scope.go:122] "RemoveContainer" containerID="bc2a4e535a03a824ce7352aadb71e3507d7d92f70e2b9156e11966ff5d27f647" Nov 04 05:03:58.272200 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:58.272175 4509 scope.go:122] "RemoveContainer" containerID="bc2a4e535a03a824ce7352aadb71e3507d7d92f70e2b9156e11966ff5d27f647" Nov 04 05:03:58.273068 ip-10-0-133-180 kubenswrapper[4509]: E1104 05:03:58.272680 4509 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc2a4e535a03a824ce7352aadb71e3507d7d92f70e2b9156e11966ff5d27f647\": container with ID starting with bc2a4e535a03a824ce7352aadb71e3507d7d92f70e2b9156e11966ff5d27f647 not found: ID does not exist" containerID="bc2a4e535a03a824ce7352aadb71e3507d7d92f70e2b9156e11966ff5d27f647" Nov 04 05:03:58.273192 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:58.273083 4509 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc2a4e535a03a824ce7352aadb71e3507d7d92f70e2b9156e11966ff5d27f647"} err="failed to get container status \"bc2a4e535a03a824ce7352aadb71e3507d7d92f70e2b9156e11966ff5d27f647\": rpc error: code = NotFound desc = could not find container \"bc2a4e535a03a824ce7352aadb71e3507d7d92f70e2b9156e11966ff5d27f647\": container with ID starting with bc2a4e535a03a824ce7352aadb71e3507d7d92f70e2b9156e11966ff5d27f647 not found: ID does not exist" Nov 04 05:03:58.417698 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:58.417621 4509 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-0aae2f-74664f8f76-rzd5v"] Nov 04 05:03:58.421443 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:03:58.421417 4509 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-0aae2f-74664f8f76-rzd5v"] Nov 04 05:04:00.263557 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:00.263520 4509 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f1d0cf5-2aa4-49c7-9153-4bc44df28fac" path="/var/lib/kubelet/pods/2f1d0cf5-2aa4-49c7-9153-4bc44df28fac/volumes" Nov 04 05:04:06.687959 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:06.687917 4509 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-1874d8-78f4d9f8b7-9d2tj"] Nov 04 05:04:07.216747 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:07.216715 4509 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c621778-k2mxk"] Nov 04 05:04:07.217224 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:07.217210 4509 cpu_manager.go:414] "containerMap: removing container" podUID="2f1d0cf5-2aa4-49c7-9153-4bc44df28fac" containerName="payload-processing" Nov 04 05:04:07.217276 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:07.217225 4509 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f1d0cf5-2aa4-49c7-9153-4bc44df28fac" containerName="payload-processing" Nov 04 05:04:07.217276 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:07.217245 4509 cpu_manager.go:414] "containerMap: removing container" podUID="b1179b74-131f-49ae-9e4b-da6c1ac617be" containerName="revoke-keys" Nov 04 05:04:07.217276 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:07.217250 4509 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1179b74-131f-49ae-9e4b-da6c1ac617be" containerName="revoke-keys" Nov 04 05:04:07.217276 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:07.217257 4509 cpu_manager.go:414] "containerMap: removing container" podUID="fdc76a64-cc07-4e07-b02d-3007a7f1e83f" containerName="revoke-keys" Nov 04 05:04:07.217276 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:07.217262 4509 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdc76a64-cc07-4e07-b02d-3007a7f1e83f" containerName="revoke-keys" Nov 04 05:04:07.217479 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:07.217372 4509 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1179b74-131f-49ae-9e4b-da6c1ac617be" containerName="revoke-keys" Nov 04 05:04:07.217479 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:07.217380 4509 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdc76a64-cc07-4e07-b02d-3007a7f1e83f" containerName="revoke-keys" Nov 04 05:04:07.217479 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:07.217390 4509 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f1d0cf5-2aa4-49c7-9153-4bc44df28fac" containerName="payload-processing" Nov 04 05:04:07.220653 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:07.220634 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c621778-k2mxk" Nov 04 05:04:07.243070 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:07.243047 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b78d9eb7-681a-4f6e-a61b-790ee1e814fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6c621778-k2mxk\" (UID: \"b78d9eb7-681a-4f6e-a61b-790ee1e814fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c621778-k2mxk" Nov 04 05:04:07.243193 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:07.243177 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkn8f\" (UniqueName: \"kubernetes.io/projected/b78d9eb7-681a-4f6e-a61b-790ee1e814fd-kube-api-access-fkn8f\") pod \"maas-api-revoke-sub-6c621778-k2mxk\" (UID: \"b78d9eb7-681a-4f6e-a61b-790ee1e814fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c621778-k2mxk" Nov 04 05:04:07.243235 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:07.243220 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b78d9eb7-681a-4f6e-a61b-790ee1e814fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6c621778-k2mxk\" (UID: \"b78d9eb7-681a-4f6e-a61b-790ee1e814fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c621778-k2mxk" Nov 04 05:04:07.273507 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:07.273458 4509 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 05:04:07.273650 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:07.273532 4509 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 05:04:07.273650 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:07.273623 4509 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 05:04:07.273817 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:07.273762 4509 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-krgvj\"" Nov 04 05:04:07.277191 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:07.277167 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c621778-k2mxk"] Nov 04 05:04:07.344732 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:07.344697 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkn8f\" (UniqueName: \"kubernetes.io/projected/b78d9eb7-681a-4f6e-a61b-790ee1e814fd-kube-api-access-fkn8f\") pod \"maas-api-revoke-sub-6c621778-k2mxk\" (UID: \"b78d9eb7-681a-4f6e-a61b-790ee1e814fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c621778-k2mxk" Nov 04 05:04:07.344911 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:07.344756 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b78d9eb7-681a-4f6e-a61b-790ee1e814fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6c621778-k2mxk\" (UID: \"b78d9eb7-681a-4f6e-a61b-790ee1e814fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c621778-k2mxk" Nov 04 05:04:07.344911 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:07.344841 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b78d9eb7-681a-4f6e-a61b-790ee1e814fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6c621778-k2mxk\" (UID: \"b78d9eb7-681a-4f6e-a61b-790ee1e814fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c621778-k2mxk" Nov 04 05:04:07.345512 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:07.345489 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b78d9eb7-681a-4f6e-a61b-790ee1e814fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6c621778-k2mxk\" (UID: \"b78d9eb7-681a-4f6e-a61b-790ee1e814fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c621778-k2mxk" Nov 04 05:04:07.347204 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:07.347186 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b78d9eb7-681a-4f6e-a61b-790ee1e814fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6c621778-k2mxk\" (UID: \"b78d9eb7-681a-4f6e-a61b-790ee1e814fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c621778-k2mxk" Nov 04 05:04:07.401010 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:07.400979 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkn8f\" (UniqueName: \"kubernetes.io/projected/b78d9eb7-681a-4f6e-a61b-790ee1e814fd-kube-api-access-fkn8f\") pod \"maas-api-revoke-sub-6c621778-k2mxk\" (UID: \"b78d9eb7-681a-4f6e-a61b-790ee1e814fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c621778-k2mxk" Nov 04 05:04:07.534314 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:07.534228 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c621778-k2mxk" Nov 04 05:04:07.912550 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:07.912522 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c621778-k2mxk"] Nov 04 05:04:07.913773 ip-10-0-133-180 kubenswrapper[4509]: W1104 05:04:07.913747 4509 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb78d9eb7_681a_4f6e_a61b_790ee1e814fd.slice/crio-8d26c1d2bad503b3267f09100d8821285fc003aca2b6311da80fa1910dfa9d23 WatchSource:0}: Error finding container 8d26c1d2bad503b3267f09100d8821285fc003aca2b6311da80fa1910dfa9d23: Status 404 returned error can't find the container with id 8d26c1d2bad503b3267f09100d8821285fc003aca2b6311da80fa1910dfa9d23 Nov 04 05:04:08.104520 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:08.104483 4509 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-0aae2f-5ff7778f8c-4tbhr" podUID="e1f20ac4-af33-4c51-8ae3-a286a4c9bc2c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 05:04:08.314362 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:08.313251 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c621778-k2mxk" event={"ID":"b78d9eb7-681a-4f6e-a61b-790ee1e814fd","Type":"ContainerStarted","Data":"954f1b5cdc2f3ef29fd8139d89b5a36de54be8cb3ed0b05a15005bbd8e310ef1"} Nov 04 05:04:08.314362 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:08.313343 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c621778-k2mxk" event={"ID":"b78d9eb7-681a-4f6e-a61b-790ee1e814fd","Type":"ContainerStarted","Data":"8d26c1d2bad503b3267f09100d8821285fc003aca2b6311da80fa1910dfa9d23"} Nov 04 05:04:08.430942 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:08.430881 4509 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c621778-k2mxk" podStartSLOduration=1.430868172 podStartE2EDuration="1.430868172s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 05:04:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 05:04:08.427538549 +0000 UTC m=+1012.305262072" watchObservedRunningTime="2026-11-04 05:04:08.430868172 +0000 UTC m=+1012.308591693" Nov 04 05:04:09.321220 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:09.321186 4509 generic.go:356] "Generic (PLEG): container finished" podID="b78d9eb7-681a-4f6e-a61b-790ee1e814fd" containerID="954f1b5cdc2f3ef29fd8139d89b5a36de54be8cb3ed0b05a15005bbd8e310ef1" exitCode=0 Nov 04 05:04:09.321622 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:09.321226 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c621778-k2mxk" event={"ID":"b78d9eb7-681a-4f6e-a61b-790ee1e814fd","Type":"ContainerDied","Data":"954f1b5cdc2f3ef29fd8139d89b5a36de54be8cb3ed0b05a15005bbd8e310ef1"} Nov 04 05:04:09.543254 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:09.543212 4509 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-1874d8/simulator-w1-1874d8-kserve-85dc6d466-lkxlz" podUID="4655c4c9-ad98-4f20-919c-56905e179bad" containerName="main" containerID="cri-o://c7d9a605c5402d9e1ebb3091313249e4dc823474049b155d63ba61482479d721" gracePeriod=45 Nov 04 05:04:09.821165 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:09.821128 4509 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-1874d8/simulator-w1-1874d8-kserve-85dc6d466-lkxlz" Nov 04 05:04:09.873114 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:09.873080 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4655c4c9-ad98-4f20-919c-56905e179bad-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4655c4c9-ad98-4f20-919c-56905e179bad-dshm\") pod \"4655c4c9-ad98-4f20-919c-56905e179bad\" (UID: \"4655c4c9-ad98-4f20-919c-56905e179bad\") " Nov 04 05:04:09.873286 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:09.873170 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4655c4c9-ad98-4f20-919c-56905e179bad-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4655c4c9-ad98-4f20-919c-56905e179bad-model-cache\") pod \"4655c4c9-ad98-4f20-919c-56905e179bad\" (UID: \"4655c4c9-ad98-4f20-919c-56905e179bad\") " Nov 04 05:04:09.873286 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:09.873205 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4655c4c9-ad98-4f20-919c-56905e179bad-kube-api-access-wcqtt\" (UniqueName: \"kubernetes.io/projected/4655c4c9-ad98-4f20-919c-56905e179bad-kube-api-access-wcqtt\") pod \"4655c4c9-ad98-4f20-919c-56905e179bad\" (UID: \"4655c4c9-ad98-4f20-919c-56905e179bad\") " Nov 04 05:04:09.873432 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:09.873302 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4655c4c9-ad98-4f20-919c-56905e179bad-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4655c4c9-ad98-4f20-919c-56905e179bad-tmp-dir\") pod \"4655c4c9-ad98-4f20-919c-56905e179bad\" (UID: \"4655c4c9-ad98-4f20-919c-56905e179bad\") " Nov 04 05:04:09.873432 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:09.873355 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4655c4c9-ad98-4f20-919c-56905e179bad-home\" (UniqueName: \"kubernetes.io/empty-dir/4655c4c9-ad98-4f20-919c-56905e179bad-home\") pod \"4655c4c9-ad98-4f20-919c-56905e179bad\" (UID: \"4655c4c9-ad98-4f20-919c-56905e179bad\") " Nov 04 05:04:09.873531 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:09.873441 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4655c4c9-ad98-4f20-919c-56905e179bad-tls-certs\" (UniqueName: \"kubernetes.io/secret/4655c4c9-ad98-4f20-919c-56905e179bad-tls-certs\") pod \"4655c4c9-ad98-4f20-919c-56905e179bad\" (UID: \"4655c4c9-ad98-4f20-919c-56905e179bad\") " Nov 04 05:04:09.873731 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:09.873433 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4655c4c9-ad98-4f20-919c-56905e179bad-model-cache" pod "4655c4c9-ad98-4f20-919c-56905e179bad" (UID: "4655c4c9-ad98-4f20-919c-56905e179bad"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:04:09.873731 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:09.873638 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4655c4c9-ad98-4f20-919c-56905e179bad-home" pod "4655c4c9-ad98-4f20-919c-56905e179bad" (UID: "4655c4c9-ad98-4f20-919c-56905e179bad"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:04:09.873899 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:09.873771 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4655c4c9-ad98-4f20-919c-56905e179bad-tmp-dir" pod "4655c4c9-ad98-4f20-919c-56905e179bad" (UID: "4655c4c9-ad98-4f20-919c-56905e179bad"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:04:09.874179 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:09.874160 4509 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4655c4c9-ad98-4f20-919c-56905e179bad-tmp-dir\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:04:09.874242 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:09.874186 4509 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4655c4c9-ad98-4f20-919c-56905e179bad-home\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:04:09.874242 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:09.874200 4509 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4655c4c9-ad98-4f20-919c-56905e179bad-model-cache\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:04:09.875908 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:09.875886 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4655c4c9-ad98-4f20-919c-56905e179bad-kube-api-access-wcqtt" pod "4655c4c9-ad98-4f20-919c-56905e179bad" (UID: "4655c4c9-ad98-4f20-919c-56905e179bad"). InnerVolumeSpecName "kube-api-access-wcqtt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 05:04:09.876026 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:09.876005 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4655c4c9-ad98-4f20-919c-56905e179bad-dshm" pod "4655c4c9-ad98-4f20-919c-56905e179bad" (UID: "4655c4c9-ad98-4f20-919c-56905e179bad"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:04:09.876026 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:09.876011 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4655c4c9-ad98-4f20-919c-56905e179bad-tls-certs" pod "4655c4c9-ad98-4f20-919c-56905e179bad" (UID: "4655c4c9-ad98-4f20-919c-56905e179bad"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 05:04:09.974965 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:09.974938 4509 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wcqtt\" (UniqueName: \"kubernetes.io/projected/4655c4c9-ad98-4f20-919c-56905e179bad-kube-api-access-wcqtt\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:04:09.974965 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:09.974964 4509 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4655c4c9-ad98-4f20-919c-56905e179bad-tls-certs\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:04:09.975140 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:09.974975 4509 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4655c4c9-ad98-4f20-919c-56905e179bad-dshm\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:04:10.328989 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:10.328953 4509 generic.go:356] "Generic (PLEG): container finished" podID="4655c4c9-ad98-4f20-919c-56905e179bad" containerID="c7d9a605c5402d9e1ebb3091313249e4dc823474049b155d63ba61482479d721" exitCode=0 Nov 04 05:04:10.329462 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:10.329013 4509 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-1874d8/simulator-w1-1874d8-kserve-85dc6d466-lkxlz" Nov 04 05:04:10.329462 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:10.329036 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-1874d8/simulator-w1-1874d8-kserve-85dc6d466-lkxlz" event={"ID":"4655c4c9-ad98-4f20-919c-56905e179bad","Type":"ContainerDied","Data":"c7d9a605c5402d9e1ebb3091313249e4dc823474049b155d63ba61482479d721"} Nov 04 05:04:10.329462 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:10.329087 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-1874d8/simulator-w1-1874d8-kserve-85dc6d466-lkxlz" event={"ID":"4655c4c9-ad98-4f20-919c-56905e179bad","Type":"ContainerDied","Data":"cc13bec347afb47dc11fef8ec243927089bea9c1a5584f566c85c9cbe4f5a68b"} Nov 04 05:04:10.329462 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:10.329104 4509 scope.go:122] "RemoveContainer" containerID="c7d9a605c5402d9e1ebb3091313249e4dc823474049b155d63ba61482479d721" Nov 04 05:04:10.338482 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:10.338465 4509 scope.go:122] "RemoveContainer" containerID="c7d9a605c5402d9e1ebb3091313249e4dc823474049b155d63ba61482479d721" Nov 04 05:04:10.338767 ip-10-0-133-180 kubenswrapper[4509]: E1104 05:04:10.338741 4509 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7d9a605c5402d9e1ebb3091313249e4dc823474049b155d63ba61482479d721\": container with ID starting with c7d9a605c5402d9e1ebb3091313249e4dc823474049b155d63ba61482479d721 not found: ID does not exist" containerID="c7d9a605c5402d9e1ebb3091313249e4dc823474049b155d63ba61482479d721" Nov 04 05:04:10.338826 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:10.338776 4509 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7d9a605c5402d9e1ebb3091313249e4dc823474049b155d63ba61482479d721"} err="failed to get container status \"c7d9a605c5402d9e1ebb3091313249e4dc823474049b155d63ba61482479d721\": rpc error: code = NotFound desc = could not find container \"c7d9a605c5402d9e1ebb3091313249e4dc823474049b155d63ba61482479d721\": container with ID starting with c7d9a605c5402d9e1ebb3091313249e4dc823474049b155d63ba61482479d721 not found: ID does not exist" Nov 04 05:04:10.493259 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:10.493227 4509 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c621778-k2mxk" Nov 04 05:04:10.496632 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:10.496611 4509 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-1874d8/simulator-w1-1874d8-kserve-85dc6d466-lkxlz"] Nov 04 05:04:10.560854 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:10.560533 4509 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-1874d8/simulator-w1-1874d8-kserve-85dc6d466-lkxlz"] Nov 04 05:04:10.585345 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:10.584827 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b78d9eb7-681a-4f6e-a61b-790ee1e814fd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b78d9eb7-681a-4f6e-a61b-790ee1e814fd-maas-api-openshift-ca\") pod \"b78d9eb7-681a-4f6e-a61b-790ee1e814fd\" (UID: \"b78d9eb7-681a-4f6e-a61b-790ee1e814fd\") " Nov 04 05:04:10.585345 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:10.584925 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b78d9eb7-681a-4f6e-a61b-790ee1e814fd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b78d9eb7-681a-4f6e-a61b-790ee1e814fd-maas-api-xks-ca\") pod \"b78d9eb7-681a-4f6e-a61b-790ee1e814fd\" (UID: \"b78d9eb7-681a-4f6e-a61b-790ee1e814fd\") " Nov 04 05:04:10.585345 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:10.585005 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b78d9eb7-681a-4f6e-a61b-790ee1e814fd-kube-api-access-fkn8f\" (UniqueName: \"kubernetes.io/projected/b78d9eb7-681a-4f6e-a61b-790ee1e814fd-kube-api-access-fkn8f\") pod \"b78d9eb7-681a-4f6e-a61b-790ee1e814fd\" (UID: \"b78d9eb7-681a-4f6e-a61b-790ee1e814fd\") " Nov 04 05:04:10.585345 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:10.585262 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b78d9eb7-681a-4f6e-a61b-790ee1e814fd-maas-api-openshift-ca" pod "b78d9eb7-681a-4f6e-a61b-790ee1e814fd" (UID: "b78d9eb7-681a-4f6e-a61b-790ee1e814fd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 05:04:10.590102 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:10.585984 4509 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b78d9eb7-681a-4f6e-a61b-790ee1e814fd-maas-api-openshift-ca\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:04:10.590102 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:10.588545 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b78d9eb7-681a-4f6e-a61b-790ee1e814fd-maas-api-xks-ca" pod "b78d9eb7-681a-4f6e-a61b-790ee1e814fd" (UID: "b78d9eb7-681a-4f6e-a61b-790ee1e814fd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 05:04:10.591010 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:10.590483 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b78d9eb7-681a-4f6e-a61b-790ee1e814fd-kube-api-access-fkn8f" pod "b78d9eb7-681a-4f6e-a61b-790ee1e814fd" (UID: "b78d9eb7-681a-4f6e-a61b-790ee1e814fd"). InnerVolumeSpecName "kube-api-access-fkn8f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 05:04:10.687753 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:10.687723 4509 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b78d9eb7-681a-4f6e-a61b-790ee1e814fd-maas-api-xks-ca\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:04:10.687753 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:10.687750 4509 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fkn8f\" (UniqueName: \"kubernetes.io/projected/b78d9eb7-681a-4f6e-a61b-790ee1e814fd-kube-api-access-fkn8f\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:04:11.338269 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:11.338239 4509 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c621778-k2mxk" Nov 04 05:04:11.338269 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:11.338261 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c621778-k2mxk" event={"ID":"b78d9eb7-681a-4f6e-a61b-790ee1e814fd","Type":"ContainerDied","Data":"8d26c1d2bad503b3267f09100d8821285fc003aca2b6311da80fa1910dfa9d23"} Nov 04 05:04:11.338704 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:11.338307 4509 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8d26c1d2bad503b3267f09100d8821285fc003aca2b6311da80fa1910dfa9d23" Nov 04 05:04:11.689158 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:11.689049 4509 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-1874d8-78f4d9f8b7-9d2tj" podUID="e5103c9a-0929-422e-8c95-b342181f0669" containerName="payload-pre-processing" containerID="cri-o://0a0e3c88f305b3df3b86b01e30a32fbf0d1eb8586f55658dcdd43e85de8750fe" gracePeriod=25 Nov 04 05:04:12.263427 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:12.263396 4509 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4655c4c9-ad98-4f20-919c-56905e179bad" path="/var/lib/kubelet/pods/4655c4c9-ad98-4f20-919c-56905e179bad/volumes" Nov 04 05:04:14.453573 ip-10-0-133-180 kubenswrapper[4509]: E1104 05:04:14.453538 4509 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmdhl" containerName="extractor" Nov 04 05:04:17.814987 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:17.814957 4509 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-0aae2f-5ff7778f8c-4tbhr" Nov 04 05:04:17.886669 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:17.886588 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1f20ac4-af33-4c51-8ae3-a286a4c9bc2c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1f20ac4-af33-4c51-8ae3-a286a4c9bc2c-plugins-config-volume\") pod \"e1f20ac4-af33-4c51-8ae3-a286a4c9bc2c\" (UID: \"e1f20ac4-af33-4c51-8ae3-a286a4c9bc2c\") " Nov 04 05:04:17.886811 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:17.886725 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1f20ac4-af33-4c51-8ae3-a286a4c9bc2c-kube-api-access-gt9tx\" (UniqueName: \"kubernetes.io/projected/e1f20ac4-af33-4c51-8ae3-a286a4c9bc2c-kube-api-access-gt9tx\") pod \"e1f20ac4-af33-4c51-8ae3-a286a4c9bc2c\" (UID: \"e1f20ac4-af33-4c51-8ae3-a286a4c9bc2c\") " Nov 04 05:04:17.886930 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:17.886909 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1f20ac4-af33-4c51-8ae3-a286a4c9bc2c-plugins-config-volume" pod "e1f20ac4-af33-4c51-8ae3-a286a4c9bc2c" (UID: "e1f20ac4-af33-4c51-8ae3-a286a4c9bc2c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 05:04:17.887180 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:17.887163 4509 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1f20ac4-af33-4c51-8ae3-a286a4c9bc2c-plugins-config-volume\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:04:17.888942 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:17.888919 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1f20ac4-af33-4c51-8ae3-a286a4c9bc2c-kube-api-access-gt9tx" pod "e1f20ac4-af33-4c51-8ae3-a286a4c9bc2c" (UID: "e1f20ac4-af33-4c51-8ae3-a286a4c9bc2c"). InnerVolumeSpecName "kube-api-access-gt9tx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 05:04:17.987995 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:17.987966 4509 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gt9tx\" (UniqueName: \"kubernetes.io/projected/e1f20ac4-af33-4c51-8ae3-a286a4c9bc2c-kube-api-access-gt9tx\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:04:18.381695 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:18.381658 4509 generic.go:356] "Generic (PLEG): container finished" podID="e1f20ac4-af33-4c51-8ae3-a286a4c9bc2c" containerID="08f4b0e09701b0cc67798a27385f32cf7c5aa1f27a06a63ed3afbd73c6c74079" exitCode=0 Nov 04 05:04:18.381891 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:18.381712 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-0aae2f-5ff7778f8c-4tbhr" event={"ID":"e1f20ac4-af33-4c51-8ae3-a286a4c9bc2c","Type":"ContainerDied","Data":"08f4b0e09701b0cc67798a27385f32cf7c5aa1f27a06a63ed3afbd73c6c74079"} Nov 04 05:04:18.382012 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:18.381996 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-0aae2f-5ff7778f8c-4tbhr" event={"ID":"e1f20ac4-af33-4c51-8ae3-a286a4c9bc2c","Type":"ContainerDied","Data":"70edbca3af1fe3e9f644639ec3406d05531856b05e21b3bf66cedccc7671afe1"} Nov 04 05:04:18.382100 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:18.382035 4509 scope.go:122] "RemoveContainer" containerID="08f4b0e09701b0cc67798a27385f32cf7c5aa1f27a06a63ed3afbd73c6c74079" Nov 04 05:04:18.382100 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:18.381749 4509 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-0aae2f-5ff7778f8c-4tbhr" Nov 04 05:04:18.397392 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:18.397364 4509 scope.go:122] "RemoveContainer" containerID="08f4b0e09701b0cc67798a27385f32cf7c5aa1f27a06a63ed3afbd73c6c74079" Nov 04 05:04:18.397745 ip-10-0-133-180 kubenswrapper[4509]: E1104 05:04:18.397716 4509 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08f4b0e09701b0cc67798a27385f32cf7c5aa1f27a06a63ed3afbd73c6c74079\": container with ID starting with 08f4b0e09701b0cc67798a27385f32cf7c5aa1f27a06a63ed3afbd73c6c74079 not found: ID does not exist" containerID="08f4b0e09701b0cc67798a27385f32cf7c5aa1f27a06a63ed3afbd73c6c74079" Nov 04 05:04:18.397830 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:18.397757 4509 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08f4b0e09701b0cc67798a27385f32cf7c5aa1f27a06a63ed3afbd73c6c74079"} err="failed to get container status \"08f4b0e09701b0cc67798a27385f32cf7c5aa1f27a06a63ed3afbd73c6c74079\": rpc error: code = NotFound desc = could not find container \"08f4b0e09701b0cc67798a27385f32cf7c5aa1f27a06a63ed3afbd73c6c74079\": container with ID starting with 08f4b0e09701b0cc67798a27385f32cf7c5aa1f27a06a63ed3afbd73c6c74079 not found: ID does not exist" Nov 04 05:04:18.547686 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:18.547599 4509 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-0aae2f-5ff7778f8c-4tbhr"] Nov 04 05:04:18.553012 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:18.552983 4509 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-0aae2f-5ff7778f8c-4tbhr"] Nov 04 05:04:19.067314 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:19.067276 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-42dnm"] Nov 04 05:04:20.294741 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:20.291748 4509 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1f20ac4-af33-4c51-8ae3-a286a4c9bc2c" path="/var/lib/kubelet/pods/e1f20ac4-af33-4c51-8ae3-a286a4c9bc2c/volumes" Nov 04 05:04:21.125078 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:21.125038 4509 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-1874d8-78f4d9f8b7-9d2tj" podUID="e5103c9a-0929-422e-8c95-b342181f0669" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 05:04:23.099650 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:23.099621 4509 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ed9dfcd-sg78z"] Nov 04 05:04:23.100139 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:23.100126 4509 cpu_manager.go:414] "containerMap: removing container" podUID="b78d9eb7-681a-4f6e-a61b-790ee1e814fd" containerName="revoke-keys" Nov 04 05:04:23.100198 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:23.100140 4509 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b78d9eb7-681a-4f6e-a61b-790ee1e814fd" containerName="revoke-keys" Nov 04 05:04:23.100198 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:23.100154 4509 cpu_manager.go:414] "containerMap: removing container" podUID="e1f20ac4-af33-4c51-8ae3-a286a4c9bc2c" containerName="payload-pre-processing" Nov 04 05:04:23.100198 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:23.100159 4509 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1f20ac4-af33-4c51-8ae3-a286a4c9bc2c" containerName="payload-pre-processing" Nov 04 05:04:23.100198 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:23.100170 4509 cpu_manager.go:414] "containerMap: removing container" podUID="4655c4c9-ad98-4f20-919c-56905e179bad" containerName="main" Nov 04 05:04:23.100198 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:23.100175 4509 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4655c4c9-ad98-4f20-919c-56905e179bad" containerName="main" Nov 04 05:04:23.100381 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:23.100245 4509 memory_manager.go:365] "RemoveStaleState removing state" podUID="4655c4c9-ad98-4f20-919c-56905e179bad" containerName="main" Nov 04 05:04:23.100381 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:23.100259 4509 memory_manager.go:365] "RemoveStaleState removing state" podUID="b78d9eb7-681a-4f6e-a61b-790ee1e814fd" containerName="revoke-keys" Nov 04 05:04:23.100381 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:23.100265 4509 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1f20ac4-af33-4c51-8ae3-a286a4c9bc2c" containerName="payload-pre-processing" Nov 04 05:04:23.104827 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:23.104809 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ed9dfcd-sg78z" Nov 04 05:04:23.162596 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:23.162553 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ed9dfcd-sg78z"] Nov 04 05:04:23.205220 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:23.205179 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e13aa5f1-8d64-4ad9-86c0-1d6d3df9d98c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9ed9dfcd-sg78z\" (UID: \"e13aa5f1-8d64-4ad9-86c0-1d6d3df9d98c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ed9dfcd-sg78z" Nov 04 05:04:23.205428 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:23.205230 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e13aa5f1-8d64-4ad9-86c0-1d6d3df9d98c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9ed9dfcd-sg78z\" (UID: \"e13aa5f1-8d64-4ad9-86c0-1d6d3df9d98c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ed9dfcd-sg78z" Nov 04 05:04:23.205428 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:23.205260 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnl5w\" (UniqueName: \"kubernetes.io/projected/e13aa5f1-8d64-4ad9-86c0-1d6d3df9d98c-kube-api-access-wnl5w\") pod \"maas-api-revoke-sub-9ed9dfcd-sg78z\" (UID: \"e13aa5f1-8d64-4ad9-86c0-1d6d3df9d98c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ed9dfcd-sg78z" Nov 04 05:04:23.208741 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:23.208709 4509 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 05:04:23.208741 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:23.208715 4509 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 05:04:23.208919 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:23.208730 4509 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 05:04:23.208919 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:23.208812 4509 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-krgvj\"" Nov 04 05:04:23.306899 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:23.306864 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e13aa5f1-8d64-4ad9-86c0-1d6d3df9d98c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9ed9dfcd-sg78z\" (UID: \"e13aa5f1-8d64-4ad9-86c0-1d6d3df9d98c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ed9dfcd-sg78z" Nov 04 05:04:23.307079 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:23.306920 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e13aa5f1-8d64-4ad9-86c0-1d6d3df9d98c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9ed9dfcd-sg78z\" (UID: \"e13aa5f1-8d64-4ad9-86c0-1d6d3df9d98c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ed9dfcd-sg78z" Nov 04 05:04:23.307079 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:23.306964 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnl5w\" (UniqueName: \"kubernetes.io/projected/e13aa5f1-8d64-4ad9-86c0-1d6d3df9d98c-kube-api-access-wnl5w\") pod \"maas-api-revoke-sub-9ed9dfcd-sg78z\" (UID: \"e13aa5f1-8d64-4ad9-86c0-1d6d3df9d98c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ed9dfcd-sg78z" Nov 04 05:04:23.307618 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:23.307600 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e13aa5f1-8d64-4ad9-86c0-1d6d3df9d98c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9ed9dfcd-sg78z\" (UID: \"e13aa5f1-8d64-4ad9-86c0-1d6d3df9d98c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ed9dfcd-sg78z" Nov 04 05:04:23.309462 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:23.309444 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e13aa5f1-8d64-4ad9-86c0-1d6d3df9d98c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9ed9dfcd-sg78z\" (UID: \"e13aa5f1-8d64-4ad9-86c0-1d6d3df9d98c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ed9dfcd-sg78z" Nov 04 05:04:23.364369 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:23.364271 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnl5w\" (UniqueName: \"kubernetes.io/projected/e13aa5f1-8d64-4ad9-86c0-1d6d3df9d98c-kube-api-access-wnl5w\") pod \"maas-api-revoke-sub-9ed9dfcd-sg78z\" (UID: \"e13aa5f1-8d64-4ad9-86c0-1d6d3df9d98c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ed9dfcd-sg78z" Nov 04 05:04:23.418139 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:23.418106 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ed9dfcd-sg78z" Nov 04 05:04:23.797954 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:23.797923 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ed9dfcd-sg78z"] Nov 04 05:04:23.799452 ip-10-0-133-180 kubenswrapper[4509]: W1104 05:04:23.799422 4509 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode13aa5f1_8d64_4ad9_86c0_1d6d3df9d98c.slice/crio-0b5691967dce2604e29721bf6ec2dc1b2cd55f1caf2a6a8c253ecb7357647890 WatchSource:0}: Error finding container 0b5691967dce2604e29721bf6ec2dc1b2cd55f1caf2a6a8c253ecb7357647890: Status 404 returned error can't find the container with id 0b5691967dce2604e29721bf6ec2dc1b2cd55f1caf2a6a8c253ecb7357647890 Nov 04 05:04:24.457304 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:24.457277 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ed9dfcd-sg78z" event={"ID":"e13aa5f1-8d64-4ad9-86c0-1d6d3df9d98c","Type":"ContainerStarted","Data":"88bada5ad8c02cea49b39b5215d84e478081a64b40529295d3305ee555a96db3"} Nov 04 05:04:24.457729 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:24.457362 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ed9dfcd-sg78z" event={"ID":"e13aa5f1-8d64-4ad9-86c0-1d6d3df9d98c","Type":"ContainerStarted","Data":"0b5691967dce2604e29721bf6ec2dc1b2cd55f1caf2a6a8c253ecb7357647890"} Nov 04 05:04:24.571780 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:24.571725 4509 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ed9dfcd-sg78z" podStartSLOduration=1.571708472 podStartE2EDuration="1.571708472s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 05:04:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 05:04:24.571266006 +0000 UTC m=+1028.448989532" watchObservedRunningTime="2026-11-04 05:04:24.571708472 +0000 UTC m=+1028.449431992" Nov 04 05:04:25.135100 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:25.135016 4509 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-0aae2f-openshift-default-5f99b6799-x69j7"] Nov 04 05:04:25.135245 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:25.135212 4509 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-0aae2f-openshift-default-5f99b6799-x69j7" podUID="0b878b44-bc26-4e68-94c0-7297413ecd02" containerName="istio-proxy" containerID="cri-o://734f16dad94dc264a31288e392f10f79fd05883259f3782b080dd733d0b9f468" gracePeriod=30 Nov 04 05:04:25.464259 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:25.464213 4509 generic.go:356] "Generic (PLEG): container finished" podID="e13aa5f1-8d64-4ad9-86c0-1d6d3df9d98c" containerID="88bada5ad8c02cea49b39b5215d84e478081a64b40529295d3305ee555a96db3" exitCode=0 Nov 04 05:04:25.464696 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:25.464292 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ed9dfcd-sg78z" event={"ID":"e13aa5f1-8d64-4ad9-86c0-1d6d3df9d98c","Type":"ContainerDied","Data":"88bada5ad8c02cea49b39b5215d84e478081a64b40529295d3305ee555a96db3"} Nov 04 05:04:26.620316 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:26.620285 4509 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ed9dfcd-sg78z" Nov 04 05:04:26.750799 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:26.750715 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e13aa5f1-8d64-4ad9-86c0-1d6d3df9d98c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e13aa5f1-8d64-4ad9-86c0-1d6d3df9d98c-maas-api-xks-ca\") pod \"e13aa5f1-8d64-4ad9-86c0-1d6d3df9d98c\" (UID: \"e13aa5f1-8d64-4ad9-86c0-1d6d3df9d98c\") " Nov 04 05:04:26.750799 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:26.750770 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e13aa5f1-8d64-4ad9-86c0-1d6d3df9d98c-kube-api-access-wnl5w\" (UniqueName: \"kubernetes.io/projected/e13aa5f1-8d64-4ad9-86c0-1d6d3df9d98c-kube-api-access-wnl5w\") pod \"e13aa5f1-8d64-4ad9-86c0-1d6d3df9d98c\" (UID: \"e13aa5f1-8d64-4ad9-86c0-1d6d3df9d98c\") " Nov 04 05:04:26.750799 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:26.750795 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e13aa5f1-8d64-4ad9-86c0-1d6d3df9d98c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e13aa5f1-8d64-4ad9-86c0-1d6d3df9d98c-maas-api-openshift-ca\") pod \"e13aa5f1-8d64-4ad9-86c0-1d6d3df9d98c\" (UID: \"e13aa5f1-8d64-4ad9-86c0-1d6d3df9d98c\") " Nov 04 05:04:26.751233 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:26.751205 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e13aa5f1-8d64-4ad9-86c0-1d6d3df9d98c-maas-api-openshift-ca" pod "e13aa5f1-8d64-4ad9-86c0-1d6d3df9d98c" (UID: "e13aa5f1-8d64-4ad9-86c0-1d6d3df9d98c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 05:04:26.752903 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:26.752872 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e13aa5f1-8d64-4ad9-86c0-1d6d3df9d98c-maas-api-xks-ca" pod "e13aa5f1-8d64-4ad9-86c0-1d6d3df9d98c" (UID: "e13aa5f1-8d64-4ad9-86c0-1d6d3df9d98c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 05:04:26.752996 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:26.752974 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e13aa5f1-8d64-4ad9-86c0-1d6d3df9d98c-kube-api-access-wnl5w" pod "e13aa5f1-8d64-4ad9-86c0-1d6d3df9d98c" (UID: "e13aa5f1-8d64-4ad9-86c0-1d6d3df9d98c"). InnerVolumeSpecName "kube-api-access-wnl5w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 05:04:26.852451 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:26.852394 4509 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e13aa5f1-8d64-4ad9-86c0-1d6d3df9d98c-maas-api-xks-ca\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:04:26.852451 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:26.852452 4509 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wnl5w\" (UniqueName: \"kubernetes.io/projected/e13aa5f1-8d64-4ad9-86c0-1d6d3df9d98c-kube-api-access-wnl5w\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:04:26.852705 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:26.852471 4509 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e13aa5f1-8d64-4ad9-86c0-1d6d3df9d98c-maas-api-openshift-ca\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:04:27.446570 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:27.446490 4509 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-dab70d/premium-w3-dab70d-kserve-575d94569f-9q687"] Nov 04 05:04:27.478028 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:27.478003 4509 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ed9dfcd-sg78z" Nov 04 05:04:27.478199 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:27.478039 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ed9dfcd-sg78z" event={"ID":"e13aa5f1-8d64-4ad9-86c0-1d6d3df9d98c","Type":"ContainerDied","Data":"0b5691967dce2604e29721bf6ec2dc1b2cd55f1caf2a6a8c253ecb7357647890"} Nov 04 05:04:27.478199 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:27.478091 4509 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0b5691967dce2604e29721bf6ec2dc1b2cd55f1caf2a6a8c253ecb7357647890" Nov 04 05:04:30.407509 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:30.407447 4509 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0aae2f-openshift-default-5f99b6799-x69j7" Nov 04 05:04:30.495341 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:30.495291 4509 generic.go:356] "Generic (PLEG): container finished" podID="0b878b44-bc26-4e68-94c0-7297413ecd02" containerID="734f16dad94dc264a31288e392f10f79fd05883259f3782b080dd733d0b9f468" exitCode=0 Nov 04 05:04:30.495511 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:30.495365 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0aae2f-openshift-default-5f99b6799-x69j7" event={"ID":"0b878b44-bc26-4e68-94c0-7297413ecd02","Type":"ContainerDied","Data":"734f16dad94dc264a31288e392f10f79fd05883259f3782b080dd733d0b9f468"} Nov 04 05:04:30.495511 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:30.495378 4509 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0aae2f-openshift-default-5f99b6799-x69j7" Nov 04 05:04:30.495511 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:30.495431 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0aae2f-openshift-default-5f99b6799-x69j7" event={"ID":"0b878b44-bc26-4e68-94c0-7297413ecd02","Type":"ContainerDied","Data":"756907cd8bca0214d67e6b71c66f72db10dd331054eac548520927d74eaad1fd"} Nov 04 05:04:30.495511 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:30.495461 4509 scope.go:122] "RemoveContainer" containerID="734f16dad94dc264a31288e392f10f79fd05883259f3782b080dd733d0b9f468" Nov 04 05:04:30.495983 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:30.495965 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b878b44-bc26-4e68-94c0-7297413ecd02-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b878b44-bc26-4e68-94c0-7297413ecd02-istiod-ca-cert\") pod \"0b878b44-bc26-4e68-94c0-7297413ecd02\" (UID: \"0b878b44-bc26-4e68-94c0-7297413ecd02\") " Nov 04 05:04:30.496059 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:30.495996 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b878b44-bc26-4e68-94c0-7297413ecd02-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b878b44-bc26-4e68-94c0-7297413ecd02-workload-certs\") pod \"0b878b44-bc26-4e68-94c0-7297413ecd02\" (UID: \"0b878b44-bc26-4e68-94c0-7297413ecd02\") " Nov 04 05:04:30.496059 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:30.496028 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b878b44-bc26-4e68-94c0-7297413ecd02-istio-token\" (UniqueName: \"kubernetes.io/projected/0b878b44-bc26-4e68-94c0-7297413ecd02-istio-token\") pod \"0b878b44-bc26-4e68-94c0-7297413ecd02\" (UID: \"0b878b44-bc26-4e68-94c0-7297413ecd02\") " Nov 04 05:04:30.496059 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:30.496051 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b878b44-bc26-4e68-94c0-7297413ecd02-kube-api-access-mlfcz\" (UniqueName: \"kubernetes.io/projected/0b878b44-bc26-4e68-94c0-7297413ecd02-kube-api-access-mlfcz\") pod \"0b878b44-bc26-4e68-94c0-7297413ecd02\" (UID: \"0b878b44-bc26-4e68-94c0-7297413ecd02\") " Nov 04 05:04:30.496203 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:30.496085 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b878b44-bc26-4e68-94c0-7297413ecd02-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b878b44-bc26-4e68-94c0-7297413ecd02-credential-socket\") pod \"0b878b44-bc26-4e68-94c0-7297413ecd02\" (UID: \"0b878b44-bc26-4e68-94c0-7297413ecd02\") " Nov 04 05:04:30.496203 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:30.496136 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0b878b44-bc26-4e68-94c0-7297413ecd02-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b878b44-bc26-4e68-94c0-7297413ecd02-istio-podinfo\") pod \"0b878b44-bc26-4e68-94c0-7297413ecd02\" (UID: \"0b878b44-bc26-4e68-94c0-7297413ecd02\") " Nov 04 05:04:30.496298 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:30.496220 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b878b44-bc26-4e68-94c0-7297413ecd02-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b878b44-bc26-4e68-94c0-7297413ecd02-istio-envoy\") pod \"0b878b44-bc26-4e68-94c0-7297413ecd02\" (UID: \"0b878b44-bc26-4e68-94c0-7297413ecd02\") " Nov 04 05:04:30.496298 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:30.496261 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b878b44-bc26-4e68-94c0-7297413ecd02-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b878b44-bc26-4e68-94c0-7297413ecd02-workload-socket\") pod \"0b878b44-bc26-4e68-94c0-7297413ecd02\" (UID: \"0b878b44-bc26-4e68-94c0-7297413ecd02\") " Nov 04 05:04:30.496420 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:30.496380 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b878b44-bc26-4e68-94c0-7297413ecd02-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b878b44-bc26-4e68-94c0-7297413ecd02-istio-data\") pod \"0b878b44-bc26-4e68-94c0-7297413ecd02\" (UID: \"0b878b44-bc26-4e68-94c0-7297413ecd02\") " Nov 04 05:04:30.496420 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:30.496373 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b878b44-bc26-4e68-94c0-7297413ecd02-workload-certs" pod "0b878b44-bc26-4e68-94c0-7297413ecd02" (UID: "0b878b44-bc26-4e68-94c0-7297413ecd02"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:04:30.496555 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:30.496499 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b878b44-bc26-4e68-94c0-7297413ecd02-istiod-ca-cert" pod "0b878b44-bc26-4e68-94c0-7297413ecd02" (UID: "0b878b44-bc26-4e68-94c0-7297413ecd02"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 05:04:30.496555 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:30.496465 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b878b44-bc26-4e68-94c0-7297413ecd02-credential-socket" pod "0b878b44-bc26-4e68-94c0-7297413ecd02" (UID: "0b878b44-bc26-4e68-94c0-7297413ecd02"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:04:30.496646 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:30.496623 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b878b44-bc26-4e68-94c0-7297413ecd02-workload-socket" pod "0b878b44-bc26-4e68-94c0-7297413ecd02" (UID: "0b878b44-bc26-4e68-94c0-7297413ecd02"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:04:30.496846 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:30.496817 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b878b44-bc26-4e68-94c0-7297413ecd02-istio-data" pod "0b878b44-bc26-4e68-94c0-7297413ecd02" (UID: "0b878b44-bc26-4e68-94c0-7297413ecd02"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:04:30.497173 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:30.497150 4509 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b878b44-bc26-4e68-94c0-7297413ecd02-istio-data\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:04:30.497259 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:30.497175 4509 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b878b44-bc26-4e68-94c0-7297413ecd02-istiod-ca-cert\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:04:30.497259 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:30.497191 4509 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b878b44-bc26-4e68-94c0-7297413ecd02-workload-certs\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:04:30.497259 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:30.497206 4509 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b878b44-bc26-4e68-94c0-7297413ecd02-credential-socket\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:04:30.497259 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:30.497220 4509 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b878b44-bc26-4e68-94c0-7297413ecd02-workload-socket\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:04:30.498752 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:30.498723 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b878b44-bc26-4e68-94c0-7297413ecd02-istio-token" pod "0b878b44-bc26-4e68-94c0-7297413ecd02" (UID: "0b878b44-bc26-4e68-94c0-7297413ecd02"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 05:04:30.499110 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:30.499089 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b878b44-bc26-4e68-94c0-7297413ecd02-kube-api-access-mlfcz" pod "0b878b44-bc26-4e68-94c0-7297413ecd02" (UID: "0b878b44-bc26-4e68-94c0-7297413ecd02"). InnerVolumeSpecName "kube-api-access-mlfcz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 05:04:30.499172 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:30.499130 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b878b44-bc26-4e68-94c0-7297413ecd02-istio-envoy" pod "0b878b44-bc26-4e68-94c0-7297413ecd02" (UID: "0b878b44-bc26-4e68-94c0-7297413ecd02"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:04:30.500661 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:30.500643 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0b878b44-bc26-4e68-94c0-7297413ecd02-istio-podinfo" pod "0b878b44-bc26-4e68-94c0-7297413ecd02" (UID: "0b878b44-bc26-4e68-94c0-7297413ecd02"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 05:04:30.522626 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:30.522599 4509 scope.go:122] "RemoveContainer" containerID="734f16dad94dc264a31288e392f10f79fd05883259f3782b080dd733d0b9f468" Nov 04 05:04:30.522980 ip-10-0-133-180 kubenswrapper[4509]: E1104 05:04:30.522960 4509 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"734f16dad94dc264a31288e392f10f79fd05883259f3782b080dd733d0b9f468\": container with ID starting with 734f16dad94dc264a31288e392f10f79fd05883259f3782b080dd733d0b9f468 not found: ID does not exist" containerID="734f16dad94dc264a31288e392f10f79fd05883259f3782b080dd733d0b9f468" Nov 04 05:04:30.523070 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:30.522988 4509 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"734f16dad94dc264a31288e392f10f79fd05883259f3782b080dd733d0b9f468"} err="failed to get container status \"734f16dad94dc264a31288e392f10f79fd05883259f3782b080dd733d0b9f468\": rpc error: code = NotFound desc = could not find container \"734f16dad94dc264a31288e392f10f79fd05883259f3782b080dd733d0b9f468\": container with ID starting with 734f16dad94dc264a31288e392f10f79fd05883259f3782b080dd733d0b9f468 not found: ID does not exist" Nov 04 05:04:30.598172 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:30.598140 4509 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b878b44-bc26-4e68-94c0-7297413ecd02-istio-token\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:04:30.598172 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:30.598169 4509 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mlfcz\" (UniqueName: \"kubernetes.io/projected/0b878b44-bc26-4e68-94c0-7297413ecd02-kube-api-access-mlfcz\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:04:30.598172 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:30.598179 4509 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b878b44-bc26-4e68-94c0-7297413ecd02-istio-podinfo\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:04:30.598438 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:30.598187 4509 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b878b44-bc26-4e68-94c0-7297413ecd02-istio-envoy\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:04:30.916399 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:30.916357 4509 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-0aae2f-openshift-default-5f99b6799-x69j7"] Nov 04 05:04:30.919979 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:30.919948 4509 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-0aae2f-openshift-default-5f99b6799-x69j7"] Nov 04 05:04:31.124969 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:31.124883 4509 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-1874d8-78f4d9f8b7-9d2tj" podUID="e5103c9a-0929-422e-8c95-b342181f0669" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 05:04:32.002576 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:32.002549 4509 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-1874d8-78f4d9f8b7-9d2tj" Nov 04 05:04:32.019881 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:32.019860 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5103c9a-0929-422e-8c95-b342181f0669-kube-api-access-lnkns\" (UniqueName: \"kubernetes.io/projected/e5103c9a-0929-422e-8c95-b342181f0669-kube-api-access-lnkns\") pod \"e5103c9a-0929-422e-8c95-b342181f0669\" (UID: \"e5103c9a-0929-422e-8c95-b342181f0669\") " Nov 04 05:04:32.019992 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:32.019927 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5103c9a-0929-422e-8c95-b342181f0669-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5103c9a-0929-422e-8c95-b342181f0669-plugins-config-volume\") pod \"e5103c9a-0929-422e-8c95-b342181f0669\" (UID: \"e5103c9a-0929-422e-8c95-b342181f0669\") " Nov 04 05:04:32.020273 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:32.020251 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5103c9a-0929-422e-8c95-b342181f0669-plugins-config-volume" pod "e5103c9a-0929-422e-8c95-b342181f0669" (UID: "e5103c9a-0929-422e-8c95-b342181f0669"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 05:04:32.022139 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:32.022108 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5103c9a-0929-422e-8c95-b342181f0669-kube-api-access-lnkns" pod "e5103c9a-0929-422e-8c95-b342181f0669" (UID: "e5103c9a-0929-422e-8c95-b342181f0669"). InnerVolumeSpecName "kube-api-access-lnkns". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 05:04:32.121535 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:32.121503 4509 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lnkns\" (UniqueName: \"kubernetes.io/projected/e5103c9a-0929-422e-8c95-b342181f0669-kube-api-access-lnkns\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:04:32.121535 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:32.121534 4509 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5103c9a-0929-422e-8c95-b342181f0669-plugins-config-volume\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:04:32.262921 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:32.262839 4509 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b878b44-bc26-4e68-94c0-7297413ecd02" path="/var/lib/kubelet/pods/0b878b44-bc26-4e68-94c0-7297413ecd02/volumes" Nov 04 05:04:32.509234 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:32.509198 4509 generic.go:356] "Generic (PLEG): container finished" podID="e5103c9a-0929-422e-8c95-b342181f0669" containerID="0a0e3c88f305b3df3b86b01e30a32fbf0d1eb8586f55658dcdd43e85de8750fe" exitCode=0 Nov 04 05:04:32.509432 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:32.509249 4509 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-1874d8-78f4d9f8b7-9d2tj" Nov 04 05:04:32.509432 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:32.509278 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-1874d8-78f4d9f8b7-9d2tj" event={"ID":"e5103c9a-0929-422e-8c95-b342181f0669","Type":"ContainerDied","Data":"0a0e3c88f305b3df3b86b01e30a32fbf0d1eb8586f55658dcdd43e85de8750fe"} Nov 04 05:04:32.509432 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:32.509343 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-1874d8-78f4d9f8b7-9d2tj" event={"ID":"e5103c9a-0929-422e-8c95-b342181f0669","Type":"ContainerDied","Data":"1b4d6fd92b32bde69c73fb0eaaa01cc9edafcb5ca839a77dbeba64bc5748a516"} Nov 04 05:04:32.509432 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:32.509370 4509 scope.go:122] "RemoveContainer" containerID="0a0e3c88f305b3df3b86b01e30a32fbf0d1eb8586f55658dcdd43e85de8750fe" Nov 04 05:04:32.518589 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:32.518573 4509 scope.go:122] "RemoveContainer" containerID="0a0e3c88f305b3df3b86b01e30a32fbf0d1eb8586f55658dcdd43e85de8750fe" Nov 04 05:04:32.518896 ip-10-0-133-180 kubenswrapper[4509]: E1104 05:04:32.518873 4509 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a0e3c88f305b3df3b86b01e30a32fbf0d1eb8586f55658dcdd43e85de8750fe\": container with ID starting with 0a0e3c88f305b3df3b86b01e30a32fbf0d1eb8586f55658dcdd43e85de8750fe not found: ID does not exist" containerID="0a0e3c88f305b3df3b86b01e30a32fbf0d1eb8586f55658dcdd43e85de8750fe" Nov 04 05:04:32.518988 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:32.518900 4509 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a0e3c88f305b3df3b86b01e30a32fbf0d1eb8586f55658dcdd43e85de8750fe"} err="failed to get container status \"0a0e3c88f305b3df3b86b01e30a32fbf0d1eb8586f55658dcdd43e85de8750fe\": rpc error: code = NotFound desc = could not find container \"0a0e3c88f305b3df3b86b01e30a32fbf0d1eb8586f55658dcdd43e85de8750fe\": container with ID starting with 0a0e3c88f305b3df3b86b01e30a32fbf0d1eb8586f55658dcdd43e85de8750fe not found: ID does not exist" Nov 04 05:04:32.673761 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:32.673716 4509 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-1874d8-78f4d9f8b7-9d2tj"] Nov 04 05:04:32.723545 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:32.723510 4509 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-1874d8-78f4d9f8b7-9d2tj"] Nov 04 05:04:33.147786 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:33.147754 4509 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-d8rj2"] Nov 04 05:04:33.148230 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:33.147972 4509 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-d8rj2" podUID="5a89237f-e122-4578-98bc-8a0658f3998d" containerName="istio-proxy" containerID="cri-o://720158b6fbc8846f05a48c54ae0ef358db2d53f40c79e97a3e1b3f708fa35935" gracePeriod=30 Nov 04 05:04:34.261577 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:34.261545 4509 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5103c9a-0929-422e-8c95-b342181f0669" path="/var/lib/kubelet/pods/e5103c9a-0929-422e-8c95-b342181f0669/volumes" Nov 04 05:04:37.059200 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:37.059160 4509 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-tc7tm"] Nov 04 05:04:37.059742 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:37.059726 4509 cpu_manager.go:414] "containerMap: removing container" podUID="0b878b44-bc26-4e68-94c0-7297413ecd02" containerName="istio-proxy" Nov 04 05:04:37.059787 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:37.059744 4509 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b878b44-bc26-4e68-94c0-7297413ecd02" containerName="istio-proxy" Nov 04 05:04:37.059787 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:37.059780 4509 cpu_manager.go:414] "containerMap: removing container" podUID="e13aa5f1-8d64-4ad9-86c0-1d6d3df9d98c" containerName="revoke-keys" Nov 04 05:04:37.059847 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:37.059787 4509 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e13aa5f1-8d64-4ad9-86c0-1d6d3df9d98c" containerName="revoke-keys" Nov 04 05:04:37.059847 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:37.059803 4509 cpu_manager.go:414] "containerMap: removing container" podUID="e5103c9a-0929-422e-8c95-b342181f0669" containerName="payload-pre-processing" Nov 04 05:04:37.059847 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:37.059809 4509 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5103c9a-0929-422e-8c95-b342181f0669" containerName="payload-pre-processing" Nov 04 05:04:37.059938 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:37.059889 4509 memory_manager.go:365] "RemoveStaleState removing state" podUID="e13aa5f1-8d64-4ad9-86c0-1d6d3df9d98c" containerName="revoke-keys" Nov 04 05:04:37.059938 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:37.059899 4509 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5103c9a-0929-422e-8c95-b342181f0669" containerName="payload-pre-processing" Nov 04 05:04:37.059938 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:37.059906 4509 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b878b44-bc26-4e68-94c0-7297413ecd02" containerName="istio-proxy" Nov 04 05:04:37.064350 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:37.064306 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-tc7tm" Nov 04 05:04:37.082935 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:37.082899 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/098b1d63-ed94-4884-a6de-428f799c3e31-istiod-ca-cert\") pod \"e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-tc7tm\" (UID: \"098b1d63-ed94-4884-a6de-428f799c3e31\") " pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-tc7tm" Nov 04 05:04:37.082935 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:37.082939 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/098b1d63-ed94-4884-a6de-428f799c3e31-istio-token\") pod \"e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-tc7tm\" (UID: \"098b1d63-ed94-4884-a6de-428f799c3e31\") " pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-tc7tm" Nov 04 05:04:37.083173 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:37.082961 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/098b1d63-ed94-4884-a6de-428f799c3e31-istio-data\") pod \"e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-tc7tm\" (UID: \"098b1d63-ed94-4884-a6de-428f799c3e31\") " pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-tc7tm" Nov 04 05:04:37.083173 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:37.083080 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/098b1d63-ed94-4884-a6de-428f799c3e31-workload-certs\") pod \"e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-tc7tm\" (UID: \"098b1d63-ed94-4884-a6de-428f799c3e31\") " pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-tc7tm" Nov 04 05:04:37.083173 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:37.083129 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/098b1d63-ed94-4884-a6de-428f799c3e31-istio-podinfo\") pod \"e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-tc7tm\" (UID: \"098b1d63-ed94-4884-a6de-428f799c3e31\") " pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-tc7tm" Nov 04 05:04:37.083273 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:37.083211 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/098b1d63-ed94-4884-a6de-428f799c3e31-istio-envoy\") pod \"e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-tc7tm\" (UID: \"098b1d63-ed94-4884-a6de-428f799c3e31\") " pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-tc7tm" Nov 04 05:04:37.083273 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:37.083242 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgtct\" (UniqueName: \"kubernetes.io/projected/098b1d63-ed94-4884-a6de-428f799c3e31-kube-api-access-fgtct\") pod \"e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-tc7tm\" (UID: \"098b1d63-ed94-4884-a6de-428f799c3e31\") " pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-tc7tm" Nov 04 05:04:37.083376 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:37.083275 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/098b1d63-ed94-4884-a6de-428f799c3e31-workload-socket\") pod \"e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-tc7tm\" (UID: \"098b1d63-ed94-4884-a6de-428f799c3e31\") " pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-tc7tm" Nov 04 05:04:37.083376 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:37.083359 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/098b1d63-ed94-4884-a6de-428f799c3e31-credential-socket\") pod \"e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-tc7tm\" (UID: \"098b1d63-ed94-4884-a6de-428f799c3e31\") " pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-tc7tm" Nov 04 05:04:37.162536 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:37.162491 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-tc7tm"] Nov 04 05:04:37.183928 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:37.183892 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/098b1d63-ed94-4884-a6de-428f799c3e31-istiod-ca-cert\") pod \"e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-tc7tm\" (UID: \"098b1d63-ed94-4884-a6de-428f799c3e31\") " pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-tc7tm" Nov 04 05:04:37.183928 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:37.183939 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/098b1d63-ed94-4884-a6de-428f799c3e31-istio-token\") pod \"e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-tc7tm\" (UID: \"098b1d63-ed94-4884-a6de-428f799c3e31\") " pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-tc7tm" Nov 04 05:04:37.184197 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:37.183960 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/098b1d63-ed94-4884-a6de-428f799c3e31-istio-data\") pod \"e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-tc7tm\" (UID: \"098b1d63-ed94-4884-a6de-428f799c3e31\") " pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-tc7tm" Nov 04 05:04:37.184197 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:37.184087 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/098b1d63-ed94-4884-a6de-428f799c3e31-workload-certs\") pod \"e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-tc7tm\" (UID: \"098b1d63-ed94-4884-a6de-428f799c3e31\") " pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-tc7tm" Nov 04 05:04:37.184197 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:37.184168 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/098b1d63-ed94-4884-a6de-428f799c3e31-istio-podinfo\") pod \"e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-tc7tm\" (UID: \"098b1d63-ed94-4884-a6de-428f799c3e31\") " pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-tc7tm" Nov 04 05:04:37.184404 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:37.184302 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/098b1d63-ed94-4884-a6de-428f799c3e31-istio-envoy\") pod \"e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-tc7tm\" (UID: \"098b1d63-ed94-4884-a6de-428f799c3e31\") " pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-tc7tm" Nov 04 05:04:37.184404 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:37.184396 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgtct\" (UniqueName: \"kubernetes.io/projected/098b1d63-ed94-4884-a6de-428f799c3e31-kube-api-access-fgtct\") pod \"e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-tc7tm\" (UID: \"098b1d63-ed94-4884-a6de-428f799c3e31\") " pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-tc7tm" Nov 04 05:04:37.184544 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:37.184429 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/098b1d63-ed94-4884-a6de-428f799c3e31-workload-certs\") pod \"e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-tc7tm\" (UID: \"098b1d63-ed94-4884-a6de-428f799c3e31\") " pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-tc7tm" Nov 04 05:04:37.184696 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:37.184671 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/098b1d63-ed94-4884-a6de-428f799c3e31-istio-data\") pod \"e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-tc7tm\" (UID: \"098b1d63-ed94-4884-a6de-428f799c3e31\") " pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-tc7tm" Nov 04 05:04:37.184785 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:37.184436 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/098b1d63-ed94-4884-a6de-428f799c3e31-workload-socket\") pod \"e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-tc7tm\" (UID: \"098b1d63-ed94-4884-a6de-428f799c3e31\") " pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-tc7tm" Nov 04 05:04:37.184785 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:37.184736 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/098b1d63-ed94-4884-a6de-428f799c3e31-workload-socket\") pod \"e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-tc7tm\" (UID: \"098b1d63-ed94-4884-a6de-428f799c3e31\") " pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-tc7tm" Nov 04 05:04:37.184785 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:37.184765 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/098b1d63-ed94-4884-a6de-428f799c3e31-credential-socket\") pod \"e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-tc7tm\" (UID: \"098b1d63-ed94-4884-a6de-428f799c3e31\") " pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-tc7tm" Nov 04 05:04:37.184954 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:37.184816 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/098b1d63-ed94-4884-a6de-428f799c3e31-istiod-ca-cert\") pod \"e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-tc7tm\" (UID: \"098b1d63-ed94-4884-a6de-428f799c3e31\") " pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-tc7tm" Nov 04 05:04:37.185178 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:37.185155 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/098b1d63-ed94-4884-a6de-428f799c3e31-credential-socket\") pod \"e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-tc7tm\" (UID: \"098b1d63-ed94-4884-a6de-428f799c3e31\") " pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-tc7tm" Nov 04 05:04:37.186993 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:37.186970 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/098b1d63-ed94-4884-a6de-428f799c3e31-istio-envoy\") pod \"e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-tc7tm\" (UID: \"098b1d63-ed94-4884-a6de-428f799c3e31\") " pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-tc7tm" Nov 04 05:04:37.187467 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:37.187444 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/098b1d63-ed94-4884-a6de-428f799c3e31-istio-podinfo\") pod \"e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-tc7tm\" (UID: \"098b1d63-ed94-4884-a6de-428f799c3e31\") " pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-tc7tm" Nov 04 05:04:37.241180 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:37.241144 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/098b1d63-ed94-4884-a6de-428f799c3e31-istio-token\") pod \"e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-tc7tm\" (UID: \"098b1d63-ed94-4884-a6de-428f799c3e31\") " pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-tc7tm" Nov 04 05:04:37.241382 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:37.241294 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgtct\" (UniqueName: \"kubernetes.io/projected/098b1d63-ed94-4884-a6de-428f799c3e31-kube-api-access-fgtct\") pod \"e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-tc7tm\" (UID: \"098b1d63-ed94-4884-a6de-428f799c3e31\") " pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-tc7tm" Nov 04 05:04:37.378103 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:37.378005 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-tc7tm" Nov 04 05:04:37.812883 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:37.812850 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-tc7tm"] Nov 04 05:04:37.814428 ip-10-0-133-180 kubenswrapper[4509]: W1104 05:04:37.814403 4509 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod098b1d63_ed94_4884_a6de_428f799c3e31.slice/crio-3da7182488504f0fb6536356966c14ce8133a3ad8fc01973880047a4a7c1db49 WatchSource:0}: Error finding container 3da7182488504f0fb6536356966c14ce8133a3ad8fc01973880047a4a7c1db49: Status 404 returned error can't find the container with id 3da7182488504f0fb6536356966c14ce8133a3ad8fc01973880047a4a7c1db49 Nov 04 05:04:37.816780 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:37.816748 4509 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 04 05:04:37.816854 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:37.816820 4509 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 04 05:04:37.816854 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:37.816848 4509 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 04 05:04:38.490597 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:38.490564 4509 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-d8rj2" Nov 04 05:04:38.498871 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:38.498834 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a89237f-e122-4578-98bc-8a0658f3998d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a89237f-e122-4578-98bc-8a0658f3998d-istio-data\") pod \"5a89237f-e122-4578-98bc-8a0658f3998d\" (UID: \"5a89237f-e122-4578-98bc-8a0658f3998d\") " Nov 04 05:04:38.499001 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:38.498926 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5a89237f-e122-4578-98bc-8a0658f3998d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a89237f-e122-4578-98bc-8a0658f3998d-istio-podinfo\") pod \"5a89237f-e122-4578-98bc-8a0658f3998d\" (UID: \"5a89237f-e122-4578-98bc-8a0658f3998d\") " Nov 04 05:04:38.499059 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:38.499040 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a89237f-e122-4578-98bc-8a0658f3998d-istio-token\" (UniqueName: \"kubernetes.io/projected/5a89237f-e122-4578-98bc-8a0658f3998d-istio-token\") pod \"5a89237f-e122-4578-98bc-8a0658f3998d\" (UID: \"5a89237f-e122-4578-98bc-8a0658f3998d\") " Nov 04 05:04:38.499100 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:38.499078 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a89237f-e122-4578-98bc-8a0658f3998d-kube-api-access-jpc79\" (UniqueName: \"kubernetes.io/projected/5a89237f-e122-4578-98bc-8a0658f3998d-kube-api-access-jpc79\") pod \"5a89237f-e122-4578-98bc-8a0658f3998d\" (UID: \"5a89237f-e122-4578-98bc-8a0658f3998d\") " Nov 04 05:04:38.499100 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:38.499080 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a89237f-e122-4578-98bc-8a0658f3998d-istio-data" pod "5a89237f-e122-4578-98bc-8a0658f3998d" (UID: "5a89237f-e122-4578-98bc-8a0658f3998d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:04:38.499164 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:38.499108 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a89237f-e122-4578-98bc-8a0658f3998d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a89237f-e122-4578-98bc-8a0658f3998d-istio-envoy\") pod \"5a89237f-e122-4578-98bc-8a0658f3998d\" (UID: \"5a89237f-e122-4578-98bc-8a0658f3998d\") " Nov 04 05:04:38.499164 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:38.499142 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a89237f-e122-4578-98bc-8a0658f3998d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a89237f-e122-4578-98bc-8a0658f3998d-istiod-ca-cert\") pod \"5a89237f-e122-4578-98bc-8a0658f3998d\" (UID: \"5a89237f-e122-4578-98bc-8a0658f3998d\") " Nov 04 05:04:38.499265 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:38.499241 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a89237f-e122-4578-98bc-8a0658f3998d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a89237f-e122-4578-98bc-8a0658f3998d-workload-certs\") pod \"5a89237f-e122-4578-98bc-8a0658f3998d\" (UID: \"5a89237f-e122-4578-98bc-8a0658f3998d\") " Nov 04 05:04:38.499418 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:38.499307 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a89237f-e122-4578-98bc-8a0658f3998d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a89237f-e122-4578-98bc-8a0658f3998d-credential-socket\") pod \"5a89237f-e122-4578-98bc-8a0658f3998d\" (UID: \"5a89237f-e122-4578-98bc-8a0658f3998d\") " Nov 04 05:04:38.499418 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:38.499394 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a89237f-e122-4578-98bc-8a0658f3998d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a89237f-e122-4578-98bc-8a0658f3998d-workload-socket\") pod \"5a89237f-e122-4578-98bc-8a0658f3998d\" (UID: \"5a89237f-e122-4578-98bc-8a0658f3998d\") " Nov 04 05:04:38.499802 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:38.499619 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a89237f-e122-4578-98bc-8a0658f3998d-credential-socket" pod "5a89237f-e122-4578-98bc-8a0658f3998d" (UID: "5a89237f-e122-4578-98bc-8a0658f3998d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:04:38.499802 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:38.499754 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a89237f-e122-4578-98bc-8a0658f3998d-workload-certs" pod "5a89237f-e122-4578-98bc-8a0658f3998d" (UID: "5a89237f-e122-4578-98bc-8a0658f3998d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:04:38.499802 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:38.499789 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a89237f-e122-4578-98bc-8a0658f3998d-istiod-ca-cert" pod "5a89237f-e122-4578-98bc-8a0658f3998d" (UID: "5a89237f-e122-4578-98bc-8a0658f3998d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 05:04:38.499987 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:38.499909 4509 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a89237f-e122-4578-98bc-8a0658f3998d-workload-certs\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:04:38.499987 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:38.499933 4509 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a89237f-e122-4578-98bc-8a0658f3998d-credential-socket\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:04:38.499987 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:38.499947 4509 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a89237f-e122-4578-98bc-8a0658f3998d-istio-data\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:04:38.499987 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:38.499961 4509 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a89237f-e122-4578-98bc-8a0658f3998d-istiod-ca-cert\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:04:38.500104 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:38.500008 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a89237f-e122-4578-98bc-8a0658f3998d-workload-socket" pod "5a89237f-e122-4578-98bc-8a0658f3998d" (UID: "5a89237f-e122-4578-98bc-8a0658f3998d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:04:38.501265 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:38.501237 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5a89237f-e122-4578-98bc-8a0658f3998d-istio-podinfo" pod "5a89237f-e122-4578-98bc-8a0658f3998d" (UID: "5a89237f-e122-4578-98bc-8a0658f3998d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 05:04:38.501395 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:38.501374 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a89237f-e122-4578-98bc-8a0658f3998d-istio-envoy" pod "5a89237f-e122-4578-98bc-8a0658f3998d" (UID: "5a89237f-e122-4578-98bc-8a0658f3998d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:04:38.501879 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:38.501859 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a89237f-e122-4578-98bc-8a0658f3998d-kube-api-access-jpc79" pod "5a89237f-e122-4578-98bc-8a0658f3998d" (UID: "5a89237f-e122-4578-98bc-8a0658f3998d"). InnerVolumeSpecName "kube-api-access-jpc79". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 05:04:38.501945 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:38.501894 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a89237f-e122-4578-98bc-8a0658f3998d-istio-token" pod "5a89237f-e122-4578-98bc-8a0658f3998d" (UID: "5a89237f-e122-4578-98bc-8a0658f3998d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 05:04:38.554588 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:38.554553 4509 generic.go:356] "Generic (PLEG): container finished" podID="5a89237f-e122-4578-98bc-8a0658f3998d" containerID="720158b6fbc8846f05a48c54ae0ef358db2d53f40c79e97a3e1b3f708fa35935" exitCode=0 Nov 04 05:04:38.554793 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:38.554596 4509 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-d8rj2" Nov 04 05:04:38.554793 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:38.554633 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-d8rj2" event={"ID":"5a89237f-e122-4578-98bc-8a0658f3998d","Type":"ContainerDied","Data":"720158b6fbc8846f05a48c54ae0ef358db2d53f40c79e97a3e1b3f708fa35935"} Nov 04 05:04:38.554793 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:38.554694 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-d8rj2" event={"ID":"5a89237f-e122-4578-98bc-8a0658f3998d","Type":"ContainerDied","Data":"e760ab8c22a9e31a471c87c86da89d1370462695406b530f8e4dcb7f18b5fd7a"} Nov 04 05:04:38.554793 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:38.554723 4509 scope.go:122] "RemoveContainer" containerID="720158b6fbc8846f05a48c54ae0ef358db2d53f40c79e97a3e1b3f708fa35935" Nov 04 05:04:38.556494 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:38.556452 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-tc7tm" event={"ID":"098b1d63-ed94-4884-a6de-428f799c3e31","Type":"ContainerStarted","Data":"dc778622824d6f0abd8df357ba9a240d356cafd95b0444f85a4785c2abdc5a1d"} Nov 04 05:04:38.556599 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:38.556519 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-tc7tm" event={"ID":"098b1d63-ed94-4884-a6de-428f799c3e31","Type":"ContainerStarted","Data":"3da7182488504f0fb6536356966c14ce8133a3ad8fc01973880047a4a7c1db49"} Nov 04 05:04:38.591557 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:38.591535 4509 scope.go:122] "RemoveContainer" containerID="720158b6fbc8846f05a48c54ae0ef358db2d53f40c79e97a3e1b3f708fa35935" Nov 04 05:04:38.591928 ip-10-0-133-180 kubenswrapper[4509]: E1104 05:04:38.591906 4509 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"720158b6fbc8846f05a48c54ae0ef358db2d53f40c79e97a3e1b3f708fa35935\": container with ID starting with 720158b6fbc8846f05a48c54ae0ef358db2d53f40c79e97a3e1b3f708fa35935 not found: ID does not exist" containerID="720158b6fbc8846f05a48c54ae0ef358db2d53f40c79e97a3e1b3f708fa35935" Nov 04 05:04:38.592015 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:38.591941 4509 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"720158b6fbc8846f05a48c54ae0ef358db2d53f40c79e97a3e1b3f708fa35935"} err="failed to get container status \"720158b6fbc8846f05a48c54ae0ef358db2d53f40c79e97a3e1b3f708fa35935\": rpc error: code = NotFound desc = could not find container \"720158b6fbc8846f05a48c54ae0ef358db2d53f40c79e97a3e1b3f708fa35935\": container with ID starting with 720158b6fbc8846f05a48c54ae0ef358db2d53f40c79e97a3e1b3f708fa35935 not found: ID does not exist" Nov 04 05:04:38.601194 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:38.601151 4509 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a89237f-e122-4578-98bc-8a0658f3998d-istio-podinfo\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:04:38.601194 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:38.601190 4509 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a89237f-e122-4578-98bc-8a0658f3998d-istio-token\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:04:38.601481 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:38.601207 4509 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jpc79\" (UniqueName: \"kubernetes.io/projected/5a89237f-e122-4578-98bc-8a0658f3998d-kube-api-access-jpc79\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:04:38.601481 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:38.601223 4509 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a89237f-e122-4578-98bc-8a0658f3998d-istio-envoy\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:04:38.601481 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:38.601237 4509 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a89237f-e122-4578-98bc-8a0658f3998d-workload-socket\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:04:38.744723 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:38.744649 4509 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-tc7tm" podStartSLOduration=1.74463175 podStartE2EDuration="1.74463175s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 05:04:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 05:04:38.741950994 +0000 UTC m=+1042.619674516" watchObservedRunningTime="2026-11-04 05:04:38.74463175 +0000 UTC m=+1042.622355275" Nov 04 05:04:38.887443 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:38.882383 4509 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-d8rj2"] Nov 04 05:04:38.887443 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:38.883391 4509 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-d8rj2"] Nov 04 05:04:39.387348 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:39.385280 4509 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-tc7tm" Nov 04 05:04:39.387348 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:39.386453 4509 patch_prober.go:29] interesting pod/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-tc7tm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 04 05:04:39.387348 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:39.386505 4509 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-tc7tm" podUID="098b1d63-ed94-4884-a6de-428f799c3e31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 04 05:04:40.276369 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:40.268338 4509 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a89237f-e122-4578-98bc-8a0658f3998d" path="/var/lib/kubelet/pods/5a89237f-e122-4578-98bc-8a0658f3998d/volumes" Nov 04 05:04:40.320566 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:40.320530 4509 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-dab70d-6ddd49994d-ggggw"] Nov 04 05:04:40.383376 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:40.380309 4509 patch_prober.go:29] interesting pod/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-tc7tm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 04 05:04:40.383376 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:40.380411 4509 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-tc7tm" podUID="098b1d63-ed94-4884-a6de-428f799c3e31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 04 05:04:40.385560 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:40.385447 4509 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-dab70d-695b6bf8dc-vlnmz"] Nov 04 05:04:40.586401 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:40.582967 4509 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3c065f20-4452q"] Nov 04 05:04:40.586401 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:40.583802 4509 cpu_manager.go:414] "containerMap: removing container" podUID="5a89237f-e122-4578-98bc-8a0658f3998d" containerName="istio-proxy" Nov 04 05:04:40.586401 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:40.583821 4509 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a89237f-e122-4578-98bc-8a0658f3998d" containerName="istio-proxy" Nov 04 05:04:40.586401 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:40.583944 4509 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a89237f-e122-4578-98bc-8a0658f3998d" containerName="istio-proxy" Nov 04 05:04:40.592528 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:40.592498 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c065f20-4452q" Nov 04 05:04:40.621676 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:40.621636 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e5e0f05-a7c1-497c-8846-fc11b15d3e0c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3c065f20-4452q\" (UID: \"6e5e0f05-a7c1-497c-8846-fc11b15d3e0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c065f20-4452q" Nov 04 05:04:40.621846 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:40.621819 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e5e0f05-a7c1-497c-8846-fc11b15d3e0c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3c065f20-4452q\" (UID: \"6e5e0f05-a7c1-497c-8846-fc11b15d3e0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c065f20-4452q" Nov 04 05:04:40.621899 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:40.621861 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f88zl\" (UniqueName: \"kubernetes.io/projected/6e5e0f05-a7c1-497c-8846-fc11b15d3e0c-kube-api-access-f88zl\") pod \"maas-api-revoke-sub-3c065f20-4452q\" (UID: \"6e5e0f05-a7c1-497c-8846-fc11b15d3e0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c065f20-4452q" Nov 04 05:04:40.638552 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:40.638513 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3c065f20-4452q"] Nov 04 05:04:40.645906 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:40.645870 4509 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 05:04:40.646064 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:40.646003 4509 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 05:04:40.646593 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:40.646576 4509 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 05:04:40.646712 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:40.646682 4509 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-krgvj\"" Nov 04 05:04:40.727878 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:40.727831 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e5e0f05-a7c1-497c-8846-fc11b15d3e0c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3c065f20-4452q\" (UID: \"6e5e0f05-a7c1-497c-8846-fc11b15d3e0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c065f20-4452q" Nov 04 05:04:40.728874 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:40.728841 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e5e0f05-a7c1-497c-8846-fc11b15d3e0c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3c065f20-4452q\" (UID: \"6e5e0f05-a7c1-497c-8846-fc11b15d3e0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c065f20-4452q" Nov 04 05:04:40.729853 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:40.729835 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f88zl\" (UniqueName: \"kubernetes.io/projected/6e5e0f05-a7c1-497c-8846-fc11b15d3e0c-kube-api-access-f88zl\") pod \"maas-api-revoke-sub-3c065f20-4452q\" (UID: \"6e5e0f05-a7c1-497c-8846-fc11b15d3e0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c065f20-4452q" Nov 04 05:04:40.731468 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:40.729746 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e5e0f05-a7c1-497c-8846-fc11b15d3e0c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3c065f20-4452q\" (UID: \"6e5e0f05-a7c1-497c-8846-fc11b15d3e0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c065f20-4452q" Nov 04 05:04:40.743675 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:40.743643 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e5e0f05-a7c1-497c-8846-fc11b15d3e0c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3c065f20-4452q\" (UID: \"6e5e0f05-a7c1-497c-8846-fc11b15d3e0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c065f20-4452q" Nov 04 05:04:40.789804 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:40.789776 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f88zl\" (UniqueName: \"kubernetes.io/projected/6e5e0f05-a7c1-497c-8846-fc11b15d3e0c-kube-api-access-f88zl\") pod \"maas-api-revoke-sub-3c065f20-4452q\" (UID: \"6e5e0f05-a7c1-497c-8846-fc11b15d3e0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c065f20-4452q" Nov 04 05:04:40.911108 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:40.910996 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c065f20-4452q" Nov 04 05:04:41.336815 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:41.331613 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3c065f20-4452q"] Nov 04 05:04:41.383348 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:41.382699 4509 patch_prober.go:29] interesting pod/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-tc7tm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 04 05:04:41.383348 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:41.382762 4509 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-tc7tm" podUID="098b1d63-ed94-4884-a6de-428f799c3e31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 04 05:04:41.606794 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:41.606760 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c065f20-4452q" event={"ID":"6e5e0f05-a7c1-497c-8846-fc11b15d3e0c","Type":"ContainerStarted","Data":"698de7f0a95ce4a4e7ba518dba03a34bf0044d4862356bd47c3ec4fbc7b784e7"} Nov 04 05:04:42.043541 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:42.043499 4509 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39ccb86b-6j5fn"] Nov 04 05:04:42.047669 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:42.047643 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39ccb86b-6j5fn" Nov 04 05:04:42.104014 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:42.103971 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39ccb86b-6j5fn"] Nov 04 05:04:42.152118 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:42.152081 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pk68v\" (UniqueName: \"kubernetes.io/projected/db69c455-c747-4c73-8c2f-752c26822262-kube-api-access-pk68v\") pod \"maas-api-revoke-sub-39ccb86b-6j5fn\" (UID: \"db69c455-c747-4c73-8c2f-752c26822262\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39ccb86b-6j5fn" Nov 04 05:04:42.152317 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:42.152137 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db69c455-c747-4c73-8c2f-752c26822262-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-39ccb86b-6j5fn\" (UID: \"db69c455-c747-4c73-8c2f-752c26822262\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39ccb86b-6j5fn" Nov 04 05:04:42.152317 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:42.152267 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db69c455-c747-4c73-8c2f-752c26822262-maas-api-xks-ca\") pod \"maas-api-revoke-sub-39ccb86b-6j5fn\" (UID: \"db69c455-c747-4c73-8c2f-752c26822262\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39ccb86b-6j5fn" Nov 04 05:04:42.253700 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:42.253661 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pk68v\" (UniqueName: \"kubernetes.io/projected/db69c455-c747-4c73-8c2f-752c26822262-kube-api-access-pk68v\") pod \"maas-api-revoke-sub-39ccb86b-6j5fn\" (UID: \"db69c455-c747-4c73-8c2f-752c26822262\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39ccb86b-6j5fn" Nov 04 05:04:42.253890 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:42.253731 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db69c455-c747-4c73-8c2f-752c26822262-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-39ccb86b-6j5fn\" (UID: \"db69c455-c747-4c73-8c2f-752c26822262\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39ccb86b-6j5fn" Nov 04 05:04:42.253890 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:42.253767 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db69c455-c747-4c73-8c2f-752c26822262-maas-api-xks-ca\") pod \"maas-api-revoke-sub-39ccb86b-6j5fn\" (UID: \"db69c455-c747-4c73-8c2f-752c26822262\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39ccb86b-6j5fn" Nov 04 05:04:42.254524 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:42.254496 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db69c455-c747-4c73-8c2f-752c26822262-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-39ccb86b-6j5fn\" (UID: \"db69c455-c747-4c73-8c2f-752c26822262\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39ccb86b-6j5fn" Nov 04 05:04:42.256376 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:42.256349 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db69c455-c747-4c73-8c2f-752c26822262-maas-api-xks-ca\") pod \"maas-api-revoke-sub-39ccb86b-6j5fn\" (UID: \"db69c455-c747-4c73-8c2f-752c26822262\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39ccb86b-6j5fn" Nov 04 05:04:42.312261 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:42.312183 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pk68v\" (UniqueName: \"kubernetes.io/projected/db69c455-c747-4c73-8c2f-752c26822262-kube-api-access-pk68v\") pod \"maas-api-revoke-sub-39ccb86b-6j5fn\" (UID: \"db69c455-c747-4c73-8c2f-752c26822262\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39ccb86b-6j5fn" Nov 04 05:04:42.379360 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:42.377532 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39ccb86b-6j5fn" Nov 04 05:04:42.387539 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:42.384948 4509 patch_prober.go:29] interesting pod/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-tc7tm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 04 05:04:42.387539 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:42.385006 4509 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-tc7tm" podUID="098b1d63-ed94-4884-a6de-428f799c3e31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 04 05:04:42.489387 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:42.488918 4509 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-dab70d/premium-w3-dab70d-kserve-575d94569f-9q687" podUID="f12cb461-1adf-4a95-8af6-9cf0fe1c3bbb" containerName="main" containerID="cri-o://8d5a7de60a68c9d03132c998a7be9a336104add480322792a9a2565bb7c9f8bc" gracePeriod=45 Nov 04 05:04:42.584965 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:42.584850 4509 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dab70d/premium-w3-dab70d-kserve-575d94569f-9q687" podUID="f12cb461-1adf-4a95-8af6-9cf0fe1c3bbb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": dial tcp 10.133.0.65:8000: connect: connection refused" Nov 04 05:04:42.616144 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:42.616113 4509 generic.go:356] "Generic (PLEG): container finished" podID="f12cb461-1adf-4a95-8af6-9cf0fe1c3bbb" containerID="8d5a7de60a68c9d03132c998a7be9a336104add480322792a9a2565bb7c9f8bc" exitCode=0 Nov 04 05:04:42.616300 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:42.616163 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-dab70d/premium-w3-dab70d-kserve-575d94569f-9q687" event={"ID":"f12cb461-1adf-4a95-8af6-9cf0fe1c3bbb","Type":"ContainerDied","Data":"8d5a7de60a68c9d03132c998a7be9a336104add480322792a9a2565bb7c9f8bc"} Nov 04 05:04:42.618070 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:42.618042 4509 generic.go:356] "Generic (PLEG): container finished" podID="6e5e0f05-a7c1-497c-8846-fc11b15d3e0c" containerID="626caa894d3c7a35491a635dd40d8fc60feb4d4f962c2b93affba03541ace5c1" exitCode=0 Nov 04 05:04:42.618188 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:42.618132 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c065f20-4452q" event={"ID":"6e5e0f05-a7c1-497c-8846-fc11b15d3e0c","Type":"ContainerDied","Data":"626caa894d3c7a35491a635dd40d8fc60feb4d4f962c2b93affba03541ace5c1"} Nov 04 05:04:42.791242 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:42.790601 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39ccb86b-6j5fn"] Nov 04 05:04:42.797443 ip-10-0-133-180 kubenswrapper[4509]: W1104 05:04:42.797402 4509 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb69c455_c747_4c73_8c2f_752c26822262.slice/crio-cc5118511af8be7dc43d47395417570a5b2bf0ff07e2bd265533141bfc18d94d WatchSource:0}: Error finding container cc5118511af8be7dc43d47395417570a5b2bf0ff07e2bd265533141bfc18d94d: Status 404 returned error can't find the container with id cc5118511af8be7dc43d47395417570a5b2bf0ff07e2bd265533141bfc18d94d Nov 04 05:04:42.926830 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:42.926799 4509 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-dab70d/premium-w3-dab70d-kserve-575d94569f-9q687" Nov 04 05:04:43.002129 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:42.998809 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f12cb461-1adf-4a95-8af6-9cf0fe1c3bbb-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f12cb461-1adf-4a95-8af6-9cf0fe1c3bbb-tmp-dir\") pod \"f12cb461-1adf-4a95-8af6-9cf0fe1c3bbb\" (UID: \"f12cb461-1adf-4a95-8af6-9cf0fe1c3bbb\") " Nov 04 05:04:43.002129 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:42.998963 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f12cb461-1adf-4a95-8af6-9cf0fe1c3bbb-tls-certs\" (UniqueName: \"kubernetes.io/secret/f12cb461-1adf-4a95-8af6-9cf0fe1c3bbb-tls-certs\") pod \"f12cb461-1adf-4a95-8af6-9cf0fe1c3bbb\" (UID: \"f12cb461-1adf-4a95-8af6-9cf0fe1c3bbb\") " Nov 04 05:04:43.002129 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:42.999035 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f12cb461-1adf-4a95-8af6-9cf0fe1c3bbb-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f12cb461-1adf-4a95-8af6-9cf0fe1c3bbb-dshm\") pod \"f12cb461-1adf-4a95-8af6-9cf0fe1c3bbb\" (UID: \"f12cb461-1adf-4a95-8af6-9cf0fe1c3bbb\") " Nov 04 05:04:43.002129 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:42.999132 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f12cb461-1adf-4a95-8af6-9cf0fe1c3bbb-home\" (UniqueName: \"kubernetes.io/empty-dir/f12cb461-1adf-4a95-8af6-9cf0fe1c3bbb-home\") pod \"f12cb461-1adf-4a95-8af6-9cf0fe1c3bbb\" (UID: \"f12cb461-1adf-4a95-8af6-9cf0fe1c3bbb\") " Nov 04 05:04:43.002129 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:42.999172 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f12cb461-1adf-4a95-8af6-9cf0fe1c3bbb-kube-api-access-wk7jj\" (UniqueName: \"kubernetes.io/projected/f12cb461-1adf-4a95-8af6-9cf0fe1c3bbb-kube-api-access-wk7jj\") pod \"f12cb461-1adf-4a95-8af6-9cf0fe1c3bbb\" (UID: \"f12cb461-1adf-4a95-8af6-9cf0fe1c3bbb\") " Nov 04 05:04:43.002129 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:42.999205 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f12cb461-1adf-4a95-8af6-9cf0fe1c3bbb-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f12cb461-1adf-4a95-8af6-9cf0fe1c3bbb-model-cache\") pod \"f12cb461-1adf-4a95-8af6-9cf0fe1c3bbb\" (UID: \"f12cb461-1adf-4a95-8af6-9cf0fe1c3bbb\") " Nov 04 05:04:43.009400 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:43.004850 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f12cb461-1adf-4a95-8af6-9cf0fe1c3bbb-tmp-dir" pod "f12cb461-1adf-4a95-8af6-9cf0fe1c3bbb" (UID: "f12cb461-1adf-4a95-8af6-9cf0fe1c3bbb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:04:43.012467 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:43.010064 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f12cb461-1adf-4a95-8af6-9cf0fe1c3bbb-home" pod "f12cb461-1adf-4a95-8af6-9cf0fe1c3bbb" (UID: "f12cb461-1adf-4a95-8af6-9cf0fe1c3bbb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:04:43.019347 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:43.013144 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f12cb461-1adf-4a95-8af6-9cf0fe1c3bbb-model-cache" pod "f12cb461-1adf-4a95-8af6-9cf0fe1c3bbb" (UID: "f12cb461-1adf-4a95-8af6-9cf0fe1c3bbb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:04:43.092120 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:43.092050 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f12cb461-1adf-4a95-8af6-9cf0fe1c3bbb-tls-certs" pod "f12cb461-1adf-4a95-8af6-9cf0fe1c3bbb" (UID: "f12cb461-1adf-4a95-8af6-9cf0fe1c3bbb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 05:04:43.092834 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:43.092787 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f12cb461-1adf-4a95-8af6-9cf0fe1c3bbb-dshm" pod "f12cb461-1adf-4a95-8af6-9cf0fe1c3bbb" (UID: "f12cb461-1adf-4a95-8af6-9cf0fe1c3bbb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:04:43.093497 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:43.093473 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f12cb461-1adf-4a95-8af6-9cf0fe1c3bbb-kube-api-access-wk7jj" pod "f12cb461-1adf-4a95-8af6-9cf0fe1c3bbb" (UID: "f12cb461-1adf-4a95-8af6-9cf0fe1c3bbb"). InnerVolumeSpecName "kube-api-access-wk7jj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 05:04:43.101193 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:43.101080 4509 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f12cb461-1adf-4a95-8af6-9cf0fe1c3bbb-home\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:04:43.101193 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:43.101108 4509 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wk7jj\" (UniqueName: \"kubernetes.io/projected/f12cb461-1adf-4a95-8af6-9cf0fe1c3bbb-kube-api-access-wk7jj\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:04:43.101193 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:43.101132 4509 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f12cb461-1adf-4a95-8af6-9cf0fe1c3bbb-model-cache\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:04:43.101193 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:43.101141 4509 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f12cb461-1adf-4a95-8af6-9cf0fe1c3bbb-tmp-dir\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:04:43.101193 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:43.101149 4509 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f12cb461-1adf-4a95-8af6-9cf0fe1c3bbb-tls-certs\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:04:43.101193 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:43.101156 4509 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f12cb461-1adf-4a95-8af6-9cf0fe1c3bbb-dshm\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:04:43.384361 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:43.381883 4509 patch_prober.go:29] interesting pod/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-tc7tm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 04 05:04:43.384361 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:43.381944 4509 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-tc7tm" podUID="098b1d63-ed94-4884-a6de-428f799c3e31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 04 05:04:43.625419 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:43.625301 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39ccb86b-6j5fn" event={"ID":"db69c455-c747-4c73-8c2f-752c26822262","Type":"ContainerStarted","Data":"abab568c99208ecfd7a1a88e5a76eef3f18d0d900c464ff8bedf216983d500b0"} Nov 04 05:04:43.625419 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:43.625373 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39ccb86b-6j5fn" event={"ID":"db69c455-c747-4c73-8c2f-752c26822262","Type":"ContainerStarted","Data":"cc5118511af8be7dc43d47395417570a5b2bf0ff07e2bd265533141bfc18d94d"} Nov 04 05:04:43.626836 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:43.626808 4509 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-dab70d/premium-w3-dab70d-kserve-575d94569f-9q687" Nov 04 05:04:43.626952 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:43.626840 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-dab70d/premium-w3-dab70d-kserve-575d94569f-9q687" event={"ID":"f12cb461-1adf-4a95-8af6-9cf0fe1c3bbb","Type":"ContainerDied","Data":"f31112ad28b60012afbe9908b9087ba4cc4ce8ae0308871555be5110659ad875"} Nov 04 05:04:43.626952 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:43.626888 4509 scope.go:122] "RemoveContainer" containerID="8d5a7de60a68c9d03132c998a7be9a336104add480322792a9a2565bb7c9f8bc" Nov 04 05:04:43.743397 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:43.743347 4509 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39ccb86b-6j5fn" podStartSLOduration=1.74331155 podStartE2EDuration="1.74331155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 05:04:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 05:04:43.741701429 +0000 UTC m=+1047.619424951" watchObservedRunningTime="2026-11-04 05:04:43.74331155 +0000 UTC m=+1047.621035071" Nov 04 05:04:43.806314 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:43.806287 4509 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c065f20-4452q" Nov 04 05:04:43.854425 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:43.854391 4509 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-dab70d/premium-w3-dab70d-kserve-575d94569f-9q687"] Nov 04 05:04:43.860920 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:43.860886 4509 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-dab70d/premium-w3-dab70d-kserve-575d94569f-9q687"] Nov 04 05:04:43.915062 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:43.914959 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e5e0f05-a7c1-497c-8846-fc11b15d3e0c-kube-api-access-f88zl\" (UniqueName: \"kubernetes.io/projected/6e5e0f05-a7c1-497c-8846-fc11b15d3e0c-kube-api-access-f88zl\") pod \"6e5e0f05-a7c1-497c-8846-fc11b15d3e0c\" (UID: \"6e5e0f05-a7c1-497c-8846-fc11b15d3e0c\") " Nov 04 05:04:43.915295 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:43.915058 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e5e0f05-a7c1-497c-8846-fc11b15d3e0c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e5e0f05-a7c1-497c-8846-fc11b15d3e0c-maas-api-xks-ca\") pod \"6e5e0f05-a7c1-497c-8846-fc11b15d3e0c\" (UID: \"6e5e0f05-a7c1-497c-8846-fc11b15d3e0c\") " Nov 04 05:04:43.915295 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:43.915119 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e5e0f05-a7c1-497c-8846-fc11b15d3e0c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e5e0f05-a7c1-497c-8846-fc11b15d3e0c-maas-api-openshift-ca\") pod \"6e5e0f05-a7c1-497c-8846-fc11b15d3e0c\" (UID: \"6e5e0f05-a7c1-497c-8846-fc11b15d3e0c\") " Nov 04 05:04:43.915960 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:43.915908 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e5e0f05-a7c1-497c-8846-fc11b15d3e0c-maas-api-openshift-ca" pod "6e5e0f05-a7c1-497c-8846-fc11b15d3e0c" (UID: "6e5e0f05-a7c1-497c-8846-fc11b15d3e0c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 05:04:43.917883 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:43.917860 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e5e0f05-a7c1-497c-8846-fc11b15d3e0c-maas-api-xks-ca" pod "6e5e0f05-a7c1-497c-8846-fc11b15d3e0c" (UID: "6e5e0f05-a7c1-497c-8846-fc11b15d3e0c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 05:04:43.918111 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:43.918089 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e5e0f05-a7c1-497c-8846-fc11b15d3e0c-kube-api-access-f88zl" pod "6e5e0f05-a7c1-497c-8846-fc11b15d3e0c" (UID: "6e5e0f05-a7c1-497c-8846-fc11b15d3e0c"). InnerVolumeSpecName "kube-api-access-f88zl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 05:04:44.016524 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:44.016467 4509 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f88zl\" (UniqueName: \"kubernetes.io/projected/6e5e0f05-a7c1-497c-8846-fc11b15d3e0c-kube-api-access-f88zl\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:04:44.016524 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:44.016513 4509 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e5e0f05-a7c1-497c-8846-fc11b15d3e0c-maas-api-xks-ca\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:04:44.016524 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:44.016523 4509 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e5e0f05-a7c1-497c-8846-fc11b15d3e0c-maas-api-openshift-ca\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:04:44.263509 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:44.263470 4509 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f12cb461-1adf-4a95-8af6-9cf0fe1c3bbb" path="/var/lib/kubelet/pods/f12cb461-1adf-4a95-8af6-9cf0fe1c3bbb/volumes" Nov 04 05:04:44.378691 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:44.378651 4509 patch_prober.go:29] interesting pod/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-tc7tm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 04 05:04:44.378862 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:44.378717 4509 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-tc7tm" podUID="098b1d63-ed94-4884-a6de-428f799c3e31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 04 05:04:44.454372 ip-10-0-133-180 kubenswrapper[4509]: E1104 05:04:44.454302 4509 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmdhl" containerName="extractor" Nov 04 05:04:44.640727 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:44.640642 4509 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c065f20-4452q" Nov 04 05:04:44.640727 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:44.640638 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c065f20-4452q" event={"ID":"6e5e0f05-a7c1-497c-8846-fc11b15d3e0c","Type":"ContainerDied","Data":"698de7f0a95ce4a4e7ba518dba03a34bf0044d4862356bd47c3ec4fbc7b784e7"} Nov 04 05:04:44.640948 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:44.640774 4509 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="698de7f0a95ce4a4e7ba518dba03a34bf0044d4862356bd47c3ec4fbc7b784e7" Nov 04 05:04:44.642531 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:44.642502 4509 generic.go:356] "Generic (PLEG): container finished" podID="db69c455-c747-4c73-8c2f-752c26822262" containerID="abab568c99208ecfd7a1a88e5a76eef3f18d0d900c464ff8bedf216983d500b0" exitCode=0 Nov 04 05:04:44.642722 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:44.642608 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39ccb86b-6j5fn" event={"ID":"db69c455-c747-4c73-8c2f-752c26822262","Type":"ContainerDied","Data":"abab568c99208ecfd7a1a88e5a76eef3f18d0d900c464ff8bedf216983d500b0"} Nov 04 05:04:45.322048 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:45.322003 4509 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-dab70d-6ddd49994d-ggggw" podUID="f71da8bf-e97b-4a2a-bf50-9588fbc73c48" containerName="payload-processing" containerID="cri-o://885724b47455392fd91c42875e22e92c1b22b7bc45ba141b3edf3876a0f8adf7" gracePeriod=25 Nov 04 05:04:45.388092 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:45.386829 4509 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-dab70d-695b6bf8dc-vlnmz" podUID="3137d988-598c-4f6a-a797-a08ed4448b8a" containerName="payload-pre-processing" containerID="cri-o://410c6a472fbecd80e92b65c59f31cd3a6b7309130c7eac15ff8712ddd723a8fd" gracePeriod=25 Nov 04 05:04:45.423711 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:45.423468 4509 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-tc7tm" Nov 04 05:04:45.424714 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:45.424693 4509 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-tc7tm" Nov 04 05:04:45.425746 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:45.425726 4509 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-tc7tm" Nov 04 05:04:45.650750 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:45.650722 4509 generic.go:356] "Generic (PLEG): container finished" podID="f71da8bf-e97b-4a2a-bf50-9588fbc73c48" containerID="885724b47455392fd91c42875e22e92c1b22b7bc45ba141b3edf3876a0f8adf7" exitCode=0 Nov 04 05:04:45.651128 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:45.650803 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-dab70d-6ddd49994d-ggggw" event={"ID":"f71da8bf-e97b-4a2a-bf50-9588fbc73c48","Type":"ContainerDied","Data":"885724b47455392fd91c42875e22e92c1b22b7bc45ba141b3edf3876a0f8adf7"} Nov 04 05:04:45.653743 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:45.653715 4509 generic.go:356] "Generic (PLEG): container finished" podID="3137d988-598c-4f6a-a797-a08ed4448b8a" containerID="410c6a472fbecd80e92b65c59f31cd3a6b7309130c7eac15ff8712ddd723a8fd" exitCode=0 Nov 04 05:04:45.653870 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:45.653771 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-dab70d-695b6bf8dc-vlnmz" event={"ID":"3137d988-598c-4f6a-a797-a08ed4448b8a","Type":"ContainerDied","Data":"410c6a472fbecd80e92b65c59f31cd3a6b7309130c7eac15ff8712ddd723a8fd"} Nov 04 05:04:45.701839 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:45.701519 4509 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-dab70d-6ddd49994d-ggggw" Nov 04 05:04:45.758210 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:45.758191 4509 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-dab70d-695b6bf8dc-vlnmz" Nov 04 05:04:45.851851 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:45.851822 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3137d988-598c-4f6a-a797-a08ed4448b8a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3137d988-598c-4f6a-a797-a08ed4448b8a-plugins-config-volume\") pod \"3137d988-598c-4f6a-a797-a08ed4448b8a\" (UID: \"3137d988-598c-4f6a-a797-a08ed4448b8a\") " Nov 04 05:04:45.851980 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:45.851918 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3137d988-598c-4f6a-a797-a08ed4448b8a-kube-api-access-4qw5s\" (UniqueName: \"kubernetes.io/projected/3137d988-598c-4f6a-a797-a08ed4448b8a-kube-api-access-4qw5s\") pod \"3137d988-598c-4f6a-a797-a08ed4448b8a\" (UID: \"3137d988-598c-4f6a-a797-a08ed4448b8a\") " Nov 04 05:04:45.852086 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:45.852056 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f71da8bf-e97b-4a2a-bf50-9588fbc73c48-kube-api-access-sc5v5\" (UniqueName: \"kubernetes.io/projected/f71da8bf-e97b-4a2a-bf50-9588fbc73c48-kube-api-access-sc5v5\") pod \"f71da8bf-e97b-4a2a-bf50-9588fbc73c48\" (UID: \"f71da8bf-e97b-4a2a-bf50-9588fbc73c48\") " Nov 04 05:04:45.852348 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:45.852286 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3137d988-598c-4f6a-a797-a08ed4448b8a-plugins-config-volume" pod "3137d988-598c-4f6a-a797-a08ed4448b8a" (UID: "3137d988-598c-4f6a-a797-a08ed4448b8a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 05:04:45.852569 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:45.852549 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f71da8bf-e97b-4a2a-bf50-9588fbc73c48-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f71da8bf-e97b-4a2a-bf50-9588fbc73c48-plugins-config-volume\") pod \"f71da8bf-e97b-4a2a-bf50-9588fbc73c48\" (UID: \"f71da8bf-e97b-4a2a-bf50-9588fbc73c48\") " Nov 04 05:04:45.853062 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:45.853039 4509 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3137d988-598c-4f6a-a797-a08ed4448b8a-plugins-config-volume\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:04:45.853062 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:45.853054 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f71da8bf-e97b-4a2a-bf50-9588fbc73c48-plugins-config-volume" pod "f71da8bf-e97b-4a2a-bf50-9588fbc73c48" (UID: "f71da8bf-e97b-4a2a-bf50-9588fbc73c48"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 05:04:45.854547 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:45.854520 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f71da8bf-e97b-4a2a-bf50-9588fbc73c48-kube-api-access-sc5v5" pod "f71da8bf-e97b-4a2a-bf50-9588fbc73c48" (UID: "f71da8bf-e97b-4a2a-bf50-9588fbc73c48"). InnerVolumeSpecName "kube-api-access-sc5v5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 05:04:45.855108 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:45.855086 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3137d988-598c-4f6a-a797-a08ed4448b8a-kube-api-access-4qw5s" pod "3137d988-598c-4f6a-a797-a08ed4448b8a" (UID: "3137d988-598c-4f6a-a797-a08ed4448b8a"). InnerVolumeSpecName "kube-api-access-4qw5s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 05:04:45.881406 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:45.881382 4509 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39ccb86b-6j5fn" Nov 04 05:04:45.953836 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:45.953799 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db69c455-c747-4c73-8c2f-752c26822262-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db69c455-c747-4c73-8c2f-752c26822262-maas-api-openshift-ca\") pod \"db69c455-c747-4c73-8c2f-752c26822262\" (UID: \"db69c455-c747-4c73-8c2f-752c26822262\") " Nov 04 05:04:45.954293 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:45.953929 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db69c455-c747-4c73-8c2f-752c26822262-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db69c455-c747-4c73-8c2f-752c26822262-maas-api-xks-ca\") pod \"db69c455-c747-4c73-8c2f-752c26822262\" (UID: \"db69c455-c747-4c73-8c2f-752c26822262\") " Nov 04 05:04:45.954293 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:45.954017 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db69c455-c747-4c73-8c2f-752c26822262-kube-api-access-pk68v\" (UniqueName: \"kubernetes.io/projected/db69c455-c747-4c73-8c2f-752c26822262-kube-api-access-pk68v\") pod \"db69c455-c747-4c73-8c2f-752c26822262\" (UID: \"db69c455-c747-4c73-8c2f-752c26822262\") " Nov 04 05:04:45.954293 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:45.954232 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db69c455-c747-4c73-8c2f-752c26822262-maas-api-openshift-ca" pod "db69c455-c747-4c73-8c2f-752c26822262" (UID: "db69c455-c747-4c73-8c2f-752c26822262"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 05:04:45.954594 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:45.954575 4509 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4qw5s\" (UniqueName: \"kubernetes.io/projected/3137d988-598c-4f6a-a797-a08ed4448b8a-kube-api-access-4qw5s\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:04:45.954666 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:45.954598 4509 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sc5v5\" (UniqueName: \"kubernetes.io/projected/f71da8bf-e97b-4a2a-bf50-9588fbc73c48-kube-api-access-sc5v5\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:04:45.954666 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:45.954613 4509 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db69c455-c747-4c73-8c2f-752c26822262-maas-api-openshift-ca\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:04:45.954666 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:45.954626 4509 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f71da8bf-e97b-4a2a-bf50-9588fbc73c48-plugins-config-volume\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:04:45.958671 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:45.956724 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db69c455-c747-4c73-8c2f-752c26822262-maas-api-xks-ca" pod "db69c455-c747-4c73-8c2f-752c26822262" (UID: "db69c455-c747-4c73-8c2f-752c26822262"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 05:04:45.958671 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:45.956994 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db69c455-c747-4c73-8c2f-752c26822262-kube-api-access-pk68v" pod "db69c455-c747-4c73-8c2f-752c26822262" (UID: "db69c455-c747-4c73-8c2f-752c26822262"). InnerVolumeSpecName "kube-api-access-pk68v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 05:04:46.055752 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:46.055711 4509 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db69c455-c747-4c73-8c2f-752c26822262-maas-api-xks-ca\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:04:46.055752 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:46.055750 4509 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pk68v\" (UniqueName: \"kubernetes.io/projected/db69c455-c747-4c73-8c2f-752c26822262-kube-api-access-pk68v\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:04:46.323811 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:46.323776 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-42dnm"] Nov 04 05:04:46.388714 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:46.388679 4509 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-370078/premium-w2-370078-kserve-54d45b6bc-q4xfk"] Nov 04 05:04:46.661189 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:46.661100 4509 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-dab70d-695b6bf8dc-vlnmz" Nov 04 05:04:46.661189 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:46.661121 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-dab70d-695b6bf8dc-vlnmz" event={"ID":"3137d988-598c-4f6a-a797-a08ed4448b8a","Type":"ContainerDied","Data":"cbd03943f3ed8f758dd07f1e2d3496299dd4c9cfcc0f850d79014f49c42b7ad2"} Nov 04 05:04:46.661778 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:46.661195 4509 scope.go:122] "RemoveContainer" containerID="410c6a472fbecd80e92b65c59f31cd3a6b7309130c7eac15ff8712ddd723a8fd" Nov 04 05:04:46.663879 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:46.663856 4509 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39ccb86b-6j5fn" Nov 04 05:04:46.664494 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:46.664392 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39ccb86b-6j5fn" event={"ID":"db69c455-c747-4c73-8c2f-752c26822262","Type":"ContainerDied","Data":"cc5118511af8be7dc43d47395417570a5b2bf0ff07e2bd265533141bfc18d94d"} Nov 04 05:04:46.664494 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:46.664438 4509 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cc5118511af8be7dc43d47395417570a5b2bf0ff07e2bd265533141bfc18d94d" Nov 04 05:04:46.666656 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:46.666634 4509 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-dab70d-6ddd49994d-ggggw" Nov 04 05:04:46.666756 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:46.666679 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-dab70d-6ddd49994d-ggggw" event={"ID":"f71da8bf-e97b-4a2a-bf50-9588fbc73c48","Type":"ContainerDied","Data":"3a8987080f89e9bec9f601d13ade7779170bd7fd7edfd589d851e661c4627861"} Nov 04 05:04:46.673673 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:46.673639 4509 scope.go:122] "RemoveContainer" containerID="885724b47455392fd91c42875e22e92c1b22b7bc45ba141b3edf3876a0f8adf7" Nov 04 05:04:46.841375 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:46.839474 4509 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-dab70d-695b6bf8dc-vlnmz"] Nov 04 05:04:46.847345 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:46.844225 4509 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-dab70d-695b6bf8dc-vlnmz"] Nov 04 05:04:46.961877 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:46.961843 4509 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-dab70d-6ddd49994d-ggggw"] Nov 04 05:04:46.974272 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:46.974239 4509 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-dab70d-6ddd49994d-ggggw"] Nov 04 05:04:48.272609 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:48.270671 4509 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3137d988-598c-4f6a-a797-a08ed4448b8a" path="/var/lib/kubelet/pods/3137d988-598c-4f6a-a797-a08ed4448b8a/volumes" Nov 04 05:04:48.287152 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:48.287077 4509 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f71da8bf-e97b-4a2a-bf50-9588fbc73c48" path="/var/lib/kubelet/pods/f71da8bf-e97b-4a2a-bf50-9588fbc73c48/volumes" Nov 04 05:04:49.882479 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:49.882435 4509 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80b24922-sdsg5"] Nov 04 05:04:49.883096 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:49.883074 4509 cpu_manager.go:414] "containerMap: removing container" podUID="6e5e0f05-a7c1-497c-8846-fc11b15d3e0c" containerName="revoke-keys" Nov 04 05:04:49.883200 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:49.883098 4509 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e5e0f05-a7c1-497c-8846-fc11b15d3e0c" containerName="revoke-keys" Nov 04 05:04:49.883200 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:49.883124 4509 cpu_manager.go:414] "containerMap: removing container" podUID="f12cb461-1adf-4a95-8af6-9cf0fe1c3bbb" containerName="main" Nov 04 05:04:49.883200 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:49.883132 4509 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f12cb461-1adf-4a95-8af6-9cf0fe1c3bbb" containerName="main" Nov 04 05:04:49.883200 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:49.883145 4509 cpu_manager.go:414] "containerMap: removing container" podUID="db69c455-c747-4c73-8c2f-752c26822262" containerName="revoke-keys" Nov 04 05:04:49.883200 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:49.883153 4509 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db69c455-c747-4c73-8c2f-752c26822262" containerName="revoke-keys" Nov 04 05:04:49.883200 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:49.883187 4509 cpu_manager.go:414] "containerMap: removing container" podUID="3137d988-598c-4f6a-a797-a08ed4448b8a" containerName="payload-pre-processing" Nov 04 05:04:49.883200 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:49.883194 4509 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3137d988-598c-4f6a-a797-a08ed4448b8a" containerName="payload-pre-processing" Nov 04 05:04:49.883549 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:49.883214 4509 cpu_manager.go:414] "containerMap: removing container" podUID="f71da8bf-e97b-4a2a-bf50-9588fbc73c48" containerName="payload-processing" Nov 04 05:04:49.883549 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:49.883222 4509 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f71da8bf-e97b-4a2a-bf50-9588fbc73c48" containerName="payload-processing" Nov 04 05:04:49.883549 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:49.883385 4509 memory_manager.go:365] "RemoveStaleState removing state" podUID="3137d988-598c-4f6a-a797-a08ed4448b8a" containerName="payload-pre-processing" Nov 04 05:04:49.883549 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:49.883400 4509 memory_manager.go:365] "RemoveStaleState removing state" podUID="f71da8bf-e97b-4a2a-bf50-9588fbc73c48" containerName="payload-processing" Nov 04 05:04:49.883549 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:49.883412 4509 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e5e0f05-a7c1-497c-8846-fc11b15d3e0c" containerName="revoke-keys" Nov 04 05:04:49.883549 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:49.883423 4509 memory_manager.go:365] "RemoveStaleState removing state" podUID="f12cb461-1adf-4a95-8af6-9cf0fe1c3bbb" containerName="main" Nov 04 05:04:49.883549 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:49.883432 4509 memory_manager.go:365] "RemoveStaleState removing state" podUID="db69c455-c747-4c73-8c2f-752c26822262" containerName="revoke-keys" Nov 04 05:04:49.888289 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:49.888263 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80b24922-sdsg5" Nov 04 05:04:49.991753 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:49.991720 4509 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 05:04:49.991753 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:49.991741 4509 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 05:04:49.991753 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:49.991752 4509 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 05:04:49.992015 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:49.991813 4509 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-krgvj\"" Nov 04 05:04:49.994924 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:49.994898 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80b24922-sdsg5"] Nov 04 05:04:50.009244 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:50.009206 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tw4nt\" (UniqueName: \"kubernetes.io/projected/0c9909c8-ffcb-4601-b38d-923c92a786d0-kube-api-access-tw4nt\") pod \"maas-api-revoke-sub-80b24922-sdsg5\" (UID: \"0c9909c8-ffcb-4601-b38d-923c92a786d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80b24922-sdsg5" Nov 04 05:04:50.009244 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:50.009248 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c9909c8-ffcb-4601-b38d-923c92a786d0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-80b24922-sdsg5\" (UID: \"0c9909c8-ffcb-4601-b38d-923c92a786d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80b24922-sdsg5" Nov 04 05:04:50.009445 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:50.009373 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c9909c8-ffcb-4601-b38d-923c92a786d0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-80b24922-sdsg5\" (UID: \"0c9909c8-ffcb-4601-b38d-923c92a786d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80b24922-sdsg5" Nov 04 05:04:50.110394 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:50.110307 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tw4nt\" (UniqueName: \"kubernetes.io/projected/0c9909c8-ffcb-4601-b38d-923c92a786d0-kube-api-access-tw4nt\") pod \"maas-api-revoke-sub-80b24922-sdsg5\" (UID: \"0c9909c8-ffcb-4601-b38d-923c92a786d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80b24922-sdsg5" Nov 04 05:04:50.110573 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:50.110406 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c9909c8-ffcb-4601-b38d-923c92a786d0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-80b24922-sdsg5\" (UID: \"0c9909c8-ffcb-4601-b38d-923c92a786d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80b24922-sdsg5" Nov 04 05:04:50.110573 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:50.110454 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c9909c8-ffcb-4601-b38d-923c92a786d0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-80b24922-sdsg5\" (UID: \"0c9909c8-ffcb-4601-b38d-923c92a786d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80b24922-sdsg5" Nov 04 05:04:50.111107 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:50.111082 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c9909c8-ffcb-4601-b38d-923c92a786d0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-80b24922-sdsg5\" (UID: \"0c9909c8-ffcb-4601-b38d-923c92a786d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80b24922-sdsg5" Nov 04 05:04:50.113062 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:50.113039 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c9909c8-ffcb-4601-b38d-923c92a786d0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-80b24922-sdsg5\" (UID: \"0c9909c8-ffcb-4601-b38d-923c92a786d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80b24922-sdsg5" Nov 04 05:04:50.167742 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:50.167656 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tw4nt\" (UniqueName: \"kubernetes.io/projected/0c9909c8-ffcb-4601-b38d-923c92a786d0-kube-api-access-tw4nt\") pod \"maas-api-revoke-sub-80b24922-sdsg5\" (UID: \"0c9909c8-ffcb-4601-b38d-923c92a786d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80b24922-sdsg5" Nov 04 05:04:50.202711 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:50.202667 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80b24922-sdsg5" Nov 04 05:04:50.585563 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:50.585524 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80b24922-sdsg5"] Nov 04 05:04:50.587246 ip-10-0-133-180 kubenswrapper[4509]: W1104 05:04:50.587224 4509 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c9909c8_ffcb_4601_b38d_923c92a786d0.slice/crio-3b018decd7744a8186517dd755f54d342c46f19ed56af35b19fc0196f8626a5b WatchSource:0}: Error finding container 3b018decd7744a8186517dd755f54d342c46f19ed56af35b19fc0196f8626a5b: Status 404 returned error can't find the container with id 3b018decd7744a8186517dd755f54d342c46f19ed56af35b19fc0196f8626a5b Nov 04 05:04:50.695232 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:50.695172 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80b24922-sdsg5" event={"ID":"0c9909c8-ffcb-4601-b38d-923c92a786d0","Type":"ContainerStarted","Data":"3b018decd7744a8186517dd755f54d342c46f19ed56af35b19fc0196f8626a5b"} Nov 04 05:04:50.873903 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:50.873806 4509 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-cedfca/distinct-2-w6-cedfca-kserve-5465b749bb-g2tkm"] Nov 04 05:04:50.878902 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:50.878881 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-cedfca/distinct-2-w6-cedfca-kserve-5465b749bb-g2tkm" Nov 04 05:04:50.946090 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:50.946058 4509 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-cedfca\"/\"distinct-2-w6-cedfca-kserve-self-signed-certs\"" Nov 04 05:04:50.949252 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:50.949217 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-cedfca/distinct-2-w6-cedfca-kserve-5465b749bb-g2tkm"] Nov 04 05:04:51.022255 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:51.022209 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97e3afca-28dd-4f22-89bf-14245acc4544-tmp-dir\") pod \"distinct-2-w6-cedfca-kserve-5465b749bb-g2tkm\" (UID: \"97e3afca-28dd-4f22-89bf-14245acc4544\") " pod="e2e-models-e2e-worker-w6-cedfca/distinct-2-w6-cedfca-kserve-5465b749bb-g2tkm" Nov 04 05:04:51.022255 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:51.022257 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97e3afca-28dd-4f22-89bf-14245acc4544-tls-certs\") pod \"distinct-2-w6-cedfca-kserve-5465b749bb-g2tkm\" (UID: \"97e3afca-28dd-4f22-89bf-14245acc4544\") " pod="e2e-models-e2e-worker-w6-cedfca/distinct-2-w6-cedfca-kserve-5465b749bb-g2tkm" Nov 04 05:04:51.022641 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:51.022360 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97e3afca-28dd-4f22-89bf-14245acc4544-home\") pod \"distinct-2-w6-cedfca-kserve-5465b749bb-g2tkm\" (UID: \"97e3afca-28dd-4f22-89bf-14245acc4544\") " pod="e2e-models-e2e-worker-w6-cedfca/distinct-2-w6-cedfca-kserve-5465b749bb-g2tkm" Nov 04 05:04:51.022641 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:51.022442 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97e3afca-28dd-4f22-89bf-14245acc4544-dshm\") pod \"distinct-2-w6-cedfca-kserve-5465b749bb-g2tkm\" (UID: \"97e3afca-28dd-4f22-89bf-14245acc4544\") " pod="e2e-models-e2e-worker-w6-cedfca/distinct-2-w6-cedfca-kserve-5465b749bb-g2tkm" Nov 04 05:04:51.022641 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:51.022465 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97e3afca-28dd-4f22-89bf-14245acc4544-model-cache\") pod \"distinct-2-w6-cedfca-kserve-5465b749bb-g2tkm\" (UID: \"97e3afca-28dd-4f22-89bf-14245acc4544\") " pod="e2e-models-e2e-worker-w6-cedfca/distinct-2-w6-cedfca-kserve-5465b749bb-g2tkm" Nov 04 05:04:51.022641 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:51.022501 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rfcn\" (UniqueName: \"kubernetes.io/projected/97e3afca-28dd-4f22-89bf-14245acc4544-kube-api-access-5rfcn\") pod \"distinct-2-w6-cedfca-kserve-5465b749bb-g2tkm\" (UID: \"97e3afca-28dd-4f22-89bf-14245acc4544\") " pod="e2e-models-e2e-worker-w6-cedfca/distinct-2-w6-cedfca-kserve-5465b749bb-g2tkm" Nov 04 05:04:51.123835 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:51.123801 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97e3afca-28dd-4f22-89bf-14245acc4544-home\") pod \"distinct-2-w6-cedfca-kserve-5465b749bb-g2tkm\" (UID: \"97e3afca-28dd-4f22-89bf-14245acc4544\") " pod="e2e-models-e2e-worker-w6-cedfca/distinct-2-w6-cedfca-kserve-5465b749bb-g2tkm" Nov 04 05:04:51.124043 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:51.123878 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97e3afca-28dd-4f22-89bf-14245acc4544-dshm\") pod \"distinct-2-w6-cedfca-kserve-5465b749bb-g2tkm\" (UID: \"97e3afca-28dd-4f22-89bf-14245acc4544\") " pod="e2e-models-e2e-worker-w6-cedfca/distinct-2-w6-cedfca-kserve-5465b749bb-g2tkm" Nov 04 05:04:51.124043 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:51.123899 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97e3afca-28dd-4f22-89bf-14245acc4544-model-cache\") pod \"distinct-2-w6-cedfca-kserve-5465b749bb-g2tkm\" (UID: \"97e3afca-28dd-4f22-89bf-14245acc4544\") " pod="e2e-models-e2e-worker-w6-cedfca/distinct-2-w6-cedfca-kserve-5465b749bb-g2tkm" Nov 04 05:04:51.124043 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:51.123925 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rfcn\" (UniqueName: \"kubernetes.io/projected/97e3afca-28dd-4f22-89bf-14245acc4544-kube-api-access-5rfcn\") pod \"distinct-2-w6-cedfca-kserve-5465b749bb-g2tkm\" (UID: \"97e3afca-28dd-4f22-89bf-14245acc4544\") " pod="e2e-models-e2e-worker-w6-cedfca/distinct-2-w6-cedfca-kserve-5465b749bb-g2tkm" Nov 04 05:04:51.124043 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:51.124019 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97e3afca-28dd-4f22-89bf-14245acc4544-tmp-dir\") pod \"distinct-2-w6-cedfca-kserve-5465b749bb-g2tkm\" (UID: \"97e3afca-28dd-4f22-89bf-14245acc4544\") " pod="e2e-models-e2e-worker-w6-cedfca/distinct-2-w6-cedfca-kserve-5465b749bb-g2tkm" Nov 04 05:04:51.124043 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:51.124042 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97e3afca-28dd-4f22-89bf-14245acc4544-tls-certs\") pod \"distinct-2-w6-cedfca-kserve-5465b749bb-g2tkm\" (UID: \"97e3afca-28dd-4f22-89bf-14245acc4544\") " pod="e2e-models-e2e-worker-w6-cedfca/distinct-2-w6-cedfca-kserve-5465b749bb-g2tkm" Nov 04 05:04:51.124335 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:51.124303 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97e3afca-28dd-4f22-89bf-14245acc4544-home\") pod \"distinct-2-w6-cedfca-kserve-5465b749bb-g2tkm\" (UID: \"97e3afca-28dd-4f22-89bf-14245acc4544\") " pod="e2e-models-e2e-worker-w6-cedfca/distinct-2-w6-cedfca-kserve-5465b749bb-g2tkm" Nov 04 05:04:51.124474 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:51.124434 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97e3afca-28dd-4f22-89bf-14245acc4544-model-cache\") pod \"distinct-2-w6-cedfca-kserve-5465b749bb-g2tkm\" (UID: \"97e3afca-28dd-4f22-89bf-14245acc4544\") " pod="e2e-models-e2e-worker-w6-cedfca/distinct-2-w6-cedfca-kserve-5465b749bb-g2tkm" Nov 04 05:04:51.124588 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:51.124504 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97e3afca-28dd-4f22-89bf-14245acc4544-tmp-dir\") pod \"distinct-2-w6-cedfca-kserve-5465b749bb-g2tkm\" (UID: \"97e3afca-28dd-4f22-89bf-14245acc4544\") " pod="e2e-models-e2e-worker-w6-cedfca/distinct-2-w6-cedfca-kserve-5465b749bb-g2tkm" Nov 04 05:04:51.126477 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:51.126455 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97e3afca-28dd-4f22-89bf-14245acc4544-dshm\") pod \"distinct-2-w6-cedfca-kserve-5465b749bb-g2tkm\" (UID: \"97e3afca-28dd-4f22-89bf-14245acc4544\") " pod="e2e-models-e2e-worker-w6-cedfca/distinct-2-w6-cedfca-kserve-5465b749bb-g2tkm" Nov 04 05:04:51.126703 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:51.126686 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97e3afca-28dd-4f22-89bf-14245acc4544-tls-certs\") pod \"distinct-2-w6-cedfca-kserve-5465b749bb-g2tkm\" (UID: \"97e3afca-28dd-4f22-89bf-14245acc4544\") " pod="e2e-models-e2e-worker-w6-cedfca/distinct-2-w6-cedfca-kserve-5465b749bb-g2tkm" Nov 04 05:04:51.181679 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:51.181640 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rfcn\" (UniqueName: \"kubernetes.io/projected/97e3afca-28dd-4f22-89bf-14245acc4544-kube-api-access-5rfcn\") pod \"distinct-2-w6-cedfca-kserve-5465b749bb-g2tkm\" (UID: \"97e3afca-28dd-4f22-89bf-14245acc4544\") " pod="e2e-models-e2e-worker-w6-cedfca/distinct-2-w6-cedfca-kserve-5465b749bb-g2tkm" Nov 04 05:04:51.194749 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:51.194719 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-cedfca/distinct-2-w6-cedfca-kserve-5465b749bb-g2tkm" Nov 04 05:04:51.604986 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:51.604955 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-cedfca/distinct-2-w6-cedfca-kserve-5465b749bb-g2tkm"] Nov 04 05:04:51.606374 ip-10-0-133-180 kubenswrapper[4509]: W1104 05:04:51.606314 4509 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod97e3afca_28dd_4f22_89bf_14245acc4544.slice/crio-c62e34d7083533c9d1ce92d8a43f5daa29af4737c0ce4696d134aea343cd6528 WatchSource:0}: Error finding container c62e34d7083533c9d1ce92d8a43f5daa29af4737c0ce4696d134aea343cd6528: Status 404 returned error can't find the container with id c62e34d7083533c9d1ce92d8a43f5daa29af4737c0ce4696d134aea343cd6528 Nov 04 05:04:51.702788 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:51.702749 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-cedfca/distinct-2-w6-cedfca-kserve-5465b749bb-g2tkm" event={"ID":"97e3afca-28dd-4f22-89bf-14245acc4544","Type":"ContainerStarted","Data":"c62e34d7083533c9d1ce92d8a43f5daa29af4737c0ce4696d134aea343cd6528"} Nov 04 05:04:51.704537 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:51.704506 4509 generic.go:356] "Generic (PLEG): container finished" podID="0c9909c8-ffcb-4601-b38d-923c92a786d0" containerID="306b306d9b814119f5696f0ee54a385aa7988fd833d4b96f45099c1262173f48" exitCode=0 Nov 04 05:04:51.704688 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:51.704583 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80b24922-sdsg5" event={"ID":"0c9909c8-ffcb-4601-b38d-923c92a786d0","Type":"ContainerDied","Data":"306b306d9b814119f5696f0ee54a385aa7988fd833d4b96f45099c1262173f48"} Nov 04 05:04:52.712951 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:52.712916 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-cedfca/distinct-2-w6-cedfca-kserve-5465b749bb-g2tkm" event={"ID":"97e3afca-28dd-4f22-89bf-14245acc4544","Type":"ContainerStarted","Data":"ed6811d301fd1b7cb6e597ec418ba0765deb72b2154eb05e591561d7a6fc4ae1"} Nov 04 05:04:52.829189 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:52.829016 4509 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-cedfca/distinct-2-w6-cedfca-kserve-5465b749bb-g2tkm" podStartSLOduration=2.829000603 podStartE2EDuration="2.829000603s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 05:04:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 05:04:52.826878641 +0000 UTC m=+1056.704602161" watchObservedRunningTime="2026-11-04 05:04:52.829000603 +0000 UTC m=+1056.706724125" Nov 04 05:04:52.857935 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:52.857909 4509 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80b24922-sdsg5" Nov 04 05:04:53.046053 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:53.045968 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c9909c8-ffcb-4601-b38d-923c92a786d0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c9909c8-ffcb-4601-b38d-923c92a786d0-maas-api-xks-ca\") pod \"0c9909c8-ffcb-4601-b38d-923c92a786d0\" (UID: \"0c9909c8-ffcb-4601-b38d-923c92a786d0\") " Nov 04 05:04:53.046190 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:53.046064 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c9909c8-ffcb-4601-b38d-923c92a786d0-kube-api-access-tw4nt\" (UniqueName: \"kubernetes.io/projected/0c9909c8-ffcb-4601-b38d-923c92a786d0-kube-api-access-tw4nt\") pod \"0c9909c8-ffcb-4601-b38d-923c92a786d0\" (UID: \"0c9909c8-ffcb-4601-b38d-923c92a786d0\") " Nov 04 05:04:53.046190 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:53.046137 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c9909c8-ffcb-4601-b38d-923c92a786d0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c9909c8-ffcb-4601-b38d-923c92a786d0-maas-api-openshift-ca\") pod \"0c9909c8-ffcb-4601-b38d-923c92a786d0\" (UID: \"0c9909c8-ffcb-4601-b38d-923c92a786d0\") " Nov 04 05:04:53.046591 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:53.046559 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c9909c8-ffcb-4601-b38d-923c92a786d0-maas-api-openshift-ca" pod "0c9909c8-ffcb-4601-b38d-923c92a786d0" (UID: "0c9909c8-ffcb-4601-b38d-923c92a786d0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 05:04:53.048423 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:53.048399 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c9909c8-ffcb-4601-b38d-923c92a786d0-maas-api-xks-ca" pod "0c9909c8-ffcb-4601-b38d-923c92a786d0" (UID: "0c9909c8-ffcb-4601-b38d-923c92a786d0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 05:04:53.048749 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:53.048735 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c9909c8-ffcb-4601-b38d-923c92a786d0-kube-api-access-tw4nt" pod "0c9909c8-ffcb-4601-b38d-923c92a786d0" (UID: "0c9909c8-ffcb-4601-b38d-923c92a786d0"). InnerVolumeSpecName "kube-api-access-tw4nt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 05:04:53.147409 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:53.147371 4509 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tw4nt\" (UniqueName: \"kubernetes.io/projected/0c9909c8-ffcb-4601-b38d-923c92a786d0-kube-api-access-tw4nt\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:04:53.147409 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:53.147406 4509 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c9909c8-ffcb-4601-b38d-923c92a786d0-maas-api-openshift-ca\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:04:53.147616 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:53.147419 4509 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c9909c8-ffcb-4601-b38d-923c92a786d0-maas-api-xks-ca\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:04:53.195444 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:53.195411 4509 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-cedfca/distinct-2-w6-cedfca-kserve-5465b749bb-g2tkm" Nov 04 05:04:53.720981 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:53.720930 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80b24922-sdsg5" event={"ID":"0c9909c8-ffcb-4601-b38d-923c92a786d0","Type":"ContainerDied","Data":"3b018decd7744a8186517dd755f54d342c46f19ed56af35b19fc0196f8626a5b"} Nov 04 05:04:53.721392 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:53.721001 4509 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3b018decd7744a8186517dd755f54d342c46f19ed56af35b19fc0196f8626a5b" Nov 04 05:04:53.721392 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:53.720949 4509 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80b24922-sdsg5" Nov 04 05:04:54.220456 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:54.220418 4509 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-3eb31c/premium-w4-3eb31c-kserve-58bc6d6c55-kh2c4"] Nov 04 05:04:57.802517 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:57.802480 4509 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9823838-jnppq"] Nov 04 05:04:57.803002 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:57.802988 4509 cpu_manager.go:414] "containerMap: removing container" podUID="0c9909c8-ffcb-4601-b38d-923c92a786d0" containerName="revoke-keys" Nov 04 05:04:57.803063 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:57.803003 4509 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c9909c8-ffcb-4601-b38d-923c92a786d0" containerName="revoke-keys" Nov 04 05:04:57.803101 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:57.803091 4509 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c9909c8-ffcb-4601-b38d-923c92a786d0" containerName="revoke-keys" Nov 04 05:04:57.806436 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:57.806421 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9823838-jnppq" Nov 04 05:04:57.859471 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:57.859443 4509 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 05:04:57.859471 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:57.859444 4509 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 05:04:57.907995 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:57.907957 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fab7a88a-df4b-41d2-b96a-ae4938a024c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e9823838-jnppq\" (UID: \"fab7a88a-df4b-41d2-b96a-ae4938a024c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9823838-jnppq" Nov 04 05:04:57.908416 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:57.908393 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbwxn\" (UniqueName: \"kubernetes.io/projected/fab7a88a-df4b-41d2-b96a-ae4938a024c9-kube-api-access-mbwxn\") pod \"maas-api-revoke-sub-e9823838-jnppq\" (UID: \"fab7a88a-df4b-41d2-b96a-ae4938a024c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9823838-jnppq" Nov 04 05:04:57.908586 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:57.908566 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fab7a88a-df4b-41d2-b96a-ae4938a024c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e9823838-jnppq\" (UID: \"fab7a88a-df4b-41d2-b96a-ae4938a024c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9823838-jnppq" Nov 04 05:04:57.916621 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:57.916588 4509 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-krgvj\"" Nov 04 05:04:57.916851 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:57.916836 4509 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 05:04:57.921636 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:57.921242 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9823838-jnppq"] Nov 04 05:04:58.009789 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:58.009757 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbwxn\" (UniqueName: \"kubernetes.io/projected/fab7a88a-df4b-41d2-b96a-ae4938a024c9-kube-api-access-mbwxn\") pod \"maas-api-revoke-sub-e9823838-jnppq\" (UID: \"fab7a88a-df4b-41d2-b96a-ae4938a024c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9823838-jnppq" Nov 04 05:04:58.009789 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:58.009794 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fab7a88a-df4b-41d2-b96a-ae4938a024c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e9823838-jnppq\" (UID: \"fab7a88a-df4b-41d2-b96a-ae4938a024c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9823838-jnppq" Nov 04 05:04:58.010056 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:58.009881 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fab7a88a-df4b-41d2-b96a-ae4938a024c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e9823838-jnppq\" (UID: \"fab7a88a-df4b-41d2-b96a-ae4938a024c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9823838-jnppq" Nov 04 05:04:58.010507 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:58.010485 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fab7a88a-df4b-41d2-b96a-ae4938a024c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e9823838-jnppq\" (UID: \"fab7a88a-df4b-41d2-b96a-ae4938a024c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9823838-jnppq" Nov 04 05:04:58.012498 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:58.012440 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fab7a88a-df4b-41d2-b96a-ae4938a024c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e9823838-jnppq\" (UID: \"fab7a88a-df4b-41d2-b96a-ae4938a024c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9823838-jnppq" Nov 04 05:04:58.067301 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:58.067216 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbwxn\" (UniqueName: \"kubernetes.io/projected/fab7a88a-df4b-41d2-b96a-ae4938a024c9-kube-api-access-mbwxn\") pod \"maas-api-revoke-sub-e9823838-jnppq\" (UID: \"fab7a88a-df4b-41d2-b96a-ae4938a024c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9823838-jnppq" Nov 04 05:04:58.119378 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:58.119313 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9823838-jnppq" Nov 04 05:04:58.494131 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:58.494103 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9823838-jnppq"] Nov 04 05:04:58.495379 ip-10-0-133-180 kubenswrapper[4509]: W1104 05:04:58.495345 4509 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfab7a88a_df4b_41d2_b96a_ae4938a024c9.slice/crio-180838e7e4db2dc1eca796c76b0d05431e906ec58311f5b3aed21edd204eb890 WatchSource:0}: Error finding container 180838e7e4db2dc1eca796c76b0d05431e906ec58311f5b3aed21edd204eb890: Status 404 returned error can't find the container with id 180838e7e4db2dc1eca796c76b0d05431e906ec58311f5b3aed21edd204eb890 Nov 04 05:04:58.751605 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:58.751518 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9823838-jnppq" event={"ID":"fab7a88a-df4b-41d2-b96a-ae4938a024c9","Type":"ContainerStarted","Data":"180838e7e4db2dc1eca796c76b0d05431e906ec58311f5b3aed21edd204eb890"} Nov 04 05:04:59.132358 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:59.132248 4509 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-370078-dc9467d44-l4644"] Nov 04 05:04:59.233416 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:59.233378 4509 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-370078-6c55687f5f-gwlxz"] Nov 04 05:04:59.760085 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:59.760051 4509 generic.go:356] "Generic (PLEG): container finished" podID="fab7a88a-df4b-41d2-b96a-ae4938a024c9" containerID="3c6fdd633d94d7e06fef09d9acbf6cb236c9678a87c1811e3ce1d2ad34fbfd1c" exitCode=0 Nov 04 05:04:59.760269 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:04:59.760129 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9823838-jnppq" event={"ID":"fab7a88a-df4b-41d2-b96a-ae4938a024c9","Type":"ContainerDied","Data":"3c6fdd633d94d7e06fef09d9acbf6cb236c9678a87c1811e3ce1d2ad34fbfd1c"} Nov 04 05:05:00.956641 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:00.956610 4509 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9823838-jnppq" Nov 04 05:05:01.050480 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:01.050393 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fab7a88a-df4b-41d2-b96a-ae4938a024c9-kube-api-access-mbwxn\" (UniqueName: \"kubernetes.io/projected/fab7a88a-df4b-41d2-b96a-ae4938a024c9-kube-api-access-mbwxn\") pod \"fab7a88a-df4b-41d2-b96a-ae4938a024c9\" (UID: \"fab7a88a-df4b-41d2-b96a-ae4938a024c9\") " Nov 04 05:05:01.050659 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:01.050496 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fab7a88a-df4b-41d2-b96a-ae4938a024c9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fab7a88a-df4b-41d2-b96a-ae4938a024c9-maas-api-openshift-ca\") pod \"fab7a88a-df4b-41d2-b96a-ae4938a024c9\" (UID: \"fab7a88a-df4b-41d2-b96a-ae4938a024c9\") " Nov 04 05:05:01.050659 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:01.050529 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fab7a88a-df4b-41d2-b96a-ae4938a024c9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fab7a88a-df4b-41d2-b96a-ae4938a024c9-maas-api-xks-ca\") pod \"fab7a88a-df4b-41d2-b96a-ae4938a024c9\" (UID: \"fab7a88a-df4b-41d2-b96a-ae4938a024c9\") " Nov 04 05:05:01.050905 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:01.050881 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fab7a88a-df4b-41d2-b96a-ae4938a024c9-maas-api-openshift-ca" pod "fab7a88a-df4b-41d2-b96a-ae4938a024c9" (UID: "fab7a88a-df4b-41d2-b96a-ae4938a024c9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 05:05:01.052739 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:01.052709 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fab7a88a-df4b-41d2-b96a-ae4938a024c9-maas-api-xks-ca" pod "fab7a88a-df4b-41d2-b96a-ae4938a024c9" (UID: "fab7a88a-df4b-41d2-b96a-ae4938a024c9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 05:05:01.052824 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:01.052786 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fab7a88a-df4b-41d2-b96a-ae4938a024c9-kube-api-access-mbwxn" pod "fab7a88a-df4b-41d2-b96a-ae4938a024c9" (UID: "fab7a88a-df4b-41d2-b96a-ae4938a024c9"). InnerVolumeSpecName "kube-api-access-mbwxn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 05:05:01.151544 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:01.151507 4509 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fab7a88a-df4b-41d2-b96a-ae4938a024c9-maas-api-openshift-ca\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:05:01.151544 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:01.151538 4509 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fab7a88a-df4b-41d2-b96a-ae4938a024c9-maas-api-xks-ca\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:05:01.151544 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:01.151547 4509 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbwxn\" (UniqueName: \"kubernetes.io/projected/fab7a88a-df4b-41d2-b96a-ae4938a024c9-kube-api-access-mbwxn\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:05:01.195508 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:01.195469 4509 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-cedfca/distinct-2-w6-cedfca-kserve-5465b749bb-g2tkm" Nov 04 05:05:01.208776 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:01.208737 4509 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-cedfca/distinct-2-w6-cedfca-kserve-5465b749bb-g2tkm" Nov 04 05:05:01.410391 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:01.410294 4509 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-370078/premium-w2-370078-kserve-54d45b6bc-q4xfk" podUID="19bf8535-557c-4598-a6fd-e937faeee299" containerName="main" containerID="cri-o://387b3c4e818e00fb7a52728f2397e371625f2a8231ae09cd4932f44ee46bf254" gracePeriod=45 Nov 04 05:05:01.663637 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:01.663571 4509 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-370078/premium-w2-370078-kserve-54d45b6bc-q4xfk" Nov 04 05:05:01.757427 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:01.757395 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19bf8535-557c-4598-a6fd-e937faeee299-dshm\" (UniqueName: \"kubernetes.io/empty-dir/19bf8535-557c-4598-a6fd-e937faeee299-dshm\") pod \"19bf8535-557c-4598-a6fd-e937faeee299\" (UID: \"19bf8535-557c-4598-a6fd-e937faeee299\") " Nov 04 05:05:01.757637 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:01.757441 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19bf8535-557c-4598-a6fd-e937faeee299-home\" (UniqueName: \"kubernetes.io/empty-dir/19bf8535-557c-4598-a6fd-e937faeee299-home\") pod \"19bf8535-557c-4598-a6fd-e937faeee299\" (UID: \"19bf8535-557c-4598-a6fd-e937faeee299\") " Nov 04 05:05:01.757637 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:01.757521 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19bf8535-557c-4598-a6fd-e937faeee299-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19bf8535-557c-4598-a6fd-e937faeee299-model-cache\") pod \"19bf8535-557c-4598-a6fd-e937faeee299\" (UID: \"19bf8535-557c-4598-a6fd-e937faeee299\") " Nov 04 05:05:01.757637 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:01.757547 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19bf8535-557c-4598-a6fd-e937faeee299-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19bf8535-557c-4598-a6fd-e937faeee299-tmp-dir\") pod \"19bf8535-557c-4598-a6fd-e937faeee299\" (UID: \"19bf8535-557c-4598-a6fd-e937faeee299\") " Nov 04 05:05:01.757637 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:01.757608 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19bf8535-557c-4598-a6fd-e937faeee299-kube-api-access-qj7fh\" (UniqueName: \"kubernetes.io/projected/19bf8535-557c-4598-a6fd-e937faeee299-kube-api-access-qj7fh\") pod \"19bf8535-557c-4598-a6fd-e937faeee299\" (UID: \"19bf8535-557c-4598-a6fd-e937faeee299\") " Nov 04 05:05:01.757637 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:01.757635 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/19bf8535-557c-4598-a6fd-e937faeee299-tls-certs\" (UniqueName: \"kubernetes.io/secret/19bf8535-557c-4598-a6fd-e937faeee299-tls-certs\") pod \"19bf8535-557c-4598-a6fd-e937faeee299\" (UID: \"19bf8535-557c-4598-a6fd-e937faeee299\") " Nov 04 05:05:01.758168 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:01.757781 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19bf8535-557c-4598-a6fd-e937faeee299-home" pod "19bf8535-557c-4598-a6fd-e937faeee299" (UID: "19bf8535-557c-4598-a6fd-e937faeee299"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:05:01.758168 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:01.757792 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19bf8535-557c-4598-a6fd-e937faeee299-model-cache" pod "19bf8535-557c-4598-a6fd-e937faeee299" (UID: "19bf8535-557c-4598-a6fd-e937faeee299"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:05:01.758168 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:01.757847 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19bf8535-557c-4598-a6fd-e937faeee299-tmp-dir" pod "19bf8535-557c-4598-a6fd-e937faeee299" (UID: "19bf8535-557c-4598-a6fd-e937faeee299"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:05:01.758168 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:01.758133 4509 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19bf8535-557c-4598-a6fd-e937faeee299-home\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:05:01.758168 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:01.758148 4509 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19bf8535-557c-4598-a6fd-e937faeee299-model-cache\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:05:01.758168 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:01.758161 4509 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19bf8535-557c-4598-a6fd-e937faeee299-tmp-dir\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:05:01.759896 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:01.759869 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19bf8535-557c-4598-a6fd-e937faeee299-dshm" pod "19bf8535-557c-4598-a6fd-e937faeee299" (UID: "19bf8535-557c-4598-a6fd-e937faeee299"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:05:01.759982 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:01.759869 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19bf8535-557c-4598-a6fd-e937faeee299-tls-certs" pod "19bf8535-557c-4598-a6fd-e937faeee299" (UID: "19bf8535-557c-4598-a6fd-e937faeee299"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 05:05:01.759982 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:01.759874 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19bf8535-557c-4598-a6fd-e937faeee299-kube-api-access-qj7fh" pod "19bf8535-557c-4598-a6fd-e937faeee299" (UID: "19bf8535-557c-4598-a6fd-e937faeee299"). InnerVolumeSpecName "kube-api-access-qj7fh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 05:05:01.773285 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:01.773256 4509 generic.go:356] "Generic (PLEG): container finished" podID="19bf8535-557c-4598-a6fd-e937faeee299" containerID="387b3c4e818e00fb7a52728f2397e371625f2a8231ae09cd4932f44ee46bf254" exitCode=0 Nov 04 05:05:01.773427 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:01.773305 4509 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-370078/premium-w2-370078-kserve-54d45b6bc-q4xfk" Nov 04 05:05:01.773427 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:01.773352 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-370078/premium-w2-370078-kserve-54d45b6bc-q4xfk" event={"ID":"19bf8535-557c-4598-a6fd-e937faeee299","Type":"ContainerDied","Data":"387b3c4e818e00fb7a52728f2397e371625f2a8231ae09cd4932f44ee46bf254"} Nov 04 05:05:01.773427 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:01.773396 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-370078/premium-w2-370078-kserve-54d45b6bc-q4xfk" event={"ID":"19bf8535-557c-4598-a6fd-e937faeee299","Type":"ContainerDied","Data":"68569abd741a71b9fd3cb2de2d9fe5943ee8779de9cd0193e284a3d9ddb942a5"} Nov 04 05:05:01.773427 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:01.773418 4509 scope.go:122] "RemoveContainer" containerID="387b3c4e818e00fb7a52728f2397e371625f2a8231ae09cd4932f44ee46bf254" Nov 04 05:05:01.774699 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:01.774685 4509 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9823838-jnppq" Nov 04 05:05:01.774745 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:01.774683 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9823838-jnppq" event={"ID":"fab7a88a-df4b-41d2-b96a-ae4938a024c9","Type":"ContainerDied","Data":"180838e7e4db2dc1eca796c76b0d05431e906ec58311f5b3aed21edd204eb890"} Nov 04 05:05:01.774745 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:01.774739 4509 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="180838e7e4db2dc1eca796c76b0d05431e906ec58311f5b3aed21edd204eb890" Nov 04 05:05:01.787199 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:01.787179 4509 scope.go:122] "RemoveContainer" containerID="387b3c4e818e00fb7a52728f2397e371625f2a8231ae09cd4932f44ee46bf254" Nov 04 05:05:01.793357 ip-10-0-133-180 kubenswrapper[4509]: E1104 05:05:01.788547 4509 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"387b3c4e818e00fb7a52728f2397e371625f2a8231ae09cd4932f44ee46bf254\": container with ID starting with 387b3c4e818e00fb7a52728f2397e371625f2a8231ae09cd4932f44ee46bf254 not found: ID does not exist" containerID="387b3c4e818e00fb7a52728f2397e371625f2a8231ae09cd4932f44ee46bf254" Nov 04 05:05:01.793357 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:01.788584 4509 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"387b3c4e818e00fb7a52728f2397e371625f2a8231ae09cd4932f44ee46bf254"} err="failed to get container status \"387b3c4e818e00fb7a52728f2397e371625f2a8231ae09cd4932f44ee46bf254\": rpc error: code = NotFound desc = could not find container \"387b3c4e818e00fb7a52728f2397e371625f2a8231ae09cd4932f44ee46bf254\": container with ID starting with 387b3c4e818e00fb7a52728f2397e371625f2a8231ae09cd4932f44ee46bf254 not found: ID does not exist" Nov 04 05:05:01.793357 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:01.790846 4509 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-cedfca/distinct-2-w6-cedfca-kserve-5465b749bb-g2tkm" Nov 04 05:05:01.859344 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:01.859289 4509 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19bf8535-557c-4598-a6fd-e937faeee299-dshm\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:05:01.859500 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:01.859352 4509 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qj7fh\" (UniqueName: \"kubernetes.io/projected/19bf8535-557c-4598-a6fd-e937faeee299-kube-api-access-qj7fh\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:05:01.859500 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:01.859368 4509 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19bf8535-557c-4598-a6fd-e937faeee299-tls-certs\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:05:02.043001 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:02.042966 4509 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-370078/premium-w2-370078-kserve-54d45b6bc-q4xfk"] Nov 04 05:05:02.046977 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:02.046949 4509 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-370078/premium-w2-370078-kserve-54d45b6bc-q4xfk"] Nov 04 05:05:02.262292 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:02.262260 4509 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19bf8535-557c-4598-a6fd-e937faeee299" path="/var/lib/kubelet/pods/19bf8535-557c-4598-a6fd-e937faeee299/volumes" Nov 04 05:05:04.133530 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:04.133490 4509 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-370078-dc9467d44-l4644" podUID="64a2b08c-1fc3-4334-96f9-97bfc88859b2" containerName="payload-processing" containerID="cri-o://eb0fde9670adda43659fde6b48727ea0b095c807b57412b34c78854fb02799b3" gracePeriod=25 Nov 04 05:05:04.238857 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:04.237393 4509 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-370078-6c55687f5f-gwlxz" podUID="bbe04b59-da69-46ac-ba51-5b617ca5f174" containerName="payload-pre-processing" containerID="cri-o://1a3f43f3121445a046535453d28cc44f61593dee2ed5566852feabffc5869d13" gracePeriod=25 Nov 04 05:05:04.444617 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:04.444588 4509 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-370078-dc9467d44-l4644" Nov 04 05:05:04.490723 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:04.489802 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64a2b08c-1fc3-4334-96f9-97bfc88859b2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64a2b08c-1fc3-4334-96f9-97bfc88859b2-plugins-config-volume\") pod \"64a2b08c-1fc3-4334-96f9-97bfc88859b2\" (UID: \"64a2b08c-1fc3-4334-96f9-97bfc88859b2\") " Nov 04 05:05:04.490723 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:04.489893 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64a2b08c-1fc3-4334-96f9-97bfc88859b2-kube-api-access-wqscs\" (UniqueName: \"kubernetes.io/projected/64a2b08c-1fc3-4334-96f9-97bfc88859b2-kube-api-access-wqscs\") pod \"64a2b08c-1fc3-4334-96f9-97bfc88859b2\" (UID: \"64a2b08c-1fc3-4334-96f9-97bfc88859b2\") " Nov 04 05:05:04.491409 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:04.491371 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64a2b08c-1fc3-4334-96f9-97bfc88859b2-plugins-config-volume" pod "64a2b08c-1fc3-4334-96f9-97bfc88859b2" (UID: "64a2b08c-1fc3-4334-96f9-97bfc88859b2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 05:05:04.493970 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:04.493944 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64a2b08c-1fc3-4334-96f9-97bfc88859b2-kube-api-access-wqscs" pod "64a2b08c-1fc3-4334-96f9-97bfc88859b2" (UID: "64a2b08c-1fc3-4334-96f9-97bfc88859b2"). InnerVolumeSpecName "kube-api-access-wqscs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 05:05:04.531432 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:04.531410 4509 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-370078-6c55687f5f-gwlxz" Nov 04 05:05:04.591031 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:04.590995 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bbe04b59-da69-46ac-ba51-5b617ca5f174-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbe04b59-da69-46ac-ba51-5b617ca5f174-plugins-config-volume\") pod \"bbe04b59-da69-46ac-ba51-5b617ca5f174\" (UID: \"bbe04b59-da69-46ac-ba51-5b617ca5f174\") " Nov 04 05:05:04.591190 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:04.591102 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbe04b59-da69-46ac-ba51-5b617ca5f174-kube-api-access-jdwlb\" (UniqueName: \"kubernetes.io/projected/bbe04b59-da69-46ac-ba51-5b617ca5f174-kube-api-access-jdwlb\") pod \"bbe04b59-da69-46ac-ba51-5b617ca5f174\" (UID: \"bbe04b59-da69-46ac-ba51-5b617ca5f174\") " Nov 04 05:05:04.591388 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:04.591366 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbe04b59-da69-46ac-ba51-5b617ca5f174-plugins-config-volume" pod "bbe04b59-da69-46ac-ba51-5b617ca5f174" (UID: "bbe04b59-da69-46ac-ba51-5b617ca5f174"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 05:05:04.591494 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:04.591478 4509 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64a2b08c-1fc3-4334-96f9-97bfc88859b2-plugins-config-volume\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:05:04.591552 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:04.591501 4509 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqscs\" (UniqueName: \"kubernetes.io/projected/64a2b08c-1fc3-4334-96f9-97bfc88859b2-kube-api-access-wqscs\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:05:04.591552 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:04.591516 4509 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbe04b59-da69-46ac-ba51-5b617ca5f174-plugins-config-volume\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:05:04.593406 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:04.593380 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbe04b59-da69-46ac-ba51-5b617ca5f174-kube-api-access-jdwlb" pod "bbe04b59-da69-46ac-ba51-5b617ca5f174" (UID: "bbe04b59-da69-46ac-ba51-5b617ca5f174"). InnerVolumeSpecName "kube-api-access-jdwlb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 05:05:04.693193 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:04.693112 4509 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdwlb\" (UniqueName: \"kubernetes.io/projected/bbe04b59-da69-46ac-ba51-5b617ca5f174-kube-api-access-jdwlb\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:05:04.796945 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:04.796899 4509 generic.go:356] "Generic (PLEG): container finished" podID="bbe04b59-da69-46ac-ba51-5b617ca5f174" containerID="1a3f43f3121445a046535453d28cc44f61593dee2ed5566852feabffc5869d13" exitCode=0 Nov 04 05:05:04.797136 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:04.796967 4509 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-370078-6c55687f5f-gwlxz" Nov 04 05:05:04.797136 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:04.796976 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-370078-6c55687f5f-gwlxz" event={"ID":"bbe04b59-da69-46ac-ba51-5b617ca5f174","Type":"ContainerDied","Data":"1a3f43f3121445a046535453d28cc44f61593dee2ed5566852feabffc5869d13"} Nov 04 05:05:04.797136 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:04.797053 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-370078-6c55687f5f-gwlxz" event={"ID":"bbe04b59-da69-46ac-ba51-5b617ca5f174","Type":"ContainerDied","Data":"d6e797ce9e249fd1517e02bcde7b5e767e1229c2fc76610be28776dc99c88f82"} Nov 04 05:05:04.797136 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:04.797075 4509 scope.go:122] "RemoveContainer" containerID="1a3f43f3121445a046535453d28cc44f61593dee2ed5566852feabffc5869d13" Nov 04 05:05:04.798534 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:04.798513 4509 generic.go:356] "Generic (PLEG): container finished" podID="64a2b08c-1fc3-4334-96f9-97bfc88859b2" containerID="eb0fde9670adda43659fde6b48727ea0b095c807b57412b34c78854fb02799b3" exitCode=0 Nov 04 05:05:04.798635 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:04.798572 4509 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-370078-dc9467d44-l4644" Nov 04 05:05:04.798635 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:04.798592 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-370078-dc9467d44-l4644" event={"ID":"64a2b08c-1fc3-4334-96f9-97bfc88859b2","Type":"ContainerDied","Data":"eb0fde9670adda43659fde6b48727ea0b095c807b57412b34c78854fb02799b3"} Nov 04 05:05:04.798716 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:04.798647 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-370078-dc9467d44-l4644" event={"ID":"64a2b08c-1fc3-4334-96f9-97bfc88859b2","Type":"ContainerDied","Data":"4ec0a2e01f9676bb6189efb8b39b9f82ad5ff462dedfb331d19db3fc05f79b8c"} Nov 04 05:05:04.807138 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:04.807114 4509 scope.go:122] "RemoveContainer" containerID="1a3f43f3121445a046535453d28cc44f61593dee2ed5566852feabffc5869d13" Nov 04 05:05:04.807472 ip-10-0-133-180 kubenswrapper[4509]: E1104 05:05:04.807450 4509 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a3f43f3121445a046535453d28cc44f61593dee2ed5566852feabffc5869d13\": container with ID starting with 1a3f43f3121445a046535453d28cc44f61593dee2ed5566852feabffc5869d13 not found: ID does not exist" containerID="1a3f43f3121445a046535453d28cc44f61593dee2ed5566852feabffc5869d13" Nov 04 05:05:04.807561 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:04.807479 4509 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a3f43f3121445a046535453d28cc44f61593dee2ed5566852feabffc5869d13"} err="failed to get container status \"1a3f43f3121445a046535453d28cc44f61593dee2ed5566852feabffc5869d13\": rpc error: code = NotFound desc = could not find container \"1a3f43f3121445a046535453d28cc44f61593dee2ed5566852feabffc5869d13\": container with ID starting with 1a3f43f3121445a046535453d28cc44f61593dee2ed5566852feabffc5869d13 not found: ID does not exist" Nov 04 05:05:04.807561 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:04.807496 4509 scope.go:122] "RemoveContainer" containerID="eb0fde9670adda43659fde6b48727ea0b095c807b57412b34c78854fb02799b3" Nov 04 05:05:04.819278 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:04.819252 4509 scope.go:122] "RemoveContainer" containerID="eb0fde9670adda43659fde6b48727ea0b095c807b57412b34c78854fb02799b3" Nov 04 05:05:04.819802 ip-10-0-133-180 kubenswrapper[4509]: E1104 05:05:04.819778 4509 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb0fde9670adda43659fde6b48727ea0b095c807b57412b34c78854fb02799b3\": container with ID starting with eb0fde9670adda43659fde6b48727ea0b095c807b57412b34c78854fb02799b3 not found: ID does not exist" containerID="eb0fde9670adda43659fde6b48727ea0b095c807b57412b34c78854fb02799b3" Nov 04 05:05:04.819892 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:04.819811 4509 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb0fde9670adda43659fde6b48727ea0b095c807b57412b34c78854fb02799b3"} err="failed to get container status \"eb0fde9670adda43659fde6b48727ea0b095c807b57412b34c78854fb02799b3\": rpc error: code = NotFound desc = could not find container \"eb0fde9670adda43659fde6b48727ea0b095c807b57412b34c78854fb02799b3\": container with ID starting with eb0fde9670adda43659fde6b48727ea0b095c807b57412b34c78854fb02799b3 not found: ID does not exist" Nov 04 05:05:04.966296 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:04.966265 4509 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-370078-6c55687f5f-gwlxz"] Nov 04 05:05:04.968729 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:04.968703 4509 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-370078-6c55687f5f-gwlxz"] Nov 04 05:05:05.074857 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:05.074821 4509 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-370078-dc9467d44-l4644"] Nov 04 05:05:05.078420 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:05.078397 4509 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-370078-dc9467d44-l4644"] Nov 04 05:05:06.146196 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:06.146169 4509 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b12986e-h6fr5"] Nov 04 05:05:06.146643 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:06.146628 4509 cpu_manager.go:414] "containerMap: removing container" podUID="fab7a88a-df4b-41d2-b96a-ae4938a024c9" containerName="revoke-keys" Nov 04 05:05:06.146687 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:06.146644 4509 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fab7a88a-df4b-41d2-b96a-ae4938a024c9" containerName="revoke-keys" Nov 04 05:05:06.146687 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:06.146658 4509 cpu_manager.go:414] "containerMap: removing container" podUID="19bf8535-557c-4598-a6fd-e937faeee299" containerName="main" Nov 04 05:05:06.146687 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:06.146663 4509 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19bf8535-557c-4598-a6fd-e937faeee299" containerName="main" Nov 04 05:05:06.146687 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:06.146687 4509 cpu_manager.go:414] "containerMap: removing container" podUID="bbe04b59-da69-46ac-ba51-5b617ca5f174" containerName="payload-pre-processing" Nov 04 05:05:06.146807 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:06.146692 4509 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbe04b59-da69-46ac-ba51-5b617ca5f174" containerName="payload-pre-processing" Nov 04 05:05:06.146807 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:06.146700 4509 cpu_manager.go:414] "containerMap: removing container" podUID="64a2b08c-1fc3-4334-96f9-97bfc88859b2" containerName="payload-processing" Nov 04 05:05:06.146807 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:06.146705 4509 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64a2b08c-1fc3-4334-96f9-97bfc88859b2" containerName="payload-processing" Nov 04 05:05:06.146807 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:06.146774 4509 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbe04b59-da69-46ac-ba51-5b617ca5f174" containerName="payload-pre-processing" Nov 04 05:05:06.146807 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:06.146781 4509 memory_manager.go:365] "RemoveStaleState removing state" podUID="64a2b08c-1fc3-4334-96f9-97bfc88859b2" containerName="payload-processing" Nov 04 05:05:06.146807 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:06.146790 4509 memory_manager.go:365] "RemoveStaleState removing state" podUID="fab7a88a-df4b-41d2-b96a-ae4938a024c9" containerName="revoke-keys" Nov 04 05:05:06.146807 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:06.146798 4509 memory_manager.go:365] "RemoveStaleState removing state" podUID="19bf8535-557c-4598-a6fd-e937faeee299" containerName="main" Nov 04 05:05:06.151394 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:06.151377 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b12986e-h6fr5" Nov 04 05:05:06.205785 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:06.205753 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b12986e-h6fr5"] Nov 04 05:05:06.208594 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:06.208564 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7w5t\" (UniqueName: \"kubernetes.io/projected/950a4971-bcd5-43b7-a2c6-7545c36a63ea-kube-api-access-v7w5t\") pod \"maas-api-revoke-sub-3b12986e-h6fr5\" (UID: \"950a4971-bcd5-43b7-a2c6-7545c36a63ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b12986e-h6fr5" Nov 04 05:05:06.208736 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:06.208628 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/950a4971-bcd5-43b7-a2c6-7545c36a63ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3b12986e-h6fr5\" (UID: \"950a4971-bcd5-43b7-a2c6-7545c36a63ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b12986e-h6fr5" Nov 04 05:05:06.208782 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:06.208752 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/950a4971-bcd5-43b7-a2c6-7545c36a63ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3b12986e-h6fr5\" (UID: \"950a4971-bcd5-43b7-a2c6-7545c36a63ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b12986e-h6fr5" Nov 04 05:05:06.252620 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:06.252587 4509 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 05:05:06.252620 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:06.252591 4509 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 05:05:06.252620 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:06.252587 4509 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 05:05:06.252855 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:06.252649 4509 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-krgvj\"" Nov 04 05:05:06.261363 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:06.261308 4509 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64a2b08c-1fc3-4334-96f9-97bfc88859b2" path="/var/lib/kubelet/pods/64a2b08c-1fc3-4334-96f9-97bfc88859b2/volumes" Nov 04 05:05:06.261806 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:06.261792 4509 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bbe04b59-da69-46ac-ba51-5b617ca5f174" path="/var/lib/kubelet/pods/bbe04b59-da69-46ac-ba51-5b617ca5f174/volumes" Nov 04 05:05:06.309810 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:06.309776 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/950a4971-bcd5-43b7-a2c6-7545c36a63ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3b12986e-h6fr5\" (UID: \"950a4971-bcd5-43b7-a2c6-7545c36a63ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b12986e-h6fr5" Nov 04 05:05:06.309975 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:06.309857 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7w5t\" (UniqueName: \"kubernetes.io/projected/950a4971-bcd5-43b7-a2c6-7545c36a63ea-kube-api-access-v7w5t\") pod \"maas-api-revoke-sub-3b12986e-h6fr5\" (UID: \"950a4971-bcd5-43b7-a2c6-7545c36a63ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b12986e-h6fr5" Nov 04 05:05:06.309975 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:06.309906 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/950a4971-bcd5-43b7-a2c6-7545c36a63ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3b12986e-h6fr5\" (UID: \"950a4971-bcd5-43b7-a2c6-7545c36a63ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b12986e-h6fr5" Nov 04 05:05:06.310672 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:06.310597 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/950a4971-bcd5-43b7-a2c6-7545c36a63ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3b12986e-h6fr5\" (UID: \"950a4971-bcd5-43b7-a2c6-7545c36a63ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b12986e-h6fr5" Nov 04 05:05:06.312335 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:06.312299 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/950a4971-bcd5-43b7-a2c6-7545c36a63ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3b12986e-h6fr5\" (UID: \"950a4971-bcd5-43b7-a2c6-7545c36a63ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b12986e-h6fr5" Nov 04 05:05:06.380898 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:06.380860 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7w5t\" (UniqueName: \"kubernetes.io/projected/950a4971-bcd5-43b7-a2c6-7545c36a63ea-kube-api-access-v7w5t\") pod \"maas-api-revoke-sub-3b12986e-h6fr5\" (UID: \"950a4971-bcd5-43b7-a2c6-7545c36a63ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b12986e-h6fr5" Nov 04 05:05:06.463862 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:06.463830 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b12986e-h6fr5" Nov 04 05:05:06.841639 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:06.841611 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b12986e-h6fr5"] Nov 04 05:05:06.842687 ip-10-0-133-180 kubenswrapper[4509]: W1104 05:05:06.842662 4509 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod950a4971_bcd5_43b7_a2c6_7545c36a63ea.slice/crio-6fbacedb17bbc2c1fa1c5b24e3410a5f3cdf2880c2f1d90ea3865a000831f383 WatchSource:0}: Error finding container 6fbacedb17bbc2c1fa1c5b24e3410a5f3cdf2880c2f1d90ea3865a000831f383: Status 404 returned error can't find the container with id 6fbacedb17bbc2c1fa1c5b24e3410a5f3cdf2880c2f1d90ea3865a000831f383 Nov 04 05:05:07.227385 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:07.227348 4509 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-3eb31c-589ff8b4b8-v89hw"] Nov 04 05:05:07.331125 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:07.331090 4509 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-3eb31c-5bc57d7f87-xfpgv"] Nov 04 05:05:07.619128 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:07.619091 4509 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-dab70d-openshift-default-6496f5466c-xczrw"] Nov 04 05:05:07.622886 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:07.622859 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-dab70d-openshift-default-6496f5466c-xczrw" Nov 04 05:05:07.694292 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:07.693850 4509 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-dab70d-openshift-default-dockercfg-852vc\"" Nov 04 05:05:07.722192 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:07.721398 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-dab70d-openshift-default-6496f5466c-xczrw"] Nov 04 05:05:07.724937 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:07.724805 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1664ef62-abe8-47ed-b2b0-bb5d787c3e22-istio-token\") pod \"e2e-worker-w3-dab70d-openshift-default-6496f5466c-xczrw\" (UID: \"1664ef62-abe8-47ed-b2b0-bb5d787c3e22\") " pod="openshift-ingress/e2e-worker-w3-dab70d-openshift-default-6496f5466c-xczrw" Nov 04 05:05:07.725071 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:07.724972 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wm77f\" (UniqueName: \"kubernetes.io/projected/1664ef62-abe8-47ed-b2b0-bb5d787c3e22-kube-api-access-wm77f\") pod \"e2e-worker-w3-dab70d-openshift-default-6496f5466c-xczrw\" (UID: \"1664ef62-abe8-47ed-b2b0-bb5d787c3e22\") " pod="openshift-ingress/e2e-worker-w3-dab70d-openshift-default-6496f5466c-xczrw" Nov 04 05:05:07.725071 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:07.725037 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1664ef62-abe8-47ed-b2b0-bb5d787c3e22-istio-podinfo\") pod \"e2e-worker-w3-dab70d-openshift-default-6496f5466c-xczrw\" (UID: \"1664ef62-abe8-47ed-b2b0-bb5d787c3e22\") " pod="openshift-ingress/e2e-worker-w3-dab70d-openshift-default-6496f5466c-xczrw" Nov 04 05:05:07.725206 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:07.725085 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1664ef62-abe8-47ed-b2b0-bb5d787c3e22-workload-certs\") pod \"e2e-worker-w3-dab70d-openshift-default-6496f5466c-xczrw\" (UID: \"1664ef62-abe8-47ed-b2b0-bb5d787c3e22\") " pod="openshift-ingress/e2e-worker-w3-dab70d-openshift-default-6496f5466c-xczrw" Nov 04 05:05:07.725206 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:07.725114 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1664ef62-abe8-47ed-b2b0-bb5d787c3e22-istio-data\") pod \"e2e-worker-w3-dab70d-openshift-default-6496f5466c-xczrw\" (UID: \"1664ef62-abe8-47ed-b2b0-bb5d787c3e22\") " pod="openshift-ingress/e2e-worker-w3-dab70d-openshift-default-6496f5466c-xczrw" Nov 04 05:05:07.725206 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:07.725154 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1664ef62-abe8-47ed-b2b0-bb5d787c3e22-istiod-ca-cert\") pod \"e2e-worker-w3-dab70d-openshift-default-6496f5466c-xczrw\" (UID: \"1664ef62-abe8-47ed-b2b0-bb5d787c3e22\") " pod="openshift-ingress/e2e-worker-w3-dab70d-openshift-default-6496f5466c-xczrw" Nov 04 05:05:07.725398 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:07.725315 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1664ef62-abe8-47ed-b2b0-bb5d787c3e22-credential-socket\") pod \"e2e-worker-w3-dab70d-openshift-default-6496f5466c-xczrw\" (UID: \"1664ef62-abe8-47ed-b2b0-bb5d787c3e22\") " pod="openshift-ingress/e2e-worker-w3-dab70d-openshift-default-6496f5466c-xczrw" Nov 04 05:05:07.725398 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:07.725383 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1664ef62-abe8-47ed-b2b0-bb5d787c3e22-istio-envoy\") pod \"e2e-worker-w3-dab70d-openshift-default-6496f5466c-xczrw\" (UID: \"1664ef62-abe8-47ed-b2b0-bb5d787c3e22\") " pod="openshift-ingress/e2e-worker-w3-dab70d-openshift-default-6496f5466c-xczrw" Nov 04 05:05:07.725514 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:07.725436 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1664ef62-abe8-47ed-b2b0-bb5d787c3e22-workload-socket\") pod \"e2e-worker-w3-dab70d-openshift-default-6496f5466c-xczrw\" (UID: \"1664ef62-abe8-47ed-b2b0-bb5d787c3e22\") " pod="openshift-ingress/e2e-worker-w3-dab70d-openshift-default-6496f5466c-xczrw" Nov 04 05:05:07.827853 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:07.827816 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1664ef62-abe8-47ed-b2b0-bb5d787c3e22-istiod-ca-cert\") pod \"e2e-worker-w3-dab70d-openshift-default-6496f5466c-xczrw\" (UID: \"1664ef62-abe8-47ed-b2b0-bb5d787c3e22\") " pod="openshift-ingress/e2e-worker-w3-dab70d-openshift-default-6496f5466c-xczrw" Nov 04 05:05:07.828057 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:07.827876 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1664ef62-abe8-47ed-b2b0-bb5d787c3e22-credential-socket\") pod \"e2e-worker-w3-dab70d-openshift-default-6496f5466c-xczrw\" (UID: \"1664ef62-abe8-47ed-b2b0-bb5d787c3e22\") " pod="openshift-ingress/e2e-worker-w3-dab70d-openshift-default-6496f5466c-xczrw" Nov 04 05:05:07.828057 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:07.827919 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1664ef62-abe8-47ed-b2b0-bb5d787c3e22-istio-envoy\") pod \"e2e-worker-w3-dab70d-openshift-default-6496f5466c-xczrw\" (UID: \"1664ef62-abe8-47ed-b2b0-bb5d787c3e22\") " pod="openshift-ingress/e2e-worker-w3-dab70d-openshift-default-6496f5466c-xczrw" Nov 04 05:05:07.828057 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:07.827964 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1664ef62-abe8-47ed-b2b0-bb5d787c3e22-workload-socket\") pod \"e2e-worker-w3-dab70d-openshift-default-6496f5466c-xczrw\" (UID: \"1664ef62-abe8-47ed-b2b0-bb5d787c3e22\") " pod="openshift-ingress/e2e-worker-w3-dab70d-openshift-default-6496f5466c-xczrw" Nov 04 05:05:07.828057 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:07.828043 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1664ef62-abe8-47ed-b2b0-bb5d787c3e22-istio-token\") pod \"e2e-worker-w3-dab70d-openshift-default-6496f5466c-xczrw\" (UID: \"1664ef62-abe8-47ed-b2b0-bb5d787c3e22\") " pod="openshift-ingress/e2e-worker-w3-dab70d-openshift-default-6496f5466c-xczrw" Nov 04 05:05:07.828280 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:07.828087 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wm77f\" (UniqueName: \"kubernetes.io/projected/1664ef62-abe8-47ed-b2b0-bb5d787c3e22-kube-api-access-wm77f\") pod \"e2e-worker-w3-dab70d-openshift-default-6496f5466c-xczrw\" (UID: \"1664ef62-abe8-47ed-b2b0-bb5d787c3e22\") " pod="openshift-ingress/e2e-worker-w3-dab70d-openshift-default-6496f5466c-xczrw" Nov 04 05:05:07.828280 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:07.828152 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1664ef62-abe8-47ed-b2b0-bb5d787c3e22-istio-podinfo\") pod \"e2e-worker-w3-dab70d-openshift-default-6496f5466c-xczrw\" (UID: \"1664ef62-abe8-47ed-b2b0-bb5d787c3e22\") " pod="openshift-ingress/e2e-worker-w3-dab70d-openshift-default-6496f5466c-xczrw" Nov 04 05:05:07.828280 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:07.828197 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1664ef62-abe8-47ed-b2b0-bb5d787c3e22-workload-certs\") pod \"e2e-worker-w3-dab70d-openshift-default-6496f5466c-xczrw\" (UID: \"1664ef62-abe8-47ed-b2b0-bb5d787c3e22\") " pod="openshift-ingress/e2e-worker-w3-dab70d-openshift-default-6496f5466c-xczrw" Nov 04 05:05:07.828280 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:07.828221 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1664ef62-abe8-47ed-b2b0-bb5d787c3e22-istio-data\") pod \"e2e-worker-w3-dab70d-openshift-default-6496f5466c-xczrw\" (UID: \"1664ef62-abe8-47ed-b2b0-bb5d787c3e22\") " pod="openshift-ingress/e2e-worker-w3-dab70d-openshift-default-6496f5466c-xczrw" Nov 04 05:05:07.828705 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:07.828680 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1664ef62-abe8-47ed-b2b0-bb5d787c3e22-istiod-ca-cert\") pod \"e2e-worker-w3-dab70d-openshift-default-6496f5466c-xczrw\" (UID: \"1664ef62-abe8-47ed-b2b0-bb5d787c3e22\") " pod="openshift-ingress/e2e-worker-w3-dab70d-openshift-default-6496f5466c-xczrw" Nov 04 05:05:07.828811 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:07.828790 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1664ef62-abe8-47ed-b2b0-bb5d787c3e22-istio-data\") pod \"e2e-worker-w3-dab70d-openshift-default-6496f5466c-xczrw\" (UID: \"1664ef62-abe8-47ed-b2b0-bb5d787c3e22\") " pod="openshift-ingress/e2e-worker-w3-dab70d-openshift-default-6496f5466c-xczrw" Nov 04 05:05:07.829213 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:07.829188 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1664ef62-abe8-47ed-b2b0-bb5d787c3e22-credential-socket\") pod \"e2e-worker-w3-dab70d-openshift-default-6496f5466c-xczrw\" (UID: \"1664ef62-abe8-47ed-b2b0-bb5d787c3e22\") " pod="openshift-ingress/e2e-worker-w3-dab70d-openshift-default-6496f5466c-xczrw" Nov 04 05:05:07.829876 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:07.829849 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1664ef62-abe8-47ed-b2b0-bb5d787c3e22-workload-socket\") pod \"e2e-worker-w3-dab70d-openshift-default-6496f5466c-xczrw\" (UID: \"1664ef62-abe8-47ed-b2b0-bb5d787c3e22\") " pod="openshift-ingress/e2e-worker-w3-dab70d-openshift-default-6496f5466c-xczrw" Nov 04 05:05:07.829983 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:07.829886 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1664ef62-abe8-47ed-b2b0-bb5d787c3e22-workload-certs\") pod \"e2e-worker-w3-dab70d-openshift-default-6496f5466c-xczrw\" (UID: \"1664ef62-abe8-47ed-b2b0-bb5d787c3e22\") " pod="openshift-ingress/e2e-worker-w3-dab70d-openshift-default-6496f5466c-xczrw" Nov 04 05:05:07.833189 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:07.833170 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1664ef62-abe8-47ed-b2b0-bb5d787c3e22-istio-podinfo\") pod \"e2e-worker-w3-dab70d-openshift-default-6496f5466c-xczrw\" (UID: \"1664ef62-abe8-47ed-b2b0-bb5d787c3e22\") " pod="openshift-ingress/e2e-worker-w3-dab70d-openshift-default-6496f5466c-xczrw" Nov 04 05:05:07.833770 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:07.833744 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1664ef62-abe8-47ed-b2b0-bb5d787c3e22-istio-envoy\") pod \"e2e-worker-w3-dab70d-openshift-default-6496f5466c-xczrw\" (UID: \"1664ef62-abe8-47ed-b2b0-bb5d787c3e22\") " pod="openshift-ingress/e2e-worker-w3-dab70d-openshift-default-6496f5466c-xczrw" Nov 04 05:05:07.834822 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:07.834800 4509 generic.go:356] "Generic (PLEG): container finished" podID="950a4971-bcd5-43b7-a2c6-7545c36a63ea" containerID="754c5488adafe136991f0b57e1b6ad9d7e7c05c6c83ee49ca5a6fbc1ada5b9c1" exitCode=0 Nov 04 05:05:07.834916 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:07.834864 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b12986e-h6fr5" event={"ID":"950a4971-bcd5-43b7-a2c6-7545c36a63ea","Type":"ContainerDied","Data":"754c5488adafe136991f0b57e1b6ad9d7e7c05c6c83ee49ca5a6fbc1ada5b9c1"} Nov 04 05:05:07.834976 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:07.834942 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b12986e-h6fr5" event={"ID":"950a4971-bcd5-43b7-a2c6-7545c36a63ea","Type":"ContainerStarted","Data":"6fbacedb17bbc2c1fa1c5b24e3410a5f3cdf2880c2f1d90ea3865a000831f383"} Nov 04 05:05:07.885599 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:07.885513 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wm77f\" (UniqueName: \"kubernetes.io/projected/1664ef62-abe8-47ed-b2b0-bb5d787c3e22-kube-api-access-wm77f\") pod \"e2e-worker-w3-dab70d-openshift-default-6496f5466c-xczrw\" (UID: \"1664ef62-abe8-47ed-b2b0-bb5d787c3e22\") " pod="openshift-ingress/e2e-worker-w3-dab70d-openshift-default-6496f5466c-xczrw" Nov 04 05:05:07.885599 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:07.885522 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1664ef62-abe8-47ed-b2b0-bb5d787c3e22-istio-token\") pod \"e2e-worker-w3-dab70d-openshift-default-6496f5466c-xczrw\" (UID: \"1664ef62-abe8-47ed-b2b0-bb5d787c3e22\") " pod="openshift-ingress/e2e-worker-w3-dab70d-openshift-default-6496f5466c-xczrw" Nov 04 05:05:07.973106 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:07.973066 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-dab70d-openshift-default-6496f5466c-xczrw" Nov 04 05:05:08.403498 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:08.403462 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-dab70d-openshift-default-6496f5466c-xczrw"] Nov 04 05:05:08.404869 ip-10-0-133-180 kubenswrapper[4509]: W1104 05:05:08.404835 4509 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1664ef62_abe8_47ed_b2b0_bb5d787c3e22.slice/crio-b3f52460a7bc0be84f90bea9a4f85b314e5d50140fc15bd4c1195430c1c47820 WatchSource:0}: Error finding container b3f52460a7bc0be84f90bea9a4f85b314e5d50140fc15bd4c1195430c1c47820: Status 404 returned error can't find the container with id b3f52460a7bc0be84f90bea9a4f85b314e5d50140fc15bd4c1195430c1c47820 Nov 04 05:05:08.407306 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:08.407272 4509 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 04 05:05:08.407407 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:08.407364 4509 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 04 05:05:08.407447 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:08.407412 4509 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 04 05:05:08.869344 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:08.867555 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-dab70d-openshift-default-6496f5466c-xczrw" event={"ID":"1664ef62-abe8-47ed-b2b0-bb5d787c3e22","Type":"ContainerStarted","Data":"d5c421a16fa693589cfe0881efe529aea7fc45e4fccfb0cd77640ad2f2e2787b"} Nov 04 05:05:08.869344 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:08.867639 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-dab70d-openshift-default-6496f5466c-xczrw" event={"ID":"1664ef62-abe8-47ed-b2b0-bb5d787c3e22","Type":"ContainerStarted","Data":"b3f52460a7bc0be84f90bea9a4f85b314e5d50140fc15bd4c1195430c1c47820"} Nov 04 05:05:09.006914 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:09.004549 4509 patch_prober.go:29] interesting pod/e2e-worker-w3-dab70d-openshift-default-6496f5466c-xczrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Nov 04 05:05:09.006914 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:09.004611 4509 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dab70d-openshift-default-6496f5466c-xczrw" podUID="1664ef62-abe8-47ed-b2b0-bb5d787c3e22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Nov 04 05:05:09.006914 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:09.004870 4509 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-dab70d-openshift-default-6496f5466c-xczrw" Nov 04 05:05:09.142207 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:09.136812 4509 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b12986e-h6fr5" Nov 04 05:05:09.242091 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:09.242050 4509 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-3eb31c/premium-w4-3eb31c-kserve-58bc6d6c55-kh2c4" podUID="9181aed8-6d05-4691-be54-f26391920734" containerName="main" containerID="cri-o://8d31357fb32fc0a54b637018306887fc1853f6f1c34474340f5b1b806231e998" gracePeriod=45 Nov 04 05:05:09.254215 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:09.254152 4509 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-dab70d-openshift-default-6496f5466c-xczrw" podStartSLOduration=2.254130795 podStartE2EDuration="2.254130795s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 05:05:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 05:05:09.002256301 +0000 UTC m=+1072.879979835" watchObservedRunningTime="2026-11-04 05:05:09.254130795 +0000 UTC m=+1073.131854319" Nov 04 05:05:09.298223 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:09.298186 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/950a4971-bcd5-43b7-a2c6-7545c36a63ea-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/950a4971-bcd5-43b7-a2c6-7545c36a63ea-maas-api-xks-ca\") pod \"950a4971-bcd5-43b7-a2c6-7545c36a63ea\" (UID: \"950a4971-bcd5-43b7-a2c6-7545c36a63ea\") " Nov 04 05:05:09.298428 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:09.298269 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/950a4971-bcd5-43b7-a2c6-7545c36a63ea-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/950a4971-bcd5-43b7-a2c6-7545c36a63ea-maas-api-openshift-ca\") pod \"950a4971-bcd5-43b7-a2c6-7545c36a63ea\" (UID: \"950a4971-bcd5-43b7-a2c6-7545c36a63ea\") " Nov 04 05:05:09.298503 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:09.298487 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/950a4971-bcd5-43b7-a2c6-7545c36a63ea-kube-api-access-v7w5t\" (UniqueName: \"kubernetes.io/projected/950a4971-bcd5-43b7-a2c6-7545c36a63ea-kube-api-access-v7w5t\") pod \"950a4971-bcd5-43b7-a2c6-7545c36a63ea\" (UID: \"950a4971-bcd5-43b7-a2c6-7545c36a63ea\") " Nov 04 05:05:09.298762 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:09.298717 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/950a4971-bcd5-43b7-a2c6-7545c36a63ea-maas-api-openshift-ca" pod "950a4971-bcd5-43b7-a2c6-7545c36a63ea" (UID: "950a4971-bcd5-43b7-a2c6-7545c36a63ea"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 05:05:09.298979 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:09.298957 4509 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/950a4971-bcd5-43b7-a2c6-7545c36a63ea-maas-api-openshift-ca\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:05:09.300775 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:09.300751 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/950a4971-bcd5-43b7-a2c6-7545c36a63ea-maas-api-xks-ca" pod "950a4971-bcd5-43b7-a2c6-7545c36a63ea" (UID: "950a4971-bcd5-43b7-a2c6-7545c36a63ea"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 05:05:09.300775 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:09.300773 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/950a4971-bcd5-43b7-a2c6-7545c36a63ea-kube-api-access-v7w5t" pod "950a4971-bcd5-43b7-a2c6-7545c36a63ea" (UID: "950a4971-bcd5-43b7-a2c6-7545c36a63ea"). InnerVolumeSpecName "kube-api-access-v7w5t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 05:05:09.400638 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:09.400537 4509 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7w5t\" (UniqueName: \"kubernetes.io/projected/950a4971-bcd5-43b7-a2c6-7545c36a63ea-kube-api-access-v7w5t\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:05:09.400638 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:09.400580 4509 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/950a4971-bcd5-43b7-a2c6-7545c36a63ea-maas-api-xks-ca\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:05:09.626119 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:09.626088 4509 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3eb31c/premium-w4-3eb31c-kserve-58bc6d6c55-kh2c4" Nov 04 05:05:09.807042 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:09.807000 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9181aed8-6d05-4691-be54-f26391920734-kube-api-access-n5m6t\" (UniqueName: \"kubernetes.io/projected/9181aed8-6d05-4691-be54-f26391920734-kube-api-access-n5m6t\") pod \"9181aed8-6d05-4691-be54-f26391920734\" (UID: \"9181aed8-6d05-4691-be54-f26391920734\") " Nov 04 05:05:09.808313 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:09.807140 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9181aed8-6d05-4691-be54-f26391920734-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9181aed8-6d05-4691-be54-f26391920734-dshm\") pod \"9181aed8-6d05-4691-be54-f26391920734\" (UID: \"9181aed8-6d05-4691-be54-f26391920734\") " Nov 04 05:05:09.808313 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:09.807174 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9181aed8-6d05-4691-be54-f26391920734-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9181aed8-6d05-4691-be54-f26391920734-model-cache\") pod \"9181aed8-6d05-4691-be54-f26391920734\" (UID: \"9181aed8-6d05-4691-be54-f26391920734\") " Nov 04 05:05:09.808313 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:09.807214 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9181aed8-6d05-4691-be54-f26391920734-tls-certs\" (UniqueName: \"kubernetes.io/secret/9181aed8-6d05-4691-be54-f26391920734-tls-certs\") pod \"9181aed8-6d05-4691-be54-f26391920734\" (UID: \"9181aed8-6d05-4691-be54-f26391920734\") " Nov 04 05:05:09.808313 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:09.807351 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9181aed8-6d05-4691-be54-f26391920734-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9181aed8-6d05-4691-be54-f26391920734-tmp-dir\") pod \"9181aed8-6d05-4691-be54-f26391920734\" (UID: \"9181aed8-6d05-4691-be54-f26391920734\") " Nov 04 05:05:09.808313 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:09.807393 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9181aed8-6d05-4691-be54-f26391920734-home\" (UniqueName: \"kubernetes.io/empty-dir/9181aed8-6d05-4691-be54-f26391920734-home\") pod \"9181aed8-6d05-4691-be54-f26391920734\" (UID: \"9181aed8-6d05-4691-be54-f26391920734\") " Nov 04 05:05:09.808313 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:09.808183 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9181aed8-6d05-4691-be54-f26391920734-home" pod "9181aed8-6d05-4691-be54-f26391920734" (UID: "9181aed8-6d05-4691-be54-f26391920734"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:05:09.808789 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:09.808761 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9181aed8-6d05-4691-be54-f26391920734-model-cache" pod "9181aed8-6d05-4691-be54-f26391920734" (UID: "9181aed8-6d05-4691-be54-f26391920734"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:05:09.809300 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:09.809275 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9181aed8-6d05-4691-be54-f26391920734-tmp-dir" pod "9181aed8-6d05-4691-be54-f26391920734" (UID: "9181aed8-6d05-4691-be54-f26391920734"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:05:09.815505 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:09.815158 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9181aed8-6d05-4691-be54-f26391920734-kube-api-access-n5m6t" pod "9181aed8-6d05-4691-be54-f26391920734" (UID: "9181aed8-6d05-4691-be54-f26391920734"). InnerVolumeSpecName "kube-api-access-n5m6t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 05:05:09.817305 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:09.817264 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9181aed8-6d05-4691-be54-f26391920734-dshm" pod "9181aed8-6d05-4691-be54-f26391920734" (UID: "9181aed8-6d05-4691-be54-f26391920734"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:05:09.818249 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:09.818228 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9181aed8-6d05-4691-be54-f26391920734-tls-certs" pod "9181aed8-6d05-4691-be54-f26391920734" (UID: "9181aed8-6d05-4691-be54-f26391920734"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 05:05:09.905452 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:09.905414 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b12986e-h6fr5" event={"ID":"950a4971-bcd5-43b7-a2c6-7545c36a63ea","Type":"ContainerDied","Data":"6fbacedb17bbc2c1fa1c5b24e3410a5f3cdf2880c2f1d90ea3865a000831f383"} Nov 04 05:05:09.905673 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:09.905476 4509 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6fbacedb17bbc2c1fa1c5b24e3410a5f3cdf2880c2f1d90ea3865a000831f383" Nov 04 05:05:09.905673 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:09.905551 4509 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b12986e-h6fr5" Nov 04 05:05:09.909112 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:09.908993 4509 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9181aed8-6d05-4691-be54-f26391920734-dshm\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:05:09.909112 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:09.909025 4509 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9181aed8-6d05-4691-be54-f26391920734-model-cache\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:05:09.909112 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:09.909042 4509 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9181aed8-6d05-4691-be54-f26391920734-tls-certs\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:05:09.909112 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:09.909057 4509 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9181aed8-6d05-4691-be54-f26391920734-tmp-dir\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:05:09.909112 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:09.909070 4509 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9181aed8-6d05-4691-be54-f26391920734-home\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:05:09.909112 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:09.909086 4509 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5m6t\" (UniqueName: \"kubernetes.io/projected/9181aed8-6d05-4691-be54-f26391920734-kube-api-access-n5m6t\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:05:09.914216 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:09.910067 4509 generic.go:356] "Generic (PLEG): container finished" podID="9181aed8-6d05-4691-be54-f26391920734" containerID="8d31357fb32fc0a54b637018306887fc1853f6f1c34474340f5b1b806231e998" exitCode=0 Nov 04 05:05:09.914216 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:09.910834 4509 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3eb31c/premium-w4-3eb31c-kserve-58bc6d6c55-kh2c4" Nov 04 05:05:09.914216 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:09.913684 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3eb31c/premium-w4-3eb31c-kserve-58bc6d6c55-kh2c4" event={"ID":"9181aed8-6d05-4691-be54-f26391920734","Type":"ContainerDied","Data":"8d31357fb32fc0a54b637018306887fc1853f6f1c34474340f5b1b806231e998"} Nov 04 05:05:09.914216 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:09.913733 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3eb31c/premium-w4-3eb31c-kserve-58bc6d6c55-kh2c4" event={"ID":"9181aed8-6d05-4691-be54-f26391920734","Type":"ContainerDied","Data":"e9fb1b1ad8cf42de91d29df19a0317eee7bc712514d546dd66d0e73a47a48be5"} Nov 04 05:05:09.914216 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:09.913758 4509 scope.go:122] "RemoveContainer" containerID="8d31357fb32fc0a54b637018306887fc1853f6f1c34474340f5b1b806231e998" Nov 04 05:05:09.928601 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:09.928575 4509 scope.go:122] "RemoveContainer" containerID="8d31357fb32fc0a54b637018306887fc1853f6f1c34474340f5b1b806231e998" Nov 04 05:05:09.929014 ip-10-0-133-180 kubenswrapper[4509]: E1104 05:05:09.928992 4509 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d31357fb32fc0a54b637018306887fc1853f6f1c34474340f5b1b806231e998\": container with ID starting with 8d31357fb32fc0a54b637018306887fc1853f6f1c34474340f5b1b806231e998 not found: ID does not exist" containerID="8d31357fb32fc0a54b637018306887fc1853f6f1c34474340f5b1b806231e998" Nov 04 05:05:09.929098 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:09.929023 4509 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d31357fb32fc0a54b637018306887fc1853f6f1c34474340f5b1b806231e998"} err="failed to get container status \"8d31357fb32fc0a54b637018306887fc1853f6f1c34474340f5b1b806231e998\": rpc error: code = NotFound desc = could not find container \"8d31357fb32fc0a54b637018306887fc1853f6f1c34474340f5b1b806231e998\": container with ID starting with 8d31357fb32fc0a54b637018306887fc1853f6f1c34474340f5b1b806231e998 not found: ID does not exist" Nov 04 05:05:09.978365 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:09.976351 4509 patch_prober.go:29] interesting pod/e2e-worker-w3-dab70d-openshift-default-6496f5466c-xczrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Nov 04 05:05:09.978365 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:09.976426 4509 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dab70d-openshift-default-6496f5466c-xczrw" podUID="1664ef62-abe8-47ed-b2b0-bb5d787c3e22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Nov 04 05:05:10.105494 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:10.105394 4509 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-3eb31c/premium-w4-3eb31c-kserve-58bc6d6c55-kh2c4"] Nov 04 05:05:10.110103 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:10.110070 4509 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-3eb31c/premium-w4-3eb31c-kserve-58bc6d6c55-kh2c4"] Nov 04 05:05:10.263999 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:10.263951 4509 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9181aed8-6d05-4691-be54-f26391920734" path="/var/lib/kubelet/pods/9181aed8-6d05-4691-be54-f26391920734/volumes" Nov 04 05:05:10.984350 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:10.983382 4509 patch_prober.go:29] interesting pod/e2e-worker-w3-dab70d-openshift-default-6496f5466c-xczrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Nov 04 05:05:10.984350 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:10.983444 4509 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dab70d-openshift-default-6496f5466c-xczrw" podUID="1664ef62-abe8-47ed-b2b0-bb5d787c3e22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Nov 04 05:05:11.980346 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:11.979789 4509 patch_prober.go:29] interesting pod/e2e-worker-w3-dab70d-openshift-default-6496f5466c-xczrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Nov 04 05:05:11.980346 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:11.979901 4509 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dab70d-openshift-default-6496f5466c-xczrw" podUID="1664ef62-abe8-47ed-b2b0-bb5d787c3e22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Nov 04 05:05:12.228105 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:12.228061 4509 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-3eb31c-589ff8b4b8-v89hw" podUID="1df0a5e9-fb8c-49d7-a5ba-9a04e99b6c09" containerName="payload-processing" containerID="cri-o://e866d18b9f28b4f043c2ccb972699f0011f4a2e238b534ea473e444638f23f81" gracePeriod=25 Nov 04 05:05:12.331896 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:12.331797 4509 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3eb31c-5bc57d7f87-xfpgv" podUID="c425a743-509e-4356-86a2-8a1779db79ae" containerName="payload-pre-processing" containerID="cri-o://9fe483b9696f72566747948c6bd8af9100050838fe1d8a66109d8d9aabaf9a9b" gracePeriod=25 Nov 04 05:05:12.984612 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:12.984499 4509 patch_prober.go:29] interesting pod/e2e-worker-w3-dab70d-openshift-default-6496f5466c-xczrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Nov 04 05:05:12.984612 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:12.984562 4509 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dab70d-openshift-default-6496f5466c-xczrw" podUID="1664ef62-abe8-47ed-b2b0-bb5d787c3e22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Nov 04 05:05:13.982349 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:13.981657 4509 patch_prober.go:29] interesting pod/e2e-worker-w3-dab70d-openshift-default-6496f5466c-xczrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Nov 04 05:05:13.982349 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:13.981721 4509 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dab70d-openshift-default-6496f5466c-xczrw" podUID="1664ef62-abe8-47ed-b2b0-bb5d787c3e22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Nov 04 05:05:13.986695 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:13.986663 4509 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-3eb31c-589ff8b4b8-v89hw" podUID="1df0a5e9-fb8c-49d7-a5ba-9a04e99b6c09" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 05:05:13.986845 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:13.986663 4509 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3eb31c-5bc57d7f87-xfpgv" podUID="c425a743-509e-4356-86a2-8a1779db79ae" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 05:05:14.455896 ip-10-0-133-180 kubenswrapper[4509]: E1104 05:05:14.455800 4509 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmdhl" containerName="extractor" Nov 04 05:05:15.240936 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:15.240898 4509 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-dab70d-openshift-default-6496f5466c-xczrw" Nov 04 05:05:15.242385 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:15.242361 4509 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-dab70d-openshift-default-6496f5466c-xczrw" Nov 04 05:05:15.243280 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:15.243258 4509 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-dab70d-openshift-default-6496f5466c-xczrw" Nov 04 05:05:20.313270 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:20.313237 4509 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-dab70d-openshift-default-6496f5466c-xczrw"] Nov 04 05:05:20.313724 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:20.313464 4509 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-dab70d-openshift-default-6496f5466c-xczrw" podUID="1664ef62-abe8-47ed-b2b0-bb5d787c3e22" containerName="istio-proxy" containerID="cri-o://d5c421a16fa693589cfe0881efe529aea7fc45e4fccfb0cd77640ad2f2e2787b" gracePeriod=30 Nov 04 05:05:23.992362 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:23.992278 4509 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3eb31c-5bc57d7f87-xfpgv" podUID="c425a743-509e-4356-86a2-8a1779db79ae" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 05:05:23.996558 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:23.996506 4509 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-3eb31c-589ff8b4b8-v89hw" podUID="1df0a5e9-fb8c-49d7-a5ba-9a04e99b6c09" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 05:05:25.588367 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:25.588287 4509 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-dab70d-openshift-default-6496f5466c-xczrw" Nov 04 05:05:25.716424 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:25.716382 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1664ef62-abe8-47ed-b2b0-bb5d787c3e22-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1664ef62-abe8-47ed-b2b0-bb5d787c3e22-istio-podinfo\") pod \"1664ef62-abe8-47ed-b2b0-bb5d787c3e22\" (UID: \"1664ef62-abe8-47ed-b2b0-bb5d787c3e22\") " Nov 04 05:05:25.716621 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:25.716452 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1664ef62-abe8-47ed-b2b0-bb5d787c3e22-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1664ef62-abe8-47ed-b2b0-bb5d787c3e22-workload-socket\") pod \"1664ef62-abe8-47ed-b2b0-bb5d787c3e22\" (UID: \"1664ef62-abe8-47ed-b2b0-bb5d787c3e22\") " Nov 04 05:05:25.716621 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:25.716509 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1664ef62-abe8-47ed-b2b0-bb5d787c3e22-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1664ef62-abe8-47ed-b2b0-bb5d787c3e22-workload-certs\") pod \"1664ef62-abe8-47ed-b2b0-bb5d787c3e22\" (UID: \"1664ef62-abe8-47ed-b2b0-bb5d787c3e22\") " Nov 04 05:05:25.716621 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:25.716538 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1664ef62-abe8-47ed-b2b0-bb5d787c3e22-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1664ef62-abe8-47ed-b2b0-bb5d787c3e22-credential-socket\") pod \"1664ef62-abe8-47ed-b2b0-bb5d787c3e22\" (UID: \"1664ef62-abe8-47ed-b2b0-bb5d787c3e22\") " Nov 04 05:05:25.716621 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:25.716564 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1664ef62-abe8-47ed-b2b0-bb5d787c3e22-kube-api-access-wm77f\" (UniqueName: \"kubernetes.io/projected/1664ef62-abe8-47ed-b2b0-bb5d787c3e22-kube-api-access-wm77f\") pod \"1664ef62-abe8-47ed-b2b0-bb5d787c3e22\" (UID: \"1664ef62-abe8-47ed-b2b0-bb5d787c3e22\") " Nov 04 05:05:25.716869 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:25.716647 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1664ef62-abe8-47ed-b2b0-bb5d787c3e22-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1664ef62-abe8-47ed-b2b0-bb5d787c3e22-istio-data\") pod \"1664ef62-abe8-47ed-b2b0-bb5d787c3e22\" (UID: \"1664ef62-abe8-47ed-b2b0-bb5d787c3e22\") " Nov 04 05:05:25.716869 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:25.716713 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1664ef62-abe8-47ed-b2b0-bb5d787c3e22-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1664ef62-abe8-47ed-b2b0-bb5d787c3e22-istio-envoy\") pod \"1664ef62-abe8-47ed-b2b0-bb5d787c3e22\" (UID: \"1664ef62-abe8-47ed-b2b0-bb5d787c3e22\") " Nov 04 05:05:25.716869 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:25.716739 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1664ef62-abe8-47ed-b2b0-bb5d787c3e22-workload-socket" pod "1664ef62-abe8-47ed-b2b0-bb5d787c3e22" (UID: "1664ef62-abe8-47ed-b2b0-bb5d787c3e22"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:05:25.716869 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:25.716753 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1664ef62-abe8-47ed-b2b0-bb5d787c3e22-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1664ef62-abe8-47ed-b2b0-bb5d787c3e22-istiod-ca-cert\") pod \"1664ef62-abe8-47ed-b2b0-bb5d787c3e22\" (UID: \"1664ef62-abe8-47ed-b2b0-bb5d787c3e22\") " Nov 04 05:05:25.717073 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:25.716941 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1664ef62-abe8-47ed-b2b0-bb5d787c3e22-istio-token\" (UniqueName: \"kubernetes.io/projected/1664ef62-abe8-47ed-b2b0-bb5d787c3e22-istio-token\") pod \"1664ef62-abe8-47ed-b2b0-bb5d787c3e22\" (UID: \"1664ef62-abe8-47ed-b2b0-bb5d787c3e22\") " Nov 04 05:05:25.717073 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:25.717054 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1664ef62-abe8-47ed-b2b0-bb5d787c3e22-istio-data" pod "1664ef62-abe8-47ed-b2b0-bb5d787c3e22" (UID: "1664ef62-abe8-47ed-b2b0-bb5d787c3e22"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:05:25.717178 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:25.717069 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1664ef62-abe8-47ed-b2b0-bb5d787c3e22-istiod-ca-cert" pod "1664ef62-abe8-47ed-b2b0-bb5d787c3e22" (UID: "1664ef62-abe8-47ed-b2b0-bb5d787c3e22"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 05:05:25.717178 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:25.717110 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1664ef62-abe8-47ed-b2b0-bb5d787c3e22-workload-certs" pod "1664ef62-abe8-47ed-b2b0-bb5d787c3e22" (UID: "1664ef62-abe8-47ed-b2b0-bb5d787c3e22"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:05:25.717438 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:25.717415 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1664ef62-abe8-47ed-b2b0-bb5d787c3e22-credential-socket" pod "1664ef62-abe8-47ed-b2b0-bb5d787c3e22" (UID: "1664ef62-abe8-47ed-b2b0-bb5d787c3e22"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:05:25.717606 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:25.717506 4509 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1664ef62-abe8-47ed-b2b0-bb5d787c3e22-workload-socket\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:05:25.717695 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:25.717618 4509 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1664ef62-abe8-47ed-b2b0-bb5d787c3e22-workload-certs\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:05:25.717695 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:25.717635 4509 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1664ef62-abe8-47ed-b2b0-bb5d787c3e22-istio-data\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:05:25.717695 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:25.717650 4509 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1664ef62-abe8-47ed-b2b0-bb5d787c3e22-istiod-ca-cert\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:05:25.719373 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:25.719347 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1664ef62-abe8-47ed-b2b0-bb5d787c3e22-istio-podinfo" pod "1664ef62-abe8-47ed-b2b0-bb5d787c3e22" (UID: "1664ef62-abe8-47ed-b2b0-bb5d787c3e22"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 05:05:25.719566 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:25.719536 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1664ef62-abe8-47ed-b2b0-bb5d787c3e22-istio-envoy" pod "1664ef62-abe8-47ed-b2b0-bb5d787c3e22" (UID: "1664ef62-abe8-47ed-b2b0-bb5d787c3e22"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:05:25.719631 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:25.719584 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1664ef62-abe8-47ed-b2b0-bb5d787c3e22-kube-api-access-wm77f" pod "1664ef62-abe8-47ed-b2b0-bb5d787c3e22" (UID: "1664ef62-abe8-47ed-b2b0-bb5d787c3e22"). InnerVolumeSpecName "kube-api-access-wm77f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 05:05:25.719631 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:25.719616 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1664ef62-abe8-47ed-b2b0-bb5d787c3e22-istio-token" pod "1664ef62-abe8-47ed-b2b0-bb5d787c3e22" (UID: "1664ef62-abe8-47ed-b2b0-bb5d787c3e22"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 05:05:25.818572 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:25.818540 4509 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1664ef62-abe8-47ed-b2b0-bb5d787c3e22-istio-envoy\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:05:25.818572 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:25.818567 4509 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1664ef62-abe8-47ed-b2b0-bb5d787c3e22-istio-token\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:05:25.818572 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:25.818576 4509 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1664ef62-abe8-47ed-b2b0-bb5d787c3e22-istio-podinfo\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:05:25.818805 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:25.818585 4509 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1664ef62-abe8-47ed-b2b0-bb5d787c3e22-credential-socket\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:05:25.818805 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:25.818595 4509 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wm77f\" (UniqueName: \"kubernetes.io/projected/1664ef62-abe8-47ed-b2b0-bb5d787c3e22-kube-api-access-wm77f\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:05:26.037833 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:26.037796 4509 generic.go:356] "Generic (PLEG): container finished" podID="1664ef62-abe8-47ed-b2b0-bb5d787c3e22" containerID="d5c421a16fa693589cfe0881efe529aea7fc45e4fccfb0cd77640ad2f2e2787b" exitCode=0 Nov 04 05:05:26.038006 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:26.037851 4509 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-dab70d-openshift-default-6496f5466c-xczrw" Nov 04 05:05:26.038006 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:26.037857 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-dab70d-openshift-default-6496f5466c-xczrw" event={"ID":"1664ef62-abe8-47ed-b2b0-bb5d787c3e22","Type":"ContainerDied","Data":"d5c421a16fa693589cfe0881efe529aea7fc45e4fccfb0cd77640ad2f2e2787b"} Nov 04 05:05:26.038006 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:26.037941 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-dab70d-openshift-default-6496f5466c-xczrw" event={"ID":"1664ef62-abe8-47ed-b2b0-bb5d787c3e22","Type":"ContainerDied","Data":"b3f52460a7bc0be84f90bea9a4f85b314e5d50140fc15bd4c1195430c1c47820"} Nov 04 05:05:26.038006 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:26.037978 4509 scope.go:122] "RemoveContainer" containerID="d5c421a16fa693589cfe0881efe529aea7fc45e4fccfb0cd77640ad2f2e2787b" Nov 04 05:05:26.048274 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:26.048252 4509 scope.go:122] "RemoveContainer" containerID="d5c421a16fa693589cfe0881efe529aea7fc45e4fccfb0cd77640ad2f2e2787b" Nov 04 05:05:26.048615 ip-10-0-133-180 kubenswrapper[4509]: E1104 05:05:26.048594 4509 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5c421a16fa693589cfe0881efe529aea7fc45e4fccfb0cd77640ad2f2e2787b\": container with ID starting with d5c421a16fa693589cfe0881efe529aea7fc45e4fccfb0cd77640ad2f2e2787b not found: ID does not exist" containerID="d5c421a16fa693589cfe0881efe529aea7fc45e4fccfb0cd77640ad2f2e2787b" Nov 04 05:05:26.048679 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:26.048624 4509 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5c421a16fa693589cfe0881efe529aea7fc45e4fccfb0cd77640ad2f2e2787b"} err="failed to get container status \"d5c421a16fa693589cfe0881efe529aea7fc45e4fccfb0cd77640ad2f2e2787b\": rpc error: code = NotFound desc = could not find container \"d5c421a16fa693589cfe0881efe529aea7fc45e4fccfb0cd77640ad2f2e2787b\": container with ID starting with d5c421a16fa693589cfe0881efe529aea7fc45e4fccfb0cd77640ad2f2e2787b not found: ID does not exist" Nov 04 05:05:26.203115 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:26.203080 4509 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-dab70d-openshift-default-6496f5466c-xczrw"] Nov 04 05:05:26.252710 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:26.252671 4509 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-dab70d-openshift-default-6496f5466c-xczrw"] Nov 04 05:05:26.263246 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:26.263213 4509 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1664ef62-abe8-47ed-b2b0-bb5d787c3e22" path="/var/lib/kubelet/pods/1664ef62-abe8-47ed-b2b0-bb5d787c3e22/volumes" Nov 04 05:05:28.491870 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:28.491828 4509 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-370078-openshift-default-584c45775-h6rwx"] Nov 04 05:05:28.492387 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:28.492096 4509 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-370078-openshift-default-584c45775-h6rwx" podUID="0dfa90b7-fb2a-439c-a9ca-d92c30a7f088" containerName="istio-proxy" containerID="cri-o://26445351906e2b176a05446942950cace621ccd4d3fbfafc591b6cee04bf1d05" gracePeriod=30 Nov 04 05:05:32.504628 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:32.504602 4509 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-3eb31c-589ff8b4b8-v89hw" Nov 04 05:05:32.577420 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:32.577397 4509 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3eb31c-5bc57d7f87-xfpgv" Nov 04 05:05:32.599486 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:32.599461 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1df0a5e9-fb8c-49d7-a5ba-9a04e99b6c09-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1df0a5e9-fb8c-49d7-a5ba-9a04e99b6c09-plugins-config-volume\") pod \"1df0a5e9-fb8c-49d7-a5ba-9a04e99b6c09\" (UID: \"1df0a5e9-fb8c-49d7-a5ba-9a04e99b6c09\") " Nov 04 05:05:32.599694 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:32.599574 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1df0a5e9-fb8c-49d7-a5ba-9a04e99b6c09-kube-api-access-xfv9t\" (UniqueName: \"kubernetes.io/projected/1df0a5e9-fb8c-49d7-a5ba-9a04e99b6c09-kube-api-access-xfv9t\") pod \"1df0a5e9-fb8c-49d7-a5ba-9a04e99b6c09\" (UID: \"1df0a5e9-fb8c-49d7-a5ba-9a04e99b6c09\") " Nov 04 05:05:32.599973 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:32.599950 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1df0a5e9-fb8c-49d7-a5ba-9a04e99b6c09-plugins-config-volume" pod "1df0a5e9-fb8c-49d7-a5ba-9a04e99b6c09" (UID: "1df0a5e9-fb8c-49d7-a5ba-9a04e99b6c09"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 05:05:32.601764 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:32.601745 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1df0a5e9-fb8c-49d7-a5ba-9a04e99b6c09-kube-api-access-xfv9t" pod "1df0a5e9-fb8c-49d7-a5ba-9a04e99b6c09" (UID: "1df0a5e9-fb8c-49d7-a5ba-9a04e99b6c09"). InnerVolumeSpecName "kube-api-access-xfv9t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 05:05:32.701130 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:32.701099 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c425a743-509e-4356-86a2-8a1779db79ae-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c425a743-509e-4356-86a2-8a1779db79ae-plugins-config-volume\") pod \"c425a743-509e-4356-86a2-8a1779db79ae\" (UID: \"c425a743-509e-4356-86a2-8a1779db79ae\") " Nov 04 05:05:32.701399 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:32.701184 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c425a743-509e-4356-86a2-8a1779db79ae-kube-api-access-lkfgk\" (UniqueName: \"kubernetes.io/projected/c425a743-509e-4356-86a2-8a1779db79ae-kube-api-access-lkfgk\") pod \"c425a743-509e-4356-86a2-8a1779db79ae\" (UID: \"c425a743-509e-4356-86a2-8a1779db79ae\") " Nov 04 05:05:32.701510 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:32.701495 4509 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1df0a5e9-fb8c-49d7-a5ba-9a04e99b6c09-plugins-config-volume\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:05:32.701553 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:32.701512 4509 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xfv9t\" (UniqueName: \"kubernetes.io/projected/1df0a5e9-fb8c-49d7-a5ba-9a04e99b6c09-kube-api-access-xfv9t\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:05:32.701553 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:32.701536 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c425a743-509e-4356-86a2-8a1779db79ae-plugins-config-volume" pod "c425a743-509e-4356-86a2-8a1779db79ae" (UID: "c425a743-509e-4356-86a2-8a1779db79ae"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 05:05:32.703379 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:32.703357 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c425a743-509e-4356-86a2-8a1779db79ae-kube-api-access-lkfgk" pod "c425a743-509e-4356-86a2-8a1779db79ae" (UID: "c425a743-509e-4356-86a2-8a1779db79ae"). InnerVolumeSpecName "kube-api-access-lkfgk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 05:05:32.802562 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:32.802492 4509 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c425a743-509e-4356-86a2-8a1779db79ae-plugins-config-volume\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:05:32.802562 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:32.802516 4509 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkfgk\" (UniqueName: \"kubernetes.io/projected/c425a743-509e-4356-86a2-8a1779db79ae-kube-api-access-lkfgk\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:05:33.106149 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:33.106068 4509 generic.go:356] "Generic (PLEG): container finished" podID="c425a743-509e-4356-86a2-8a1779db79ae" containerID="9fe483b9696f72566747948c6bd8af9100050838fe1d8a66109d8d9aabaf9a9b" exitCode=0 Nov 04 05:05:33.106149 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:33.106113 4509 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3eb31c-5bc57d7f87-xfpgv" Nov 04 05:05:33.106387 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:33.106154 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3eb31c-5bc57d7f87-xfpgv" event={"ID":"c425a743-509e-4356-86a2-8a1779db79ae","Type":"ContainerDied","Data":"9fe483b9696f72566747948c6bd8af9100050838fe1d8a66109d8d9aabaf9a9b"} Nov 04 05:05:33.106387 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:33.106229 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3eb31c-5bc57d7f87-xfpgv" event={"ID":"c425a743-509e-4356-86a2-8a1779db79ae","Type":"ContainerDied","Data":"f7072aa1782a673e9f5f9be3ebee8e54aa124fea166c794183adce5877042fdd"} Nov 04 05:05:33.106387 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:33.106252 4509 scope.go:122] "RemoveContainer" containerID="9fe483b9696f72566747948c6bd8af9100050838fe1d8a66109d8d9aabaf9a9b" Nov 04 05:05:33.107668 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:33.107648 4509 generic.go:356] "Generic (PLEG): container finished" podID="1df0a5e9-fb8c-49d7-a5ba-9a04e99b6c09" containerID="e866d18b9f28b4f043c2ccb972699f0011f4a2e238b534ea473e444638f23f81" exitCode=0 Nov 04 05:05:33.107760 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:33.107677 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-3eb31c-589ff8b4b8-v89hw" event={"ID":"1df0a5e9-fb8c-49d7-a5ba-9a04e99b6c09","Type":"ContainerDied","Data":"e866d18b9f28b4f043c2ccb972699f0011f4a2e238b534ea473e444638f23f81"} Nov 04 05:05:33.107760 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:33.107700 4509 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-3eb31c-589ff8b4b8-v89hw" Nov 04 05:05:33.107760 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:33.107721 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-3eb31c-589ff8b4b8-v89hw" event={"ID":"1df0a5e9-fb8c-49d7-a5ba-9a04e99b6c09","Type":"ContainerDied","Data":"ffc0ac2901187b3f7d7349055df9b469fd297c1dfc10448e1cb9913c7cc6aedc"} Nov 04 05:05:33.117340 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:33.117224 4509 scope.go:122] "RemoveContainer" containerID="9fe483b9696f72566747948c6bd8af9100050838fe1d8a66109d8d9aabaf9a9b" Nov 04 05:05:33.117649 ip-10-0-133-180 kubenswrapper[4509]: E1104 05:05:33.117628 4509 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9fe483b9696f72566747948c6bd8af9100050838fe1d8a66109d8d9aabaf9a9b\": container with ID starting with 9fe483b9696f72566747948c6bd8af9100050838fe1d8a66109d8d9aabaf9a9b not found: ID does not exist" containerID="9fe483b9696f72566747948c6bd8af9100050838fe1d8a66109d8d9aabaf9a9b" Nov 04 05:05:33.117741 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:33.117655 4509 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9fe483b9696f72566747948c6bd8af9100050838fe1d8a66109d8d9aabaf9a9b"} err="failed to get container status \"9fe483b9696f72566747948c6bd8af9100050838fe1d8a66109d8d9aabaf9a9b\": rpc error: code = NotFound desc = could not find container \"9fe483b9696f72566747948c6bd8af9100050838fe1d8a66109d8d9aabaf9a9b\": container with ID starting with 9fe483b9696f72566747948c6bd8af9100050838fe1d8a66109d8d9aabaf9a9b not found: ID does not exist" Nov 04 05:05:33.117741 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:33.117675 4509 scope.go:122] "RemoveContainer" containerID="e866d18b9f28b4f043c2ccb972699f0011f4a2e238b534ea473e444638f23f81" Nov 04 05:05:33.126042 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:33.126025 4509 scope.go:122] "RemoveContainer" containerID="e866d18b9f28b4f043c2ccb972699f0011f4a2e238b534ea473e444638f23f81" Nov 04 05:05:33.126282 ip-10-0-133-180 kubenswrapper[4509]: E1104 05:05:33.126263 4509 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e866d18b9f28b4f043c2ccb972699f0011f4a2e238b534ea473e444638f23f81\": container with ID starting with e866d18b9f28b4f043c2ccb972699f0011f4a2e238b534ea473e444638f23f81 not found: ID does not exist" containerID="e866d18b9f28b4f043c2ccb972699f0011f4a2e238b534ea473e444638f23f81" Nov 04 05:05:33.126372 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:33.126286 4509 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e866d18b9f28b4f043c2ccb972699f0011f4a2e238b534ea473e444638f23f81"} err="failed to get container status \"e866d18b9f28b4f043c2ccb972699f0011f4a2e238b534ea473e444638f23f81\": rpc error: code = NotFound desc = could not find container \"e866d18b9f28b4f043c2ccb972699f0011f4a2e238b534ea473e444638f23f81\": container with ID starting with e866d18b9f28b4f043c2ccb972699f0011f4a2e238b534ea473e444638f23f81 not found: ID does not exist" Nov 04 05:05:33.272049 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:33.272016 4509 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-3eb31c-589ff8b4b8-v89hw"] Nov 04 05:05:33.276818 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:33.276788 4509 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-3eb31c-589ff8b4b8-v89hw"] Nov 04 05:05:33.384006 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:33.383917 4509 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-3eb31c-5bc57d7f87-xfpgv"] Nov 04 05:05:33.387976 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:33.387952 4509 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-3eb31c-5bc57d7f87-xfpgv"] Nov 04 05:05:33.747892 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:33.747868 4509 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-370078-openshift-default-584c45775-h6rwx" Nov 04 05:05:33.812520 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:33.812467 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0dfa90b7-fb2a-439c-a9ca-d92c30a7f088-istio-token\" (UniqueName: \"kubernetes.io/projected/0dfa90b7-fb2a-439c-a9ca-d92c30a7f088-istio-token\") pod \"0dfa90b7-fb2a-439c-a9ca-d92c30a7f088\" (UID: \"0dfa90b7-fb2a-439c-a9ca-d92c30a7f088\") " Nov 04 05:05:33.812689 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:33.812534 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0dfa90b7-fb2a-439c-a9ca-d92c30a7f088-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dfa90b7-fb2a-439c-a9ca-d92c30a7f088-credential-socket\") pod \"0dfa90b7-fb2a-439c-a9ca-d92c30a7f088\" (UID: \"0dfa90b7-fb2a-439c-a9ca-d92c30a7f088\") " Nov 04 05:05:33.812689 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:33.812589 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0dfa90b7-fb2a-439c-a9ca-d92c30a7f088-kube-api-access-8cznd\" (UniqueName: \"kubernetes.io/projected/0dfa90b7-fb2a-439c-a9ca-d92c30a7f088-kube-api-access-8cznd\") pod \"0dfa90b7-fb2a-439c-a9ca-d92c30a7f088\" (UID: \"0dfa90b7-fb2a-439c-a9ca-d92c30a7f088\") " Nov 04 05:05:33.812689 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:33.812622 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0dfa90b7-fb2a-439c-a9ca-d92c30a7f088-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0dfa90b7-fb2a-439c-a9ca-d92c30a7f088-istio-data\") pod \"0dfa90b7-fb2a-439c-a9ca-d92c30a7f088\" (UID: \"0dfa90b7-fb2a-439c-a9ca-d92c30a7f088\") " Nov 04 05:05:33.812689 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:33.812658 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0dfa90b7-fb2a-439c-a9ca-d92c30a7f088-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dfa90b7-fb2a-439c-a9ca-d92c30a7f088-workload-socket\") pod \"0dfa90b7-fb2a-439c-a9ca-d92c30a7f088\" (UID: \"0dfa90b7-fb2a-439c-a9ca-d92c30a7f088\") " Nov 04 05:05:33.812689 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:33.812689 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0dfa90b7-fb2a-439c-a9ca-d92c30a7f088-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0dfa90b7-fb2a-439c-a9ca-d92c30a7f088-istiod-ca-cert\") pod \"0dfa90b7-fb2a-439c-a9ca-d92c30a7f088\" (UID: \"0dfa90b7-fb2a-439c-a9ca-d92c30a7f088\") " Nov 04 05:05:33.813231 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:33.812827 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0dfa90b7-fb2a-439c-a9ca-d92c30a7f088-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0dfa90b7-fb2a-439c-a9ca-d92c30a7f088-istio-podinfo\") pod \"0dfa90b7-fb2a-439c-a9ca-d92c30a7f088\" (UID: \"0dfa90b7-fb2a-439c-a9ca-d92c30a7f088\") " Nov 04 05:05:33.813231 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:33.812895 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0dfa90b7-fb2a-439c-a9ca-d92c30a7f088-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0dfa90b7-fb2a-439c-a9ca-d92c30a7f088-istio-envoy\") pod \"0dfa90b7-fb2a-439c-a9ca-d92c30a7f088\" (UID: \"0dfa90b7-fb2a-439c-a9ca-d92c30a7f088\") " Nov 04 05:05:33.813231 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:33.812933 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0dfa90b7-fb2a-439c-a9ca-d92c30a7f088-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0dfa90b7-fb2a-439c-a9ca-d92c30a7f088-workload-certs\") pod \"0dfa90b7-fb2a-439c-a9ca-d92c30a7f088\" (UID: \"0dfa90b7-fb2a-439c-a9ca-d92c30a7f088\") " Nov 04 05:05:33.813231 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:33.812922 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dfa90b7-fb2a-439c-a9ca-d92c30a7f088-credential-socket" pod "0dfa90b7-fb2a-439c-a9ca-d92c30a7f088" (UID: "0dfa90b7-fb2a-439c-a9ca-d92c30a7f088"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:05:33.813231 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:33.813005 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dfa90b7-fb2a-439c-a9ca-d92c30a7f088-istio-data" pod "0dfa90b7-fb2a-439c-a9ca-d92c30a7f088" (UID: "0dfa90b7-fb2a-439c-a9ca-d92c30a7f088"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:05:33.813231 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:33.813144 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0dfa90b7-fb2a-439c-a9ca-d92c30a7f088-istiod-ca-cert" pod "0dfa90b7-fb2a-439c-a9ca-d92c30a7f088" (UID: "0dfa90b7-fb2a-439c-a9ca-d92c30a7f088"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 05:05:33.813562 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:33.813394 4509 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dfa90b7-fb2a-439c-a9ca-d92c30a7f088-credential-socket\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:05:33.813562 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:33.813417 4509 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0dfa90b7-fb2a-439c-a9ca-d92c30a7f088-istio-data\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:05:33.813562 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:33.813431 4509 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0dfa90b7-fb2a-439c-a9ca-d92c30a7f088-istiod-ca-cert\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:05:33.813726 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:33.813569 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dfa90b7-fb2a-439c-a9ca-d92c30a7f088-workload-certs" pod "0dfa90b7-fb2a-439c-a9ca-d92c30a7f088" (UID: "0dfa90b7-fb2a-439c-a9ca-d92c30a7f088"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:05:33.813839 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:33.813797 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dfa90b7-fb2a-439c-a9ca-d92c30a7f088-workload-socket" pod "0dfa90b7-fb2a-439c-a9ca-d92c30a7f088" (UID: "0dfa90b7-fb2a-439c-a9ca-d92c30a7f088"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:05:33.814985 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:33.814960 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0dfa90b7-fb2a-439c-a9ca-d92c30a7f088-istio-token" pod "0dfa90b7-fb2a-439c-a9ca-d92c30a7f088" (UID: "0dfa90b7-fb2a-439c-a9ca-d92c30a7f088"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 05:05:33.815252 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:33.815232 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0dfa90b7-fb2a-439c-a9ca-d92c30a7f088-istio-podinfo" pod "0dfa90b7-fb2a-439c-a9ca-d92c30a7f088" (UID: "0dfa90b7-fb2a-439c-a9ca-d92c30a7f088"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 05:05:33.815494 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:33.815474 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0dfa90b7-fb2a-439c-a9ca-d92c30a7f088-kube-api-access-8cznd" pod "0dfa90b7-fb2a-439c-a9ca-d92c30a7f088" (UID: "0dfa90b7-fb2a-439c-a9ca-d92c30a7f088"). InnerVolumeSpecName "kube-api-access-8cznd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 05:05:33.815567 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:33.815546 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dfa90b7-fb2a-439c-a9ca-d92c30a7f088-istio-envoy" pod "0dfa90b7-fb2a-439c-a9ca-d92c30a7f088" (UID: "0dfa90b7-fb2a-439c-a9ca-d92c30a7f088"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:05:33.914997 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:33.914910 4509 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0dfa90b7-fb2a-439c-a9ca-d92c30a7f088-istio-token\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:05:33.914997 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:33.914940 4509 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8cznd\" (UniqueName: \"kubernetes.io/projected/0dfa90b7-fb2a-439c-a9ca-d92c30a7f088-kube-api-access-8cznd\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:05:33.914997 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:33.914954 4509 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dfa90b7-fb2a-439c-a9ca-d92c30a7f088-workload-socket\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:05:33.914997 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:33.914966 4509 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0dfa90b7-fb2a-439c-a9ca-d92c30a7f088-istio-podinfo\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:05:33.914997 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:33.914978 4509 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0dfa90b7-fb2a-439c-a9ca-d92c30a7f088-istio-envoy\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:05:33.914997 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:33.914990 4509 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0dfa90b7-fb2a-439c-a9ca-d92c30a7f088-workload-certs\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:05:34.114692 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:34.114658 4509 generic.go:356] "Generic (PLEG): container finished" podID="0dfa90b7-fb2a-439c-a9ca-d92c30a7f088" containerID="26445351906e2b176a05446942950cace621ccd4d3fbfafc591b6cee04bf1d05" exitCode=0 Nov 04 05:05:34.114879 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:34.114704 4509 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-370078-openshift-default-584c45775-h6rwx" Nov 04 05:05:34.114879 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:34.114746 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-370078-openshift-default-584c45775-h6rwx" event={"ID":"0dfa90b7-fb2a-439c-a9ca-d92c30a7f088","Type":"ContainerDied","Data":"26445351906e2b176a05446942950cace621ccd4d3fbfafc591b6cee04bf1d05"} Nov 04 05:05:34.114879 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:34.114800 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-370078-openshift-default-584c45775-h6rwx" event={"ID":"0dfa90b7-fb2a-439c-a9ca-d92c30a7f088","Type":"ContainerDied","Data":"e0949ef3def5b6fa3fa42b6e7eb2632e0bc2125f7b8e8fb0a2e4c9b2339daee4"} Nov 04 05:05:34.114879 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:34.114826 4509 scope.go:122] "RemoveContainer" containerID="26445351906e2b176a05446942950cace621ccd4d3fbfafc591b6cee04bf1d05" Nov 04 05:05:34.124660 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:34.124639 4509 scope.go:122] "RemoveContainer" containerID="26445351906e2b176a05446942950cace621ccd4d3fbfafc591b6cee04bf1d05" Nov 04 05:05:34.124964 ip-10-0-133-180 kubenswrapper[4509]: E1104 05:05:34.124936 4509 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26445351906e2b176a05446942950cace621ccd4d3fbfafc591b6cee04bf1d05\": container with ID starting with 26445351906e2b176a05446942950cace621ccd4d3fbfafc591b6cee04bf1d05 not found: ID does not exist" containerID="26445351906e2b176a05446942950cace621ccd4d3fbfafc591b6cee04bf1d05" Nov 04 05:05:34.125037 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:34.124972 4509 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26445351906e2b176a05446942950cace621ccd4d3fbfafc591b6cee04bf1d05"} err="failed to get container status \"26445351906e2b176a05446942950cace621ccd4d3fbfafc591b6cee04bf1d05\": rpc error: code = NotFound desc = could not find container \"26445351906e2b176a05446942950cace621ccd4d3fbfafc591b6cee04bf1d05\": container with ID starting with 26445351906e2b176a05446942950cace621ccd4d3fbfafc591b6cee04bf1d05 not found: ID does not exist" Nov 04 05:05:34.260683 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:34.260652 4509 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1df0a5e9-fb8c-49d7-a5ba-9a04e99b6c09" path="/var/lib/kubelet/pods/1df0a5e9-fb8c-49d7-a5ba-9a04e99b6c09/volumes" Nov 04 05:05:34.261036 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:34.261022 4509 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c425a743-509e-4356-86a2-8a1779db79ae" path="/var/lib/kubelet/pods/c425a743-509e-4356-86a2-8a1779db79ae/volumes" Nov 04 05:05:34.279545 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:34.279516 4509 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-370078-openshift-default-584c45775-h6rwx"] Nov 04 05:05:34.329290 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:34.329257 4509 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-370078-openshift-default-584c45775-h6rwx"] Nov 04 05:05:35.844678 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:35.844646 4509 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-g4ctm"] Nov 04 05:05:35.845122 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:35.844857 4509 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-g4ctm" podUID="c985ba5a-3feb-4b17-959e-b4ca34a5dfbc" containerName="istio-proxy" containerID="cri-o://daaaa3814f302b7704286c9b49a77219ed4ba6ae4241d9862bce265ef64d2722" gracePeriod=30 Nov 04 05:05:35.857616 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:35.857579 4509 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-tc7tm"] Nov 04 05:05:35.857848 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:35.857824 4509 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-tc7tm" podUID="098b1d63-ed94-4884-a6de-428f799c3e31" containerName="istio-proxy" containerID="cri-o://dc778622824d6f0abd8df357ba9a240d356cafd95b0444f85a4785c2abdc5a1d" gracePeriod=30 Nov 04 05:05:36.261307 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:36.261269 4509 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0dfa90b7-fb2a-439c-a9ca-d92c30a7f088" path="/var/lib/kubelet/pods/0dfa90b7-fb2a-439c-a9ca-d92c30a7f088/volumes" Nov 04 05:05:39.061117 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:39.061084 4509 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-cedfca/distinct-2-w6-cedfca-kserve-5465b749bb-g2tkm"] Nov 04 05:05:39.111378 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:39.111342 4509 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-cedfca/distinct-w6-cedfca-kserve-54897db797-fsx45"] Nov 04 05:05:39.114117 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:39.114084 4509 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-cedfca/simulator-w6-cedfca-kserve-687878f579-n9vnt"] Nov 04 05:05:41.107869 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:41.107847 4509 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-g4ctm" Nov 04 05:05:41.144217 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:41.144194 4509 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-tc7tm" Nov 04 05:05:41.158643 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:41.158616 4509 generic.go:356] "Generic (PLEG): container finished" podID="098b1d63-ed94-4884-a6de-428f799c3e31" containerID="dc778622824d6f0abd8df357ba9a240d356cafd95b0444f85a4785c2abdc5a1d" exitCode=0 Nov 04 05:05:41.158759 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:41.158657 4509 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-tc7tm" Nov 04 05:05:41.158759 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:41.158711 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-tc7tm" event={"ID":"098b1d63-ed94-4884-a6de-428f799c3e31","Type":"ContainerDied","Data":"dc778622824d6f0abd8df357ba9a240d356cafd95b0444f85a4785c2abdc5a1d"} Nov 04 05:05:41.158861 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:41.158764 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-tc7tm" event={"ID":"098b1d63-ed94-4884-a6de-428f799c3e31","Type":"ContainerDied","Data":"3da7182488504f0fb6536356966c14ce8133a3ad8fc01973880047a4a7c1db49"} Nov 04 05:05:41.158861 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:41.158782 4509 scope.go:122] "RemoveContainer" containerID="dc778622824d6f0abd8df357ba9a240d356cafd95b0444f85a4785c2abdc5a1d" Nov 04 05:05:41.160116 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:41.160095 4509 generic.go:356] "Generic (PLEG): container finished" podID="c985ba5a-3feb-4b17-959e-b4ca34a5dfbc" containerID="daaaa3814f302b7704286c9b49a77219ed4ba6ae4241d9862bce265ef64d2722" exitCode=0 Nov 04 05:05:41.160226 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:41.160134 4509 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-g4ctm" Nov 04 05:05:41.160270 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:41.160132 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-g4ctm" event={"ID":"c985ba5a-3feb-4b17-959e-b4ca34a5dfbc","Type":"ContainerDied","Data":"daaaa3814f302b7704286c9b49a77219ed4ba6ae4241d9862bce265ef64d2722"} Nov 04 05:05:41.160270 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:41.160258 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-g4ctm" event={"ID":"c985ba5a-3feb-4b17-959e-b4ca34a5dfbc","Type":"ContainerDied","Data":"5cb79bd4b679c69b25150a63b79a9964a5f4c1a5aabbb6698ee7c452658fb98a"} Nov 04 05:05:41.167736 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:41.167717 4509 scope.go:122] "RemoveContainer" containerID="dc778622824d6f0abd8df357ba9a240d356cafd95b0444f85a4785c2abdc5a1d" Nov 04 05:05:41.168012 ip-10-0-133-180 kubenswrapper[4509]: E1104 05:05:41.167994 4509 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc778622824d6f0abd8df357ba9a240d356cafd95b0444f85a4785c2abdc5a1d\": container with ID starting with dc778622824d6f0abd8df357ba9a240d356cafd95b0444f85a4785c2abdc5a1d not found: ID does not exist" containerID="dc778622824d6f0abd8df357ba9a240d356cafd95b0444f85a4785c2abdc5a1d" Nov 04 05:05:41.168063 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:41.168019 4509 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc778622824d6f0abd8df357ba9a240d356cafd95b0444f85a4785c2abdc5a1d"} err="failed to get container status \"dc778622824d6f0abd8df357ba9a240d356cafd95b0444f85a4785c2abdc5a1d\": rpc error: code = NotFound desc = could not find container \"dc778622824d6f0abd8df357ba9a240d356cafd95b0444f85a4785c2abdc5a1d\": container with ID starting with dc778622824d6f0abd8df357ba9a240d356cafd95b0444f85a4785c2abdc5a1d not found: ID does not exist" Nov 04 05:05:41.168063 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:41.168035 4509 scope.go:122] "RemoveContainer" containerID="daaaa3814f302b7704286c9b49a77219ed4ba6ae4241d9862bce265ef64d2722" Nov 04 05:05:41.176416 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:41.176401 4509 scope.go:122] "RemoveContainer" containerID="daaaa3814f302b7704286c9b49a77219ed4ba6ae4241d9862bce265ef64d2722" Nov 04 05:05:41.176732 ip-10-0-133-180 kubenswrapper[4509]: E1104 05:05:41.176710 4509 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"daaaa3814f302b7704286c9b49a77219ed4ba6ae4241d9862bce265ef64d2722\": container with ID starting with daaaa3814f302b7704286c9b49a77219ed4ba6ae4241d9862bce265ef64d2722 not found: ID does not exist" containerID="daaaa3814f302b7704286c9b49a77219ed4ba6ae4241d9862bce265ef64d2722" Nov 04 05:05:41.176775 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:41.176742 4509 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"daaaa3814f302b7704286c9b49a77219ed4ba6ae4241d9862bce265ef64d2722"} err="failed to get container status \"daaaa3814f302b7704286c9b49a77219ed4ba6ae4241d9862bce265ef64d2722\": rpc error: code = NotFound desc = could not find container \"daaaa3814f302b7704286c9b49a77219ed4ba6ae4241d9862bce265ef64d2722\": container with ID starting with daaaa3814f302b7704286c9b49a77219ed4ba6ae4241d9862bce265ef64d2722 not found: ID does not exist" Nov 04 05:05:41.189214 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:41.189189 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c985ba5a-3feb-4b17-959e-b4ca34a5dfbc-kube-api-access-4jrtf\" (UniqueName: \"kubernetes.io/projected/c985ba5a-3feb-4b17-959e-b4ca34a5dfbc-kube-api-access-4jrtf\") pod \"c985ba5a-3feb-4b17-959e-b4ca34a5dfbc\" (UID: \"c985ba5a-3feb-4b17-959e-b4ca34a5dfbc\") " Nov 04 05:05:41.189304 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:41.189223 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c985ba5a-3feb-4b17-959e-b4ca34a5dfbc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c985ba5a-3feb-4b17-959e-b4ca34a5dfbc-credential-socket\") pod \"c985ba5a-3feb-4b17-959e-b4ca34a5dfbc\" (UID: \"c985ba5a-3feb-4b17-959e-b4ca34a5dfbc\") " Nov 04 05:05:41.189384 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:41.189300 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c985ba5a-3feb-4b17-959e-b4ca34a5dfbc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c985ba5a-3feb-4b17-959e-b4ca34a5dfbc-istio-podinfo\") pod \"c985ba5a-3feb-4b17-959e-b4ca34a5dfbc\" (UID: \"c985ba5a-3feb-4b17-959e-b4ca34a5dfbc\") " Nov 04 05:05:41.189384 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:41.189360 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c985ba5a-3feb-4b17-959e-b4ca34a5dfbc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c985ba5a-3feb-4b17-959e-b4ca34a5dfbc-workload-socket\") pod \"c985ba5a-3feb-4b17-959e-b4ca34a5dfbc\" (UID: \"c985ba5a-3feb-4b17-959e-b4ca34a5dfbc\") " Nov 04 05:05:41.189504 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:41.189425 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c985ba5a-3feb-4b17-959e-b4ca34a5dfbc-istio-token\" (UniqueName: \"kubernetes.io/projected/c985ba5a-3feb-4b17-959e-b4ca34a5dfbc-istio-token\") pod \"c985ba5a-3feb-4b17-959e-b4ca34a5dfbc\" (UID: \"c985ba5a-3feb-4b17-959e-b4ca34a5dfbc\") " Nov 04 05:05:41.189504 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:41.189477 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c985ba5a-3feb-4b17-959e-b4ca34a5dfbc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c985ba5a-3feb-4b17-959e-b4ca34a5dfbc-workload-certs\") pod \"c985ba5a-3feb-4b17-959e-b4ca34a5dfbc\" (UID: \"c985ba5a-3feb-4b17-959e-b4ca34a5dfbc\") " Nov 04 05:05:41.189624 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:41.189539 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c985ba5a-3feb-4b17-959e-b4ca34a5dfbc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c985ba5a-3feb-4b17-959e-b4ca34a5dfbc-istiod-ca-cert\") pod \"c985ba5a-3feb-4b17-959e-b4ca34a5dfbc\" (UID: \"c985ba5a-3feb-4b17-959e-b4ca34a5dfbc\") " Nov 04 05:05:41.189624 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:41.189569 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c985ba5a-3feb-4b17-959e-b4ca34a5dfbc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c985ba5a-3feb-4b17-959e-b4ca34a5dfbc-istio-envoy\") pod \"c985ba5a-3feb-4b17-959e-b4ca34a5dfbc\" (UID: \"c985ba5a-3feb-4b17-959e-b4ca34a5dfbc\") " Nov 04 05:05:41.189624 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:41.189611 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c985ba5a-3feb-4b17-959e-b4ca34a5dfbc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c985ba5a-3feb-4b17-959e-b4ca34a5dfbc-istio-data\") pod \"c985ba5a-3feb-4b17-959e-b4ca34a5dfbc\" (UID: \"c985ba5a-3feb-4b17-959e-b4ca34a5dfbc\") " Nov 04 05:05:41.189757 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:41.189615 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c985ba5a-3feb-4b17-959e-b4ca34a5dfbc-credential-socket" pod "c985ba5a-3feb-4b17-959e-b4ca34a5dfbc" (UID: "c985ba5a-3feb-4b17-959e-b4ca34a5dfbc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:05:41.189757 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:41.189667 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c985ba5a-3feb-4b17-959e-b4ca34a5dfbc-workload-socket" pod "c985ba5a-3feb-4b17-959e-b4ca34a5dfbc" (UID: "c985ba5a-3feb-4b17-959e-b4ca34a5dfbc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:05:41.189918 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:41.189841 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c985ba5a-3feb-4b17-959e-b4ca34a5dfbc-workload-certs" pod "c985ba5a-3feb-4b17-959e-b4ca34a5dfbc" (UID: "c985ba5a-3feb-4b17-959e-b4ca34a5dfbc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:05:41.190006 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:41.189984 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c985ba5a-3feb-4b17-959e-b4ca34a5dfbc-istio-data" pod "c985ba5a-3feb-4b17-959e-b4ca34a5dfbc" (UID: "c985ba5a-3feb-4b17-959e-b4ca34a5dfbc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:05:41.190206 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:41.189906 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c985ba5a-3feb-4b17-959e-b4ca34a5dfbc-istiod-ca-cert" pod "c985ba5a-3feb-4b17-959e-b4ca34a5dfbc" (UID: "c985ba5a-3feb-4b17-959e-b4ca34a5dfbc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 05:05:41.190206 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:41.190184 4509 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c985ba5a-3feb-4b17-959e-b4ca34a5dfbc-workload-socket\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:05:41.190206 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:41.190206 4509 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c985ba5a-3feb-4b17-959e-b4ca34a5dfbc-workload-certs\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:05:41.190623 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:41.190223 4509 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c985ba5a-3feb-4b17-959e-b4ca34a5dfbc-istio-data\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:05:41.190623 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:41.190237 4509 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c985ba5a-3feb-4b17-959e-b4ca34a5dfbc-credential-socket\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:05:41.191451 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:41.191427 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c985ba5a-3feb-4b17-959e-b4ca34a5dfbc-kube-api-access-4jrtf" pod "c985ba5a-3feb-4b17-959e-b4ca34a5dfbc" (UID: "c985ba5a-3feb-4b17-959e-b4ca34a5dfbc"). InnerVolumeSpecName "kube-api-access-4jrtf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 05:05:41.191598 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:41.191575 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c985ba5a-3feb-4b17-959e-b4ca34a5dfbc-istio-podinfo" pod "c985ba5a-3feb-4b17-959e-b4ca34a5dfbc" (UID: "c985ba5a-3feb-4b17-959e-b4ca34a5dfbc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 05:05:41.191753 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:41.191737 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c985ba5a-3feb-4b17-959e-b4ca34a5dfbc-istio-envoy" pod "c985ba5a-3feb-4b17-959e-b4ca34a5dfbc" (UID: "c985ba5a-3feb-4b17-959e-b4ca34a5dfbc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:05:41.191924 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:41.191911 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c985ba5a-3feb-4b17-959e-b4ca34a5dfbc-istio-token" pod "c985ba5a-3feb-4b17-959e-b4ca34a5dfbc" (UID: "c985ba5a-3feb-4b17-959e-b4ca34a5dfbc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 05:05:41.290745 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:41.290713 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/098b1d63-ed94-4884-a6de-428f799c3e31-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/098b1d63-ed94-4884-a6de-428f799c3e31-istio-data\") pod \"098b1d63-ed94-4884-a6de-428f799c3e31\" (UID: \"098b1d63-ed94-4884-a6de-428f799c3e31\") " Nov 04 05:05:41.290923 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:41.290769 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/098b1d63-ed94-4884-a6de-428f799c3e31-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/098b1d63-ed94-4884-a6de-428f799c3e31-istiod-ca-cert\") pod \"098b1d63-ed94-4884-a6de-428f799c3e31\" (UID: \"098b1d63-ed94-4884-a6de-428f799c3e31\") " Nov 04 05:05:41.290923 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:41.290806 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/098b1d63-ed94-4884-a6de-428f799c3e31-kube-api-access-fgtct\" (UniqueName: \"kubernetes.io/projected/098b1d63-ed94-4884-a6de-428f799c3e31-kube-api-access-fgtct\") pod \"098b1d63-ed94-4884-a6de-428f799c3e31\" (UID: \"098b1d63-ed94-4884-a6de-428f799c3e31\") " Nov 04 05:05:41.290923 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:41.290845 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/098b1d63-ed94-4884-a6de-428f799c3e31-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/098b1d63-ed94-4884-a6de-428f799c3e31-istio-podinfo\") pod \"098b1d63-ed94-4884-a6de-428f799c3e31\" (UID: \"098b1d63-ed94-4884-a6de-428f799c3e31\") " Nov 04 05:05:41.291099 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:41.290920 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/098b1d63-ed94-4884-a6de-428f799c3e31-istio-token\" (UniqueName: \"kubernetes.io/projected/098b1d63-ed94-4884-a6de-428f799c3e31-istio-token\") pod \"098b1d63-ed94-4884-a6de-428f799c3e31\" (UID: \"098b1d63-ed94-4884-a6de-428f799c3e31\") " Nov 04 05:05:41.291099 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:41.290979 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/098b1d63-ed94-4884-a6de-428f799c3e31-istio-data" pod "098b1d63-ed94-4884-a6de-428f799c3e31" (UID: "098b1d63-ed94-4884-a6de-428f799c3e31"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:05:41.291099 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:41.290990 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/098b1d63-ed94-4884-a6de-428f799c3e31-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/098b1d63-ed94-4884-a6de-428f799c3e31-credential-socket\") pod \"098b1d63-ed94-4884-a6de-428f799c3e31\" (UID: \"098b1d63-ed94-4884-a6de-428f799c3e31\") " Nov 04 05:05:41.291254 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:41.291178 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/098b1d63-ed94-4884-a6de-428f799c3e31-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/098b1d63-ed94-4884-a6de-428f799c3e31-workload-certs\") pod \"098b1d63-ed94-4884-a6de-428f799c3e31\" (UID: \"098b1d63-ed94-4884-a6de-428f799c3e31\") " Nov 04 05:05:41.291254 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:41.291248 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/098b1d63-ed94-4884-a6de-428f799c3e31-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/098b1d63-ed94-4884-a6de-428f799c3e31-istio-envoy\") pod \"098b1d63-ed94-4884-a6de-428f799c3e31\" (UID: \"098b1d63-ed94-4884-a6de-428f799c3e31\") " Nov 04 05:05:41.291399 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:41.291292 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/098b1d63-ed94-4884-a6de-428f799c3e31-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/098b1d63-ed94-4884-a6de-428f799c3e31-workload-socket\") pod \"098b1d63-ed94-4884-a6de-428f799c3e31\" (UID: \"098b1d63-ed94-4884-a6de-428f799c3e31\") " Nov 04 05:05:41.291399 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:41.291314 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/098b1d63-ed94-4884-a6de-428f799c3e31-credential-socket" pod "098b1d63-ed94-4884-a6de-428f799c3e31" (UID: "098b1d63-ed94-4884-a6de-428f799c3e31"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:05:41.291503 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:41.291439 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/098b1d63-ed94-4884-a6de-428f799c3e31-workload-certs" pod "098b1d63-ed94-4884-a6de-428f799c3e31" (UID: "098b1d63-ed94-4884-a6de-428f799c3e31"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:05:41.291559 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:41.291515 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/098b1d63-ed94-4884-a6de-428f799c3e31-istiod-ca-cert" pod "098b1d63-ed94-4884-a6de-428f799c3e31" (UID: "098b1d63-ed94-4884-a6de-428f799c3e31"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 05:05:41.291838 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:41.291776 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/098b1d63-ed94-4884-a6de-428f799c3e31-workload-socket" pod "098b1d63-ed94-4884-a6de-428f799c3e31" (UID: "098b1d63-ed94-4884-a6de-428f799c3e31"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:05:41.291838 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:41.291817 4509 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c985ba5a-3feb-4b17-959e-b4ca34a5dfbc-istio-token\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:05:41.291838 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:41.291840 4509 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c985ba5a-3feb-4b17-959e-b4ca34a5dfbc-istiod-ca-cert\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:05:41.292075 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:41.291855 4509 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/098b1d63-ed94-4884-a6de-428f799c3e31-workload-certs\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:05:41.292075 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:41.291870 4509 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c985ba5a-3feb-4b17-959e-b4ca34a5dfbc-istio-envoy\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:05:41.292075 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:41.291883 4509 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/098b1d63-ed94-4884-a6de-428f799c3e31-istio-data\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:05:41.292075 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:41.291897 4509 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jrtf\" (UniqueName: \"kubernetes.io/projected/c985ba5a-3feb-4b17-959e-b4ca34a5dfbc-kube-api-access-4jrtf\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:05:41.292075 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:41.292016 4509 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/098b1d63-ed94-4884-a6de-428f799c3e31-istiod-ca-cert\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:05:41.292356 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:41.292081 4509 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c985ba5a-3feb-4b17-959e-b4ca34a5dfbc-istio-podinfo\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:05:41.292356 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:41.292096 4509 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/098b1d63-ed94-4884-a6de-428f799c3e31-credential-socket\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:05:41.293304 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:41.293276 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/098b1d63-ed94-4884-a6de-428f799c3e31-kube-api-access-fgtct" pod "098b1d63-ed94-4884-a6de-428f799c3e31" (UID: "098b1d63-ed94-4884-a6de-428f799c3e31"). InnerVolumeSpecName "kube-api-access-fgtct". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 05:05:41.293416 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:41.293383 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/098b1d63-ed94-4884-a6de-428f799c3e31-istio-envoy" pod "098b1d63-ed94-4884-a6de-428f799c3e31" (UID: "098b1d63-ed94-4884-a6de-428f799c3e31"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:05:41.293465 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:41.293449 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/098b1d63-ed94-4884-a6de-428f799c3e31-istio-podinfo" pod "098b1d63-ed94-4884-a6de-428f799c3e31" (UID: "098b1d63-ed94-4884-a6de-428f799c3e31"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 05:05:41.293852 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:41.293830 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/098b1d63-ed94-4884-a6de-428f799c3e31-istio-token" pod "098b1d63-ed94-4884-a6de-428f799c3e31" (UID: "098b1d63-ed94-4884-a6de-428f799c3e31"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 05:05:41.393106 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:41.393003 4509 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/098b1d63-ed94-4884-a6de-428f799c3e31-istio-envoy\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:05:41.393106 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:41.393038 4509 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/098b1d63-ed94-4884-a6de-428f799c3e31-workload-socket\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:05:41.393106 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:41.393055 4509 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgtct\" (UniqueName: \"kubernetes.io/projected/098b1d63-ed94-4884-a6de-428f799c3e31-kube-api-access-fgtct\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:05:41.393106 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:41.393069 4509 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/098b1d63-ed94-4884-a6de-428f799c3e31-istio-podinfo\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:05:41.393106 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:41.393078 4509 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/098b1d63-ed94-4884-a6de-428f799c3e31-istio-token\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:05:41.582852 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:41.582786 4509 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-tc7tm"] Nov 04 05:05:41.586287 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:41.586260 4509 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-tc7tm"] Nov 04 05:05:41.695478 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:41.695407 4509 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-g4ctm"] Nov 04 05:05:41.698872 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:41.698850 4509 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-g4ctm"] Nov 04 05:05:42.261434 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:42.261401 4509 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="098b1d63-ed94-4884-a6de-428f799c3e31" path="/var/lib/kubelet/pods/098b1d63-ed94-4884-a6de-428f799c3e31/volumes" Nov 04 05:05:42.261870 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:42.261856 4509 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c985ba5a-3feb-4b17-959e-b4ca34a5dfbc" path="/var/lib/kubelet/pods/c985ba5a-3feb-4b17-959e-b4ca34a5dfbc/volumes" Nov 04 05:05:44.454860 ip-10-0-133-180 kubenswrapper[4509]: E1104 05:05:44.454823 4509 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmdhl" containerName="extractor" Nov 04 05:05:45.452513 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:45.452479 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-42dnm"] Nov 04 05:05:52.000863 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:52.000816 4509 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9dc147b4-c8x2x"] Nov 04 05:05:52.001274 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:52.001258 4509 cpu_manager.go:414] "containerMap: removing container" podUID="0dfa90b7-fb2a-439c-a9ca-d92c30a7f088" containerName="istio-proxy" Nov 04 05:05:52.001317 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:52.001276 4509 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0dfa90b7-fb2a-439c-a9ca-d92c30a7f088" containerName="istio-proxy" Nov 04 05:05:52.001317 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:52.001302 4509 cpu_manager.go:414] "containerMap: removing container" podUID="098b1d63-ed94-4884-a6de-428f799c3e31" containerName="istio-proxy" Nov 04 05:05:52.001317 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:52.001307 4509 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="098b1d63-ed94-4884-a6de-428f799c3e31" containerName="istio-proxy" Nov 04 05:05:52.001317 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:52.001314 4509 cpu_manager.go:414] "containerMap: removing container" podUID="1664ef62-abe8-47ed-b2b0-bb5d787c3e22" containerName="istio-proxy" Nov 04 05:05:52.001317 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:52.001339 4509 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1664ef62-abe8-47ed-b2b0-bb5d787c3e22" containerName="istio-proxy" Nov 04 05:05:52.001494 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:52.001349 4509 cpu_manager.go:414] "containerMap: removing container" podUID="1df0a5e9-fb8c-49d7-a5ba-9a04e99b6c09" containerName="payload-processing" Nov 04 05:05:52.001494 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:52.001353 4509 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1df0a5e9-fb8c-49d7-a5ba-9a04e99b6c09" containerName="payload-processing" Nov 04 05:05:52.001494 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:52.001361 4509 cpu_manager.go:414] "containerMap: removing container" podUID="c425a743-509e-4356-86a2-8a1779db79ae" containerName="payload-pre-processing" Nov 04 05:05:52.001494 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:52.001365 4509 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c425a743-509e-4356-86a2-8a1779db79ae" containerName="payload-pre-processing" Nov 04 05:05:52.001494 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:52.001373 4509 cpu_manager.go:414] "containerMap: removing container" podUID="c985ba5a-3feb-4b17-959e-b4ca34a5dfbc" containerName="istio-proxy" Nov 04 05:05:52.001494 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:52.001377 4509 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c985ba5a-3feb-4b17-959e-b4ca34a5dfbc" containerName="istio-proxy" Nov 04 05:05:52.001494 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:52.001389 4509 cpu_manager.go:414] "containerMap: removing container" podUID="9181aed8-6d05-4691-be54-f26391920734" containerName="main" Nov 04 05:05:52.001494 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:52.001393 4509 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9181aed8-6d05-4691-be54-f26391920734" containerName="main" Nov 04 05:05:52.001494 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:52.001432 4509 cpu_manager.go:414] "containerMap: removing container" podUID="950a4971-bcd5-43b7-a2c6-7545c36a63ea" containerName="revoke-keys" Nov 04 05:05:52.001494 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:52.001437 4509 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="950a4971-bcd5-43b7-a2c6-7545c36a63ea" containerName="revoke-keys" Nov 04 05:05:52.001898 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:52.001515 4509 memory_manager.go:365] "RemoveStaleState removing state" podUID="098b1d63-ed94-4884-a6de-428f799c3e31" containerName="istio-proxy" Nov 04 05:05:52.001898 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:52.001523 4509 memory_manager.go:365] "RemoveStaleState removing state" podUID="1664ef62-abe8-47ed-b2b0-bb5d787c3e22" containerName="istio-proxy" Nov 04 05:05:52.001898 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:52.001529 4509 memory_manager.go:365] "RemoveStaleState removing state" podUID="1df0a5e9-fb8c-49d7-a5ba-9a04e99b6c09" containerName="payload-processing" Nov 04 05:05:52.001898 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:52.001538 4509 memory_manager.go:365] "RemoveStaleState removing state" podUID="c425a743-509e-4356-86a2-8a1779db79ae" containerName="payload-pre-processing" Nov 04 05:05:52.001898 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:52.001543 4509 memory_manager.go:365] "RemoveStaleState removing state" podUID="c985ba5a-3feb-4b17-959e-b4ca34a5dfbc" containerName="istio-proxy" Nov 04 05:05:52.001898 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:52.001550 4509 memory_manager.go:365] "RemoveStaleState removing state" podUID="9181aed8-6d05-4691-be54-f26391920734" containerName="main" Nov 04 05:05:52.001898 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:52.001556 4509 memory_manager.go:365] "RemoveStaleState removing state" podUID="950a4971-bcd5-43b7-a2c6-7545c36a63ea" containerName="revoke-keys" Nov 04 05:05:52.001898 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:52.001564 4509 memory_manager.go:365] "RemoveStaleState removing state" podUID="0dfa90b7-fb2a-439c-a9ca-d92c30a7f088" containerName="istio-proxy" Nov 04 05:05:52.006189 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:52.006173 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dc147b4-c8x2x" Nov 04 05:05:52.059358 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:52.059306 4509 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 05:05:52.061622 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:52.061596 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9dc147b4-c8x2x"] Nov 04 05:05:52.093865 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:52.093836 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c76c7ef3-5e20-413d-9528-c0264ba54975-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9dc147b4-c8x2x\" (UID: \"c76c7ef3-5e20-413d-9528-c0264ba54975\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dc147b4-c8x2x" Nov 04 05:05:52.094009 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:52.093890 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c76c7ef3-5e20-413d-9528-c0264ba54975-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9dc147b4-c8x2x\" (UID: \"c76c7ef3-5e20-413d-9528-c0264ba54975\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dc147b4-c8x2x" Nov 04 05:05:52.094061 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:52.094038 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7jj4\" (UniqueName: \"kubernetes.io/projected/c76c7ef3-5e20-413d-9528-c0264ba54975-kube-api-access-k7jj4\") pod \"maas-api-revoke-sub-9dc147b4-c8x2x\" (UID: \"c76c7ef3-5e20-413d-9528-c0264ba54975\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dc147b4-c8x2x" Nov 04 05:05:52.108242 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:52.108220 4509 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-krgvj\"" Nov 04 05:05:52.108416 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:52.108242 4509 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 05:05:52.109124 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:52.109104 4509 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 05:05:52.194993 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:52.194957 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7jj4\" (UniqueName: \"kubernetes.io/projected/c76c7ef3-5e20-413d-9528-c0264ba54975-kube-api-access-k7jj4\") pod \"maas-api-revoke-sub-9dc147b4-c8x2x\" (UID: \"c76c7ef3-5e20-413d-9528-c0264ba54975\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dc147b4-c8x2x" Nov 04 05:05:52.195166 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:52.195059 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c76c7ef3-5e20-413d-9528-c0264ba54975-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9dc147b4-c8x2x\" (UID: \"c76c7ef3-5e20-413d-9528-c0264ba54975\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dc147b4-c8x2x" Nov 04 05:05:52.195166 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:52.195124 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c76c7ef3-5e20-413d-9528-c0264ba54975-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9dc147b4-c8x2x\" (UID: \"c76c7ef3-5e20-413d-9528-c0264ba54975\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dc147b4-c8x2x" Nov 04 05:05:52.195825 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:52.195802 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c76c7ef3-5e20-413d-9528-c0264ba54975-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9dc147b4-c8x2x\" (UID: \"c76c7ef3-5e20-413d-9528-c0264ba54975\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dc147b4-c8x2x" Nov 04 05:05:52.197769 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:52.197748 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c76c7ef3-5e20-413d-9528-c0264ba54975-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9dc147b4-c8x2x\" (UID: \"c76c7ef3-5e20-413d-9528-c0264ba54975\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dc147b4-c8x2x" Nov 04 05:05:52.251626 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:52.251572 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7jj4\" (UniqueName: \"kubernetes.io/projected/c76c7ef3-5e20-413d-9528-c0264ba54975-kube-api-access-k7jj4\") pod \"maas-api-revoke-sub-9dc147b4-c8x2x\" (UID: \"c76c7ef3-5e20-413d-9528-c0264ba54975\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dc147b4-c8x2x" Nov 04 05:05:52.279611 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:52.279588 4509 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-cedfca-c7c8f5b7d-hdqcw"] Nov 04 05:05:52.321758 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:52.321730 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dc147b4-c8x2x" Nov 04 05:05:52.391737 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:52.391697 4509 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-cedfca-67b95cc659-6v86t"] Nov 04 05:05:52.708363 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:52.708314 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9dc147b4-c8x2x"] Nov 04 05:05:52.712283 ip-10-0-133-180 kubenswrapper[4509]: W1104 05:05:52.712259 4509 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc76c7ef3_5e20_413d_9528_c0264ba54975.slice/crio-ffa7e62e462642030e01deda550da4982d7f19383cd687d355d2f63b3f92a36c WatchSource:0}: Error finding container ffa7e62e462642030e01deda550da4982d7f19383cd687d355d2f63b3f92a36c: Status 404 returned error can't find the container with id ffa7e62e462642030e01deda550da4982d7f19383cd687d355d2f63b3f92a36c Nov 04 05:05:53.234484 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:53.234405 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dc147b4-c8x2x" event={"ID":"c76c7ef3-5e20-413d-9528-c0264ba54975","Type":"ContainerStarted","Data":"818d206feab6aa33a0a71cd5645d72194ecfd4b058e192173327472526d23085"} Nov 04 05:05:53.234484 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:53.234458 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dc147b4-c8x2x" event={"ID":"c76c7ef3-5e20-413d-9528-c0264ba54975","Type":"ContainerStarted","Data":"ffa7e62e462642030e01deda550da4982d7f19383cd687d355d2f63b3f92a36c"} Nov 04 05:05:53.347850 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:53.347807 4509 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dc147b4-c8x2x" podStartSLOduration=2.3477937 podStartE2EDuration="2.3477937s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 05:05:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 05:05:53.346614399 +0000 UTC m=+1117.224337919" watchObservedRunningTime="2026-11-04 05:05:53.3477937 +0000 UTC m=+1117.225517219" Nov 04 05:05:53.402278 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:53.401846 4509 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-4cce33/distinct-w5-4cce33-kserve-68bf9d9bb7-x62f2"] Nov 04 05:05:53.411872 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:53.411837 4509 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-4cce33/premium-w5-4cce33-kserve-5875cf9fc6-9k7jz"] Nov 04 05:05:54.081673 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:54.081627 4509 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-cedfca/distinct-2-w6-cedfca-kserve-5465b749bb-g2tkm" podUID="97e3afca-28dd-4f22-89bf-14245acc4544" containerName="main" containerID="cri-o://ed6811d301fd1b7cb6e597ec418ba0765deb72b2154eb05e591561d7a6fc4ae1" gracePeriod=45 Nov 04 05:05:54.131965 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:54.131937 4509 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-cedfca/distinct-w6-cedfca-kserve-54897db797-fsx45" podUID="e5cf9f31-04dc-4d03-a8e2-bf38bcaf3830" containerName="main" containerID="cri-o://73f05a18ee37cace663307b405a11529613e33d011290d4755ef03d1d3cd6216" gracePeriod=45 Nov 04 05:05:54.133113 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:54.133083 4509 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-cedfca/simulator-w6-cedfca-kserve-687878f579-n9vnt" podUID="cea7867b-14f2-4967-8467-1a4d4e698f10" containerName="main" containerID="cri-o://7f44fc3a1b6b633d1d9773b9a6c0bf26610cb463720e7305cd959a53fe32a118" gracePeriod=45 Nov 04 05:05:54.243205 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:54.243178 4509 generic.go:356] "Generic (PLEG): container finished" podID="c76c7ef3-5e20-413d-9528-c0264ba54975" containerID="818d206feab6aa33a0a71cd5645d72194ecfd4b058e192173327472526d23085" exitCode=0 Nov 04 05:05:54.243581 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:54.243268 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dc147b4-c8x2x" event={"ID":"c76c7ef3-5e20-413d-9528-c0264ba54975","Type":"ContainerDied","Data":"818d206feab6aa33a0a71cd5645d72194ecfd4b058e192173327472526d23085"} Nov 04 05:05:54.245391 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:54.245370 4509 generic.go:356] "Generic (PLEG): container finished" podID="97e3afca-28dd-4f22-89bf-14245acc4544" containerID="ed6811d301fd1b7cb6e597ec418ba0765deb72b2154eb05e591561d7a6fc4ae1" exitCode=0 Nov 04 05:05:54.245489 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:54.245403 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-cedfca/distinct-2-w6-cedfca-kserve-5465b749bb-g2tkm" event={"ID":"97e3afca-28dd-4f22-89bf-14245acc4544","Type":"ContainerDied","Data":"ed6811d301fd1b7cb6e597ec418ba0765deb72b2154eb05e591561d7a6fc4ae1"} Nov 04 05:05:54.408835 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:54.408814 4509 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-cedfca/simulator-w6-cedfca-kserve-687878f579-n9vnt" Nov 04 05:05:54.420409 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:54.420381 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cea7867b-14f2-4967-8467-1a4d4e698f10-kube-api-access-pcgnk\" (UniqueName: \"kubernetes.io/projected/cea7867b-14f2-4967-8467-1a4d4e698f10-kube-api-access-pcgnk\") pod \"cea7867b-14f2-4967-8467-1a4d4e698f10\" (UID: \"cea7867b-14f2-4967-8467-1a4d4e698f10\") " Nov 04 05:05:54.420527 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:54.420499 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cea7867b-14f2-4967-8467-1a4d4e698f10-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cea7867b-14f2-4967-8467-1a4d4e698f10-tmp-dir\") pod \"cea7867b-14f2-4967-8467-1a4d4e698f10\" (UID: \"cea7867b-14f2-4967-8467-1a4d4e698f10\") " Nov 04 05:05:54.420600 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:54.420533 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cea7867b-14f2-4967-8467-1a4d4e698f10-home\" (UniqueName: \"kubernetes.io/empty-dir/cea7867b-14f2-4967-8467-1a4d4e698f10-home\") pod \"cea7867b-14f2-4967-8467-1a4d4e698f10\" (UID: \"cea7867b-14f2-4967-8467-1a4d4e698f10\") " Nov 04 05:05:54.420600 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:54.420589 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cea7867b-14f2-4967-8467-1a4d4e698f10-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cea7867b-14f2-4967-8467-1a4d4e698f10-model-cache\") pod \"cea7867b-14f2-4967-8467-1a4d4e698f10\" (UID: \"cea7867b-14f2-4967-8467-1a4d4e698f10\") " Nov 04 05:05:54.421006 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:54.420693 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cea7867b-14f2-4967-8467-1a4d4e698f10-tls-certs\" (UniqueName: \"kubernetes.io/secret/cea7867b-14f2-4967-8467-1a4d4e698f10-tls-certs\") pod \"cea7867b-14f2-4967-8467-1a4d4e698f10\" (UID: \"cea7867b-14f2-4967-8467-1a4d4e698f10\") " Nov 04 05:05:54.421006 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:54.420783 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cea7867b-14f2-4967-8467-1a4d4e698f10-dshm\" (UniqueName: \"kubernetes.io/empty-dir/cea7867b-14f2-4967-8467-1a4d4e698f10-dshm\") pod \"cea7867b-14f2-4967-8467-1a4d4e698f10\" (UID: \"cea7867b-14f2-4967-8467-1a4d4e698f10\") " Nov 04 05:05:54.421494 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:54.421290 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cea7867b-14f2-4967-8467-1a4d4e698f10-model-cache" pod "cea7867b-14f2-4967-8467-1a4d4e698f10" (UID: "cea7867b-14f2-4967-8467-1a4d4e698f10"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:05:54.421494 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:54.421409 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cea7867b-14f2-4967-8467-1a4d4e698f10-home" pod "cea7867b-14f2-4967-8467-1a4d4e698f10" (UID: "cea7867b-14f2-4967-8467-1a4d4e698f10"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:05:54.422010 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:54.421979 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cea7867b-14f2-4967-8467-1a4d4e698f10-tmp-dir" pod "cea7867b-14f2-4967-8467-1a4d4e698f10" (UID: "cea7867b-14f2-4967-8467-1a4d4e698f10"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:05:54.424882 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:54.424859 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cea7867b-14f2-4967-8467-1a4d4e698f10-kube-api-access-pcgnk" pod "cea7867b-14f2-4967-8467-1a4d4e698f10" (UID: "cea7867b-14f2-4967-8467-1a4d4e698f10"). InnerVolumeSpecName "kube-api-access-pcgnk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 05:05:54.425011 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:54.424992 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cea7867b-14f2-4967-8467-1a4d4e698f10-tls-certs" pod "cea7867b-14f2-4967-8467-1a4d4e698f10" (UID: "cea7867b-14f2-4967-8467-1a4d4e698f10"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 05:05:54.425082 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:54.425017 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cea7867b-14f2-4967-8467-1a4d4e698f10-dshm" pod "cea7867b-14f2-4967-8467-1a4d4e698f10" (UID: "cea7867b-14f2-4967-8467-1a4d4e698f10"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:05:54.432598 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:54.432580 4509 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-cedfca/distinct-2-w6-cedfca-kserve-5465b749bb-g2tkm" Nov 04 05:05:54.436738 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:54.436716 4509 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-cedfca/distinct-w6-cedfca-kserve-54897db797-fsx45" Nov 04 05:05:54.522260 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:54.522232 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97e3afca-28dd-4f22-89bf-14245acc4544-kube-api-access-5rfcn\" (UniqueName: \"kubernetes.io/projected/97e3afca-28dd-4f22-89bf-14245acc4544-kube-api-access-5rfcn\") pod \"97e3afca-28dd-4f22-89bf-14245acc4544\" (UID: \"97e3afca-28dd-4f22-89bf-14245acc4544\") " Nov 04 05:05:54.522427 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:54.522287 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5cf9f31-04dc-4d03-a8e2-bf38bcaf3830-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5cf9f31-04dc-4d03-a8e2-bf38bcaf3830-model-cache\") pod \"e5cf9f31-04dc-4d03-a8e2-bf38bcaf3830\" (UID: \"e5cf9f31-04dc-4d03-a8e2-bf38bcaf3830\") " Nov 04 05:05:54.522427 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:54.522348 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5cf9f31-04dc-4d03-a8e2-bf38bcaf3830-kube-api-access-phcsq\" (UniqueName: \"kubernetes.io/projected/e5cf9f31-04dc-4d03-a8e2-bf38bcaf3830-kube-api-access-phcsq\") pod \"e5cf9f31-04dc-4d03-a8e2-bf38bcaf3830\" (UID: \"e5cf9f31-04dc-4d03-a8e2-bf38bcaf3830\") " Nov 04 05:05:54.522427 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:54.522385 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e5cf9f31-04dc-4d03-a8e2-bf38bcaf3830-tls-certs\" (UniqueName: \"kubernetes.io/secret/e5cf9f31-04dc-4d03-a8e2-bf38bcaf3830-tls-certs\") pod \"e5cf9f31-04dc-4d03-a8e2-bf38bcaf3830\" (UID: \"e5cf9f31-04dc-4d03-a8e2-bf38bcaf3830\") " Nov 04 05:05:54.522427 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:54.522423 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97e3afca-28dd-4f22-89bf-14245acc4544-dshm\" (UniqueName: \"kubernetes.io/empty-dir/97e3afca-28dd-4f22-89bf-14245acc4544-dshm\") pod \"97e3afca-28dd-4f22-89bf-14245acc4544\" (UID: \"97e3afca-28dd-4f22-89bf-14245acc4544\") " Nov 04 05:05:54.522645 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:54.522452 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5cf9f31-04dc-4d03-a8e2-bf38bcaf3830-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5cf9f31-04dc-4d03-a8e2-bf38bcaf3830-dshm\") pod \"e5cf9f31-04dc-4d03-a8e2-bf38bcaf3830\" (UID: \"e5cf9f31-04dc-4d03-a8e2-bf38bcaf3830\") " Nov 04 05:05:54.522645 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:54.522505 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5cf9f31-04dc-4d03-a8e2-bf38bcaf3830-home\" (UniqueName: \"kubernetes.io/empty-dir/e5cf9f31-04dc-4d03-a8e2-bf38bcaf3830-home\") pod \"e5cf9f31-04dc-4d03-a8e2-bf38bcaf3830\" (UID: \"e5cf9f31-04dc-4d03-a8e2-bf38bcaf3830\") " Nov 04 05:05:54.522645 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:54.522547 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/97e3afca-28dd-4f22-89bf-14245acc4544-tls-certs\" (UniqueName: \"kubernetes.io/secret/97e3afca-28dd-4f22-89bf-14245acc4544-tls-certs\") pod \"97e3afca-28dd-4f22-89bf-14245acc4544\" (UID: \"97e3afca-28dd-4f22-89bf-14245acc4544\") " Nov 04 05:05:54.522645 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:54.522561 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5cf9f31-04dc-4d03-a8e2-bf38bcaf3830-model-cache" pod "e5cf9f31-04dc-4d03-a8e2-bf38bcaf3830" (UID: "e5cf9f31-04dc-4d03-a8e2-bf38bcaf3830"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:05:54.522645 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:54.522620 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97e3afca-28dd-4f22-89bf-14245acc4544-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97e3afca-28dd-4f22-89bf-14245acc4544-tmp-dir\") pod \"97e3afca-28dd-4f22-89bf-14245acc4544\" (UID: \"97e3afca-28dd-4f22-89bf-14245acc4544\") " Nov 04 05:05:54.522897 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:54.522653 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5cf9f31-04dc-4d03-a8e2-bf38bcaf3830-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5cf9f31-04dc-4d03-a8e2-bf38bcaf3830-tmp-dir\") pod \"e5cf9f31-04dc-4d03-a8e2-bf38bcaf3830\" (UID: \"e5cf9f31-04dc-4d03-a8e2-bf38bcaf3830\") " Nov 04 05:05:54.522897 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:54.522687 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97e3afca-28dd-4f22-89bf-14245acc4544-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97e3afca-28dd-4f22-89bf-14245acc4544-model-cache\") pod \"97e3afca-28dd-4f22-89bf-14245acc4544\" (UID: \"97e3afca-28dd-4f22-89bf-14245acc4544\") " Nov 04 05:05:54.522897 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:54.522735 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97e3afca-28dd-4f22-89bf-14245acc4544-home\" (UniqueName: \"kubernetes.io/empty-dir/97e3afca-28dd-4f22-89bf-14245acc4544-home\") pod \"97e3afca-28dd-4f22-89bf-14245acc4544\" (UID: \"97e3afca-28dd-4f22-89bf-14245acc4544\") " Nov 04 05:05:54.523307 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:54.523094 4509 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cea7867b-14f2-4967-8467-1a4d4e698f10-tls-certs\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:05:54.523307 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:54.523234 4509 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cea7867b-14f2-4967-8467-1a4d4e698f10-dshm\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:05:54.523307 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:54.523248 4509 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5cf9f31-04dc-4d03-a8e2-bf38bcaf3830-model-cache\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:05:54.523307 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:54.523262 4509 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pcgnk\" (UniqueName: \"kubernetes.io/projected/cea7867b-14f2-4967-8467-1a4d4e698f10-kube-api-access-pcgnk\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:05:54.523307 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:54.523299 4509 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cea7867b-14f2-4967-8467-1a4d4e698f10-tmp-dir\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:05:54.523307 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:54.523314 4509 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cea7867b-14f2-4967-8467-1a4d4e698f10-home\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:05:54.523307 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:54.523343 4509 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cea7867b-14f2-4967-8467-1a4d4e698f10-model-cache\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:05:54.523861 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:54.523565 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97e3afca-28dd-4f22-89bf-14245acc4544-home" pod "97e3afca-28dd-4f22-89bf-14245acc4544" (UID: "97e3afca-28dd-4f22-89bf-14245acc4544"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:05:54.523861 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:54.523735 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97e3afca-28dd-4f22-89bf-14245acc4544-tmp-dir" pod "97e3afca-28dd-4f22-89bf-14245acc4544" (UID: "97e3afca-28dd-4f22-89bf-14245acc4544"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:05:54.524014 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:54.523906 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5cf9f31-04dc-4d03-a8e2-bf38bcaf3830-tmp-dir" pod "e5cf9f31-04dc-4d03-a8e2-bf38bcaf3830" (UID: "e5cf9f31-04dc-4d03-a8e2-bf38bcaf3830"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:05:54.524095 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:54.524065 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97e3afca-28dd-4f22-89bf-14245acc4544-model-cache" pod "97e3afca-28dd-4f22-89bf-14245acc4544" (UID: "97e3afca-28dd-4f22-89bf-14245acc4544"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:05:54.524244 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:54.524225 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5cf9f31-04dc-4d03-a8e2-bf38bcaf3830-home" pod "e5cf9f31-04dc-4d03-a8e2-bf38bcaf3830" (UID: "e5cf9f31-04dc-4d03-a8e2-bf38bcaf3830"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:05:54.525153 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:54.525127 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97e3afca-28dd-4f22-89bf-14245acc4544-kube-api-access-5rfcn" pod "97e3afca-28dd-4f22-89bf-14245acc4544" (UID: "97e3afca-28dd-4f22-89bf-14245acc4544"). InnerVolumeSpecName "kube-api-access-5rfcn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 05:05:54.525280 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:54.525168 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5cf9f31-04dc-4d03-a8e2-bf38bcaf3830-kube-api-access-phcsq" pod "e5cf9f31-04dc-4d03-a8e2-bf38bcaf3830" (UID: "e5cf9f31-04dc-4d03-a8e2-bf38bcaf3830"). InnerVolumeSpecName "kube-api-access-phcsq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 05:05:54.525561 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:54.525538 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97e3afca-28dd-4f22-89bf-14245acc4544-dshm" pod "97e3afca-28dd-4f22-89bf-14245acc4544" (UID: "97e3afca-28dd-4f22-89bf-14245acc4544"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:05:54.525754 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:54.525737 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5cf9f31-04dc-4d03-a8e2-bf38bcaf3830-tls-certs" pod "e5cf9f31-04dc-4d03-a8e2-bf38bcaf3830" (UID: "e5cf9f31-04dc-4d03-a8e2-bf38bcaf3830"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 05:05:54.526187 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:54.526158 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5cf9f31-04dc-4d03-a8e2-bf38bcaf3830-dshm" pod "e5cf9f31-04dc-4d03-a8e2-bf38bcaf3830" (UID: "e5cf9f31-04dc-4d03-a8e2-bf38bcaf3830"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:05:54.526272 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:54.526229 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97e3afca-28dd-4f22-89bf-14245acc4544-tls-certs" pod "97e3afca-28dd-4f22-89bf-14245acc4544" (UID: "97e3afca-28dd-4f22-89bf-14245acc4544"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 05:05:54.624750 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:54.624677 4509 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rfcn\" (UniqueName: \"kubernetes.io/projected/97e3afca-28dd-4f22-89bf-14245acc4544-kube-api-access-5rfcn\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:05:54.624750 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:54.624702 4509 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phcsq\" (UniqueName: \"kubernetes.io/projected/e5cf9f31-04dc-4d03-a8e2-bf38bcaf3830-kube-api-access-phcsq\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:05:54.624750 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:54.624712 4509 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5cf9f31-04dc-4d03-a8e2-bf38bcaf3830-tls-certs\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:05:54.624750 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:54.624721 4509 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97e3afca-28dd-4f22-89bf-14245acc4544-dshm\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:05:54.624750 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:54.624729 4509 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5cf9f31-04dc-4d03-a8e2-bf38bcaf3830-dshm\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:05:54.624750 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:54.624737 4509 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5cf9f31-04dc-4d03-a8e2-bf38bcaf3830-home\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:05:54.624750 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:54.624743 4509 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97e3afca-28dd-4f22-89bf-14245acc4544-tls-certs\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:05:54.624750 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:54.624750 4509 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97e3afca-28dd-4f22-89bf-14245acc4544-tmp-dir\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:05:54.625059 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:54.624758 4509 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5cf9f31-04dc-4d03-a8e2-bf38bcaf3830-tmp-dir\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:05:54.625059 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:54.624765 4509 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97e3afca-28dd-4f22-89bf-14245acc4544-model-cache\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:05:54.625059 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:54.624773 4509 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97e3afca-28dd-4f22-89bf-14245acc4544-home\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:05:55.252012 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:55.251981 4509 generic.go:356] "Generic (PLEG): container finished" podID="e5cf9f31-04dc-4d03-a8e2-bf38bcaf3830" containerID="73f05a18ee37cace663307b405a11529613e33d011290d4755ef03d1d3cd6216" exitCode=0 Nov 04 05:05:55.252436 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:55.252030 4509 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-cedfca/distinct-w6-cedfca-kserve-54897db797-fsx45" Nov 04 05:05:55.252436 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:55.252066 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-cedfca/distinct-w6-cedfca-kserve-54897db797-fsx45" event={"ID":"e5cf9f31-04dc-4d03-a8e2-bf38bcaf3830","Type":"ContainerDied","Data":"73f05a18ee37cace663307b405a11529613e33d011290d4755ef03d1d3cd6216"} Nov 04 05:05:55.252436 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:55.252103 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-cedfca/distinct-w6-cedfca-kserve-54897db797-fsx45" event={"ID":"e5cf9f31-04dc-4d03-a8e2-bf38bcaf3830","Type":"ContainerDied","Data":"38c511b43d4f6efc2d20b1ffa4032ab1dd522f221958f0822dd37f2e78c2809d"} Nov 04 05:05:55.252436 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:55.252120 4509 scope.go:122] "RemoveContainer" containerID="73f05a18ee37cace663307b405a11529613e33d011290d4755ef03d1d3cd6216" Nov 04 05:05:55.253472 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:55.253458 4509 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-cedfca/distinct-2-w6-cedfca-kserve-5465b749bb-g2tkm" Nov 04 05:05:55.253534 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:55.253479 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-cedfca/distinct-2-w6-cedfca-kserve-5465b749bb-g2tkm" event={"ID":"97e3afca-28dd-4f22-89bf-14245acc4544","Type":"ContainerDied","Data":"c62e34d7083533c9d1ce92d8a43f5daa29af4737c0ce4696d134aea343cd6528"} Nov 04 05:05:55.254951 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:55.254929 4509 generic.go:356] "Generic (PLEG): container finished" podID="cea7867b-14f2-4967-8467-1a4d4e698f10" containerID="7f44fc3a1b6b633d1d9773b9a6c0bf26610cb463720e7305cd959a53fe32a118" exitCode=0 Nov 04 05:05:55.255054 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:55.254968 4509 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-cedfca/simulator-w6-cedfca-kserve-687878f579-n9vnt" Nov 04 05:05:55.255054 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:55.255005 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-cedfca/simulator-w6-cedfca-kserve-687878f579-n9vnt" event={"ID":"cea7867b-14f2-4967-8467-1a4d4e698f10","Type":"ContainerDied","Data":"7f44fc3a1b6b633d1d9773b9a6c0bf26610cb463720e7305cd959a53fe32a118"} Nov 04 05:05:55.255054 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:55.255034 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-cedfca/simulator-w6-cedfca-kserve-687878f579-n9vnt" event={"ID":"cea7867b-14f2-4967-8467-1a4d4e698f10","Type":"ContainerDied","Data":"f598d0a1ef07db779fed61ce3592fb7303607661f6b7a0342094df910f60d8dc"} Nov 04 05:05:55.263215 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:55.263192 4509 scope.go:122] "RemoveContainer" containerID="73f05a18ee37cace663307b405a11529613e33d011290d4755ef03d1d3cd6216" Nov 04 05:05:55.263577 ip-10-0-133-180 kubenswrapper[4509]: E1104 05:05:55.263555 4509 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73f05a18ee37cace663307b405a11529613e33d011290d4755ef03d1d3cd6216\": container with ID starting with 73f05a18ee37cace663307b405a11529613e33d011290d4755ef03d1d3cd6216 not found: ID does not exist" containerID="73f05a18ee37cace663307b405a11529613e33d011290d4755ef03d1d3cd6216" Nov 04 05:05:55.263671 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:55.263584 4509 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73f05a18ee37cace663307b405a11529613e33d011290d4755ef03d1d3cd6216"} err="failed to get container status \"73f05a18ee37cace663307b405a11529613e33d011290d4755ef03d1d3cd6216\": rpc error: code = NotFound desc = could not find container \"73f05a18ee37cace663307b405a11529613e33d011290d4755ef03d1d3cd6216\": container with ID starting with 73f05a18ee37cace663307b405a11529613e33d011290d4755ef03d1d3cd6216 not found: ID does not exist" Nov 04 05:05:55.263671 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:55.263608 4509 scope.go:122] "RemoveContainer" containerID="ed6811d301fd1b7cb6e597ec418ba0765deb72b2154eb05e591561d7a6fc4ae1" Nov 04 05:05:55.272285 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:55.272266 4509 scope.go:122] "RemoveContainer" containerID="7f44fc3a1b6b633d1d9773b9a6c0bf26610cb463720e7305cd959a53fe32a118" Nov 04 05:05:55.280170 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:55.280154 4509 scope.go:122] "RemoveContainer" containerID="7f44fc3a1b6b633d1d9773b9a6c0bf26610cb463720e7305cd959a53fe32a118" Nov 04 05:05:55.280497 ip-10-0-133-180 kubenswrapper[4509]: E1104 05:05:55.280476 4509 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f44fc3a1b6b633d1d9773b9a6c0bf26610cb463720e7305cd959a53fe32a118\": container with ID starting with 7f44fc3a1b6b633d1d9773b9a6c0bf26610cb463720e7305cd959a53fe32a118 not found: ID does not exist" containerID="7f44fc3a1b6b633d1d9773b9a6c0bf26610cb463720e7305cd959a53fe32a118" Nov 04 05:05:55.280575 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:55.280506 4509 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f44fc3a1b6b633d1d9773b9a6c0bf26610cb463720e7305cd959a53fe32a118"} err="failed to get container status \"7f44fc3a1b6b633d1d9773b9a6c0bf26610cb463720e7305cd959a53fe32a118\": rpc error: code = NotFound desc = could not find container \"7f44fc3a1b6b633d1d9773b9a6c0bf26610cb463720e7305cd959a53fe32a118\": container with ID starting with 7f44fc3a1b6b633d1d9773b9a6c0bf26610cb463720e7305cd959a53fe32a118 not found: ID does not exist" Nov 04 05:05:55.395360 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:55.395338 4509 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dc147b4-c8x2x" Nov 04 05:05:55.417873 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:55.417845 4509 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-cedfca/distinct-w6-cedfca-kserve-54897db797-fsx45"] Nov 04 05:05:55.431509 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:55.431484 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c76c7ef3-5e20-413d-9528-c0264ba54975-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c76c7ef3-5e20-413d-9528-c0264ba54975-maas-api-openshift-ca\") pod \"c76c7ef3-5e20-413d-9528-c0264ba54975\" (UID: \"c76c7ef3-5e20-413d-9528-c0264ba54975\") " Nov 04 05:05:55.431604 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:55.431519 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c76c7ef3-5e20-413d-9528-c0264ba54975-kube-api-access-k7jj4\" (UniqueName: \"kubernetes.io/projected/c76c7ef3-5e20-413d-9528-c0264ba54975-kube-api-access-k7jj4\") pod \"c76c7ef3-5e20-413d-9528-c0264ba54975\" (UID: \"c76c7ef3-5e20-413d-9528-c0264ba54975\") " Nov 04 05:05:55.431604 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:55.431575 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c76c7ef3-5e20-413d-9528-c0264ba54975-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c76c7ef3-5e20-413d-9528-c0264ba54975-maas-api-xks-ca\") pod \"c76c7ef3-5e20-413d-9528-c0264ba54975\" (UID: \"c76c7ef3-5e20-413d-9528-c0264ba54975\") " Nov 04 05:05:55.431910 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:55.431884 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c76c7ef3-5e20-413d-9528-c0264ba54975-maas-api-openshift-ca" pod "c76c7ef3-5e20-413d-9528-c0264ba54975" (UID: "c76c7ef3-5e20-413d-9528-c0264ba54975"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 05:05:55.433696 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:55.433678 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c76c7ef3-5e20-413d-9528-c0264ba54975-kube-api-access-k7jj4" pod "c76c7ef3-5e20-413d-9528-c0264ba54975" (UID: "c76c7ef3-5e20-413d-9528-c0264ba54975"). InnerVolumeSpecName "kube-api-access-k7jj4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 05:05:55.433784 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:55.433732 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c76c7ef3-5e20-413d-9528-c0264ba54975-maas-api-xks-ca" pod "c76c7ef3-5e20-413d-9528-c0264ba54975" (UID: "c76c7ef3-5e20-413d-9528-c0264ba54975"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 05:05:55.467528 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:55.467498 4509 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-cedfca/distinct-w6-cedfca-kserve-54897db797-fsx45"] Nov 04 05:05:55.530429 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:55.530378 4509 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-cedfca/simulator-w6-cedfca-kserve-687878f579-n9vnt"] Nov 04 05:05:55.532358 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:55.532340 4509 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7jj4\" (UniqueName: \"kubernetes.io/projected/c76c7ef3-5e20-413d-9528-c0264ba54975-kube-api-access-k7jj4\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:05:55.532423 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:55.532362 4509 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c76c7ef3-5e20-413d-9528-c0264ba54975-maas-api-xks-ca\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:05:55.532423 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:55.532372 4509 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c76c7ef3-5e20-413d-9528-c0264ba54975-maas-api-openshift-ca\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:05:55.537863 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:55.537841 4509 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-cedfca/simulator-w6-cedfca-kserve-687878f579-n9vnt"] Nov 04 05:05:55.646395 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:55.646369 4509 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-cedfca/distinct-2-w6-cedfca-kserve-5465b749bb-g2tkm"] Nov 04 05:05:55.649010 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:55.648988 4509 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-cedfca/distinct-2-w6-cedfca-kserve-5465b749bb-g2tkm"] Nov 04 05:05:56.260966 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:56.260939 4509 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97e3afca-28dd-4f22-89bf-14245acc4544" path="/var/lib/kubelet/pods/97e3afca-28dd-4f22-89bf-14245acc4544/volumes" Nov 04 05:05:56.261408 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:56.261387 4509 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cea7867b-14f2-4967-8467-1a4d4e698f10" path="/var/lib/kubelet/pods/cea7867b-14f2-4967-8467-1a4d4e698f10/volumes" Nov 04 05:05:56.261844 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:56.261826 4509 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5cf9f31-04dc-4d03-a8e2-bf38bcaf3830" path="/var/lib/kubelet/pods/e5cf9f31-04dc-4d03-a8e2-bf38bcaf3830/volumes" Nov 04 05:05:56.262257 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:56.262235 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dc147b4-c8x2x" event={"ID":"c76c7ef3-5e20-413d-9528-c0264ba54975","Type":"ContainerDied","Data":"ffa7e62e462642030e01deda550da4982d7f19383cd687d355d2f63b3f92a36c"} Nov 04 05:05:56.262403 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:56.262272 4509 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ffa7e62e462642030e01deda550da4982d7f19383cd687d355d2f63b3f92a36c" Nov 04 05:05:56.262403 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:56.262273 4509 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dc147b4-c8x2x" Nov 04 05:05:57.280343 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:57.280249 4509 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w6-cedfca-c7c8f5b7d-hdqcw" podUID="c6376390-553a-41f1-87d3-31e63d9e929d" containerName="payload-processing" containerID="cri-o://41fdd7a11af2f438301f249dec8733847f297b94e70fb208b7d4258910600bdf" gracePeriod=25 Nov 04 05:05:57.392092 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:57.392030 4509 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-cedfca-67b95cc659-6v86t" podUID="5f041060-cb24-4d0a-be92-aabd9a536c92" containerName="payload-pre-processing" containerID="cri-o://d683c478c107507a70e1739f169c06fcda658c6ea0c5ab0a729b212b953e95df" gracePeriod=25 Nov 04 05:05:57.528029 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:57.528003 4509 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-cedfca-c7c8f5b7d-hdqcw" Nov 04 05:05:57.550290 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:57.550211 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6376390-553a-41f1-87d3-31e63d9e929d-kube-api-access-q7xcg\" (UniqueName: \"kubernetes.io/projected/c6376390-553a-41f1-87d3-31e63d9e929d-kube-api-access-q7xcg\") pod \"c6376390-553a-41f1-87d3-31e63d9e929d\" (UID: \"c6376390-553a-41f1-87d3-31e63d9e929d\") " Nov 04 05:05:57.550450 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:57.550378 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6376390-553a-41f1-87d3-31e63d9e929d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6376390-553a-41f1-87d3-31e63d9e929d-plugins-config-volume\") pod \"c6376390-553a-41f1-87d3-31e63d9e929d\" (UID: \"c6376390-553a-41f1-87d3-31e63d9e929d\") " Nov 04 05:05:57.550778 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:57.550758 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6376390-553a-41f1-87d3-31e63d9e929d-plugins-config-volume" pod "c6376390-553a-41f1-87d3-31e63d9e929d" (UID: "c6376390-553a-41f1-87d3-31e63d9e929d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 05:05:57.552668 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:57.552639 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6376390-553a-41f1-87d3-31e63d9e929d-kube-api-access-q7xcg" pod "c6376390-553a-41f1-87d3-31e63d9e929d" (UID: "c6376390-553a-41f1-87d3-31e63d9e929d"). InnerVolumeSpecName "kube-api-access-q7xcg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 05:05:57.651101 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:57.651064 4509 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6376390-553a-41f1-87d3-31e63d9e929d-plugins-config-volume\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:05:57.651101 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:57.651096 4509 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7xcg\" (UniqueName: \"kubernetes.io/projected/c6376390-553a-41f1-87d3-31e63d9e929d-kube-api-access-q7xcg\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:05:58.278430 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:58.278393 4509 generic.go:356] "Generic (PLEG): container finished" podID="c6376390-553a-41f1-87d3-31e63d9e929d" containerID="41fdd7a11af2f438301f249dec8733847f297b94e70fb208b7d4258910600bdf" exitCode=0 Nov 04 05:05:58.278573 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:58.278450 4509 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-cedfca-c7c8f5b7d-hdqcw" Nov 04 05:05:58.278573 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:58.278469 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-cedfca-c7c8f5b7d-hdqcw" event={"ID":"c6376390-553a-41f1-87d3-31e63d9e929d","Type":"ContainerDied","Data":"41fdd7a11af2f438301f249dec8733847f297b94e70fb208b7d4258910600bdf"} Nov 04 05:05:58.278573 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:58.278533 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-cedfca-c7c8f5b7d-hdqcw" event={"ID":"c6376390-553a-41f1-87d3-31e63d9e929d","Type":"ContainerDied","Data":"f304bb19495c418193c8503ce5d19999b026ed2322502e61a5f7313b7d907790"} Nov 04 05:05:58.278573 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:58.278561 4509 scope.go:122] "RemoveContainer" containerID="41fdd7a11af2f438301f249dec8733847f297b94e70fb208b7d4258910600bdf" Nov 04 05:05:58.289067 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:58.288962 4509 scope.go:122] "RemoveContainer" containerID="41fdd7a11af2f438301f249dec8733847f297b94e70fb208b7d4258910600bdf" Nov 04 05:05:58.289338 ip-10-0-133-180 kubenswrapper[4509]: E1104 05:05:58.289293 4509 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41fdd7a11af2f438301f249dec8733847f297b94e70fb208b7d4258910600bdf\": container with ID starting with 41fdd7a11af2f438301f249dec8733847f297b94e70fb208b7d4258910600bdf not found: ID does not exist" containerID="41fdd7a11af2f438301f249dec8733847f297b94e70fb208b7d4258910600bdf" Nov 04 05:05:58.289385 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:58.289358 4509 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41fdd7a11af2f438301f249dec8733847f297b94e70fb208b7d4258910600bdf"} err="failed to get container status \"41fdd7a11af2f438301f249dec8733847f297b94e70fb208b7d4258910600bdf\": rpc error: code = NotFound desc = could not find container \"41fdd7a11af2f438301f249dec8733847f297b94e70fb208b7d4258910600bdf\": container with ID starting with 41fdd7a11af2f438301f249dec8733847f297b94e70fb208b7d4258910600bdf not found: ID does not exist" Nov 04 05:05:58.442169 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:58.442139 4509 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-cedfca-c7c8f5b7d-hdqcw"] Nov 04 05:05:58.492035 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:05:58.491994 4509 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-cedfca-c7c8f5b7d-hdqcw"] Nov 04 05:06:00.261221 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:00.261189 4509 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6376390-553a-41f1-87d3-31e63d9e929d" path="/var/lib/kubelet/pods/c6376390-553a-41f1-87d3-31e63d9e929d/volumes" Nov 04 05:06:01.669876 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:01.669843 4509 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3a13485-chkkp"] Nov 04 05:06:01.670242 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:01.670213 4509 cpu_manager.go:414] "containerMap: removing container" podUID="c76c7ef3-5e20-413d-9528-c0264ba54975" containerName="revoke-keys" Nov 04 05:06:01.670242 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:01.670224 4509 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c76c7ef3-5e20-413d-9528-c0264ba54975" containerName="revoke-keys" Nov 04 05:06:01.670242 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:01.670235 4509 cpu_manager.go:414] "containerMap: removing container" podUID="97e3afca-28dd-4f22-89bf-14245acc4544" containerName="main" Nov 04 05:06:01.670242 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:01.670241 4509 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97e3afca-28dd-4f22-89bf-14245acc4544" containerName="main" Nov 04 05:06:01.670392 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:01.670253 4509 cpu_manager.go:414] "containerMap: removing container" podUID="cea7867b-14f2-4967-8467-1a4d4e698f10" containerName="main" Nov 04 05:06:01.670392 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:01.670259 4509 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cea7867b-14f2-4967-8467-1a4d4e698f10" containerName="main" Nov 04 05:06:01.670392 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:01.670270 4509 cpu_manager.go:414] "containerMap: removing container" podUID="c6376390-553a-41f1-87d3-31e63d9e929d" containerName="payload-processing" Nov 04 05:06:01.670392 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:01.670276 4509 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6376390-553a-41f1-87d3-31e63d9e929d" containerName="payload-processing" Nov 04 05:06:01.670392 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:01.670282 4509 cpu_manager.go:414] "containerMap: removing container" podUID="e5cf9f31-04dc-4d03-a8e2-bf38bcaf3830" containerName="main" Nov 04 05:06:01.670392 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:01.670286 4509 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5cf9f31-04dc-4d03-a8e2-bf38bcaf3830" containerName="main" Nov 04 05:06:01.670392 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:01.670379 4509 memory_manager.go:365] "RemoveStaleState removing state" podUID="c76c7ef3-5e20-413d-9528-c0264ba54975" containerName="revoke-keys" Nov 04 05:06:01.670392 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:01.670388 4509 memory_manager.go:365] "RemoveStaleState removing state" podUID="97e3afca-28dd-4f22-89bf-14245acc4544" containerName="main" Nov 04 05:06:01.670392 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:01.670395 4509 memory_manager.go:365] "RemoveStaleState removing state" podUID="cea7867b-14f2-4967-8467-1a4d4e698f10" containerName="main" Nov 04 05:06:01.670635 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:01.670402 4509 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6376390-553a-41f1-87d3-31e63d9e929d" containerName="payload-processing" Nov 04 05:06:01.670635 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:01.670407 4509 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5cf9f31-04dc-4d03-a8e2-bf38bcaf3830" containerName="main" Nov 04 05:06:01.674771 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:01.674752 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3a13485-chkkp" Nov 04 05:06:01.731787 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:01.731762 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3a13485-chkkp"] Nov 04 05:06:01.778225 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:01.778197 4509 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 05:06:01.778225 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:01.778197 4509 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 05:06:01.778462 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:01.778205 4509 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 05:06:01.778462 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:01.778212 4509 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-krgvj\"" Nov 04 05:06:01.794432 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:01.794409 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83294360-4718-4139-8287-8ec8ee6f72ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c3a13485-chkkp\" (UID: \"83294360-4718-4139-8287-8ec8ee6f72ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3a13485-chkkp" Nov 04 05:06:01.794569 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:01.794440 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ch7vz\" (UniqueName: \"kubernetes.io/projected/83294360-4718-4139-8287-8ec8ee6f72ce-kube-api-access-ch7vz\") pod \"maas-api-revoke-sub-c3a13485-chkkp\" (UID: \"83294360-4718-4139-8287-8ec8ee6f72ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3a13485-chkkp" Nov 04 05:06:01.794614 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:01.794574 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83294360-4718-4139-8287-8ec8ee6f72ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c3a13485-chkkp\" (UID: \"83294360-4718-4139-8287-8ec8ee6f72ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3a13485-chkkp" Nov 04 05:06:01.895936 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:01.895895 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83294360-4718-4139-8287-8ec8ee6f72ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c3a13485-chkkp\" (UID: \"83294360-4718-4139-8287-8ec8ee6f72ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3a13485-chkkp" Nov 04 05:06:01.896096 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:01.895972 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83294360-4718-4139-8287-8ec8ee6f72ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c3a13485-chkkp\" (UID: \"83294360-4718-4139-8287-8ec8ee6f72ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3a13485-chkkp" Nov 04 05:06:01.896096 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:01.896002 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ch7vz\" (UniqueName: \"kubernetes.io/projected/83294360-4718-4139-8287-8ec8ee6f72ce-kube-api-access-ch7vz\") pod \"maas-api-revoke-sub-c3a13485-chkkp\" (UID: \"83294360-4718-4139-8287-8ec8ee6f72ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3a13485-chkkp" Nov 04 05:06:01.896748 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:01.896717 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83294360-4718-4139-8287-8ec8ee6f72ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c3a13485-chkkp\" (UID: \"83294360-4718-4139-8287-8ec8ee6f72ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3a13485-chkkp" Nov 04 05:06:01.898550 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:01.898527 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83294360-4718-4139-8287-8ec8ee6f72ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c3a13485-chkkp\" (UID: \"83294360-4718-4139-8287-8ec8ee6f72ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3a13485-chkkp" Nov 04 05:06:01.952610 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:01.952549 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ch7vz\" (UniqueName: \"kubernetes.io/projected/83294360-4718-4139-8287-8ec8ee6f72ce-kube-api-access-ch7vz\") pod \"maas-api-revoke-sub-c3a13485-chkkp\" (UID: \"83294360-4718-4139-8287-8ec8ee6f72ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3a13485-chkkp" Nov 04 05:06:01.988126 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:01.988098 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3a13485-chkkp" Nov 04 05:06:02.360494 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:02.360470 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3a13485-chkkp"] Nov 04 05:06:02.362187 ip-10-0-133-180 kubenswrapper[4509]: W1104 05:06:02.362159 4509 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83294360_4718_4139_8287_8ec8ee6f72ce.slice/crio-6e1079b80cf9ff34741643be5df913c425a8b04af4163adbde076fe44a6c9867 WatchSource:0}: Error finding container 6e1079b80cf9ff34741643be5df913c425a8b04af4163adbde076fe44a6c9867: Status 404 returned error can't find the container with id 6e1079b80cf9ff34741643be5df913c425a8b04af4163adbde076fe44a6c9867 Nov 04 05:06:02.882527 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:02.882492 4509 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-4cce33-56666d7678-r88jh"] Nov 04 05:06:03.323755 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:03.323714 4509 generic.go:356] "Generic (PLEG): container finished" podID="83294360-4718-4139-8287-8ec8ee6f72ce" containerID="425f763cee363b897fcf8663442b935d05fbd8f06110f732807ef60b2a41bade" exitCode=0 Nov 04 05:06:03.323935 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:03.323837 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3a13485-chkkp" event={"ID":"83294360-4718-4139-8287-8ec8ee6f72ce","Type":"ContainerDied","Data":"425f763cee363b897fcf8663442b935d05fbd8f06110f732807ef60b2a41bade"} Nov 04 05:06:03.323935 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:03.323887 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3a13485-chkkp" event={"ID":"83294360-4718-4139-8287-8ec8ee6f72ce","Type":"ContainerStarted","Data":"6e1079b80cf9ff34741643be5df913c425a8b04af4163adbde076fe44a6c9867"} Nov 04 05:06:04.496012 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:04.495974 4509 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3a13485-chkkp" Nov 04 05:06:04.634070 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:04.633987 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83294360-4718-4139-8287-8ec8ee6f72ce-kube-api-access-ch7vz\" (UniqueName: \"kubernetes.io/projected/83294360-4718-4139-8287-8ec8ee6f72ce-kube-api-access-ch7vz\") pod \"83294360-4718-4139-8287-8ec8ee6f72ce\" (UID: \"83294360-4718-4139-8287-8ec8ee6f72ce\") " Nov 04 05:06:04.634218 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:04.634137 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/83294360-4718-4139-8287-8ec8ee6f72ce-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83294360-4718-4139-8287-8ec8ee6f72ce-maas-api-xks-ca\") pod \"83294360-4718-4139-8287-8ec8ee6f72ce\" (UID: \"83294360-4718-4139-8287-8ec8ee6f72ce\") " Nov 04 05:06:04.634218 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:04.634194 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83294360-4718-4139-8287-8ec8ee6f72ce-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83294360-4718-4139-8287-8ec8ee6f72ce-maas-api-openshift-ca\") pod \"83294360-4718-4139-8287-8ec8ee6f72ce\" (UID: \"83294360-4718-4139-8287-8ec8ee6f72ce\") " Nov 04 05:06:04.634868 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:04.634724 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83294360-4718-4139-8287-8ec8ee6f72ce-maas-api-openshift-ca" pod "83294360-4718-4139-8287-8ec8ee6f72ce" (UID: "83294360-4718-4139-8287-8ec8ee6f72ce"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 05:06:04.637441 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:04.637411 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83294360-4718-4139-8287-8ec8ee6f72ce-maas-api-xks-ca" pod "83294360-4718-4139-8287-8ec8ee6f72ce" (UID: "83294360-4718-4139-8287-8ec8ee6f72ce"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 05:06:04.637543 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:04.637417 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83294360-4718-4139-8287-8ec8ee6f72ce-kube-api-access-ch7vz" pod "83294360-4718-4139-8287-8ec8ee6f72ce" (UID: "83294360-4718-4139-8287-8ec8ee6f72ce"). InnerVolumeSpecName "kube-api-access-ch7vz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 05:06:04.735186 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:04.735148 4509 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ch7vz\" (UniqueName: \"kubernetes.io/projected/83294360-4718-4139-8287-8ec8ee6f72ce-kube-api-access-ch7vz\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:06:04.735186 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:04.735183 4509 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83294360-4718-4139-8287-8ec8ee6f72ce-maas-api-xks-ca\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:06:04.735440 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:04.735196 4509 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83294360-4718-4139-8287-8ec8ee6f72ce-maas-api-openshift-ca\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:06:05.336342 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:05.336292 4509 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3a13485-chkkp" Nov 04 05:06:05.336342 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:05.336311 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3a13485-chkkp" event={"ID":"83294360-4718-4139-8287-8ec8ee6f72ce","Type":"ContainerDied","Data":"6e1079b80cf9ff34741643be5df913c425a8b04af4163adbde076fe44a6c9867"} Nov 04 05:06:05.336591 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:05.336387 4509 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6e1079b80cf9ff34741643be5df913c425a8b04af4163adbde076fe44a6c9867" Nov 04 05:06:05.999907 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:05.999867 4509 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-cedfca-67b95cc659-6v86t" podUID="5f041060-cb24-4d0a-be92-aabd9a536c92" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 05:06:07.882992 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:07.882922 4509 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w5-4cce33-56666d7678-r88jh" podUID="2a16804b-60cc-46b5-9879-163bda7bd069" containerName="payload-processing" containerID="cri-o://9f5a4844648e5c09869687b1f548d63b756b2a51a6ac37f646cf789559c04cbb" gracePeriod=25 Nov 04 05:06:08.430844 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:08.430803 4509 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-4cce33/distinct-w5-4cce33-kserve-68bf9d9bb7-x62f2" podUID="c1afb6cd-a9c5-43b9-ae3b-9929840fe019" containerName="main" containerID="cri-o://23d7e72b6b5c50984a8aee1d10405a99f4d045dfb5933ebc93ea44560de5a49f" gracePeriod=45 Nov 04 05:06:08.432585 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:08.432546 4509 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-4cce33/premium-w5-4cce33-kserve-5875cf9fc6-9k7jz" podUID="0d9118e1-aff5-4fb2-8a56-78e3ee9fa3e4" containerName="main" containerID="cri-o://9aabeaf1e4937576824210ad25e559b17f36aaf258476162d5bfb21b1f1c65cc" gracePeriod=45 Nov 04 05:06:08.701356 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:08.701300 4509 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-4cce33/distinct-w5-4cce33-kserve-68bf9d9bb7-x62f2" Nov 04 05:06:08.714363 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:08.714338 4509 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-4cce33/premium-w5-4cce33-kserve-5875cf9fc6-9k7jz" Nov 04 05:06:08.773886 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:08.773846 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1afb6cd-a9c5-43b9-ae3b-9929840fe019-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1afb6cd-a9c5-43b9-ae3b-9929840fe019-model-cache\") pod \"c1afb6cd-a9c5-43b9-ae3b-9929840fe019\" (UID: \"c1afb6cd-a9c5-43b9-ae3b-9929840fe019\") " Nov 04 05:06:08.774131 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:08.773917 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d9118e1-aff5-4fb2-8a56-78e3ee9fa3e4-kube-api-access-85wqk\" (UniqueName: \"kubernetes.io/projected/0d9118e1-aff5-4fb2-8a56-78e3ee9fa3e4-kube-api-access-85wqk\") pod \"0d9118e1-aff5-4fb2-8a56-78e3ee9fa3e4\" (UID: \"0d9118e1-aff5-4fb2-8a56-78e3ee9fa3e4\") " Nov 04 05:06:08.774131 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:08.773961 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1afb6cd-a9c5-43b9-ae3b-9929840fe019-home\" (UniqueName: \"kubernetes.io/empty-dir/c1afb6cd-a9c5-43b9-ae3b-9929840fe019-home\") pod \"c1afb6cd-a9c5-43b9-ae3b-9929840fe019\" (UID: \"c1afb6cd-a9c5-43b9-ae3b-9929840fe019\") " Nov 04 05:06:08.774131 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:08.773999 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d9118e1-aff5-4fb2-8a56-78e3ee9fa3e4-tls-certs\" (UniqueName: \"kubernetes.io/secret/0d9118e1-aff5-4fb2-8a56-78e3ee9fa3e4-tls-certs\") pod \"0d9118e1-aff5-4fb2-8a56-78e3ee9fa3e4\" (UID: \"0d9118e1-aff5-4fb2-8a56-78e3ee9fa3e4\") " Nov 04 05:06:08.774131 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:08.774065 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d9118e1-aff5-4fb2-8a56-78e3ee9fa3e4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d9118e1-aff5-4fb2-8a56-78e3ee9fa3e4-dshm\") pod \"0d9118e1-aff5-4fb2-8a56-78e3ee9fa3e4\" (UID: \"0d9118e1-aff5-4fb2-8a56-78e3ee9fa3e4\") " Nov 04 05:06:08.774131 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:08.774094 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d9118e1-aff5-4fb2-8a56-78e3ee9fa3e4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d9118e1-aff5-4fb2-8a56-78e3ee9fa3e4-model-cache\") pod \"0d9118e1-aff5-4fb2-8a56-78e3ee9fa3e4\" (UID: \"0d9118e1-aff5-4fb2-8a56-78e3ee9fa3e4\") " Nov 04 05:06:08.774131 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:08.774128 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1afb6cd-a9c5-43b9-ae3b-9929840fe019-tls-certs\" (UniqueName: \"kubernetes.io/secret/c1afb6cd-a9c5-43b9-ae3b-9929840fe019-tls-certs\") pod \"c1afb6cd-a9c5-43b9-ae3b-9929840fe019\" (UID: \"c1afb6cd-a9c5-43b9-ae3b-9929840fe019\") " Nov 04 05:06:08.774465 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:08.774160 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1afb6cd-a9c5-43b9-ae3b-9929840fe019-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1afb6cd-a9c5-43b9-ae3b-9929840fe019-dshm\") pod \"c1afb6cd-a9c5-43b9-ae3b-9929840fe019\" (UID: \"c1afb6cd-a9c5-43b9-ae3b-9929840fe019\") " Nov 04 05:06:08.774465 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:08.774188 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d9118e1-aff5-4fb2-8a56-78e3ee9fa3e4-home\" (UniqueName: \"kubernetes.io/empty-dir/0d9118e1-aff5-4fb2-8a56-78e3ee9fa3e4-home\") pod \"0d9118e1-aff5-4fb2-8a56-78e3ee9fa3e4\" (UID: \"0d9118e1-aff5-4fb2-8a56-78e3ee9fa3e4\") " Nov 04 05:06:08.774465 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:08.774228 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d9118e1-aff5-4fb2-8a56-78e3ee9fa3e4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d9118e1-aff5-4fb2-8a56-78e3ee9fa3e4-tmp-dir\") pod \"0d9118e1-aff5-4fb2-8a56-78e3ee9fa3e4\" (UID: \"0d9118e1-aff5-4fb2-8a56-78e3ee9fa3e4\") " Nov 04 05:06:08.774465 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:08.774253 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1afb6cd-a9c5-43b9-ae3b-9929840fe019-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1afb6cd-a9c5-43b9-ae3b-9929840fe019-tmp-dir\") pod \"c1afb6cd-a9c5-43b9-ae3b-9929840fe019\" (UID: \"c1afb6cd-a9c5-43b9-ae3b-9929840fe019\") " Nov 04 05:06:08.774465 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:08.774355 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1afb6cd-a9c5-43b9-ae3b-9929840fe019-kube-api-access-gnrsv\" (UniqueName: \"kubernetes.io/projected/c1afb6cd-a9c5-43b9-ae3b-9929840fe019-kube-api-access-gnrsv\") pod \"c1afb6cd-a9c5-43b9-ae3b-9929840fe019\" (UID: \"c1afb6cd-a9c5-43b9-ae3b-9929840fe019\") " Nov 04 05:06:08.776545 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:08.775188 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1afb6cd-a9c5-43b9-ae3b-9929840fe019-home" pod "c1afb6cd-a9c5-43b9-ae3b-9929840fe019" (UID: "c1afb6cd-a9c5-43b9-ae3b-9929840fe019"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:06:08.776545 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:08.775220 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d9118e1-aff5-4fb2-8a56-78e3ee9fa3e4-model-cache" pod "0d9118e1-aff5-4fb2-8a56-78e3ee9fa3e4" (UID: "0d9118e1-aff5-4fb2-8a56-78e3ee9fa3e4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:06:08.776545 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:08.775529 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d9118e1-aff5-4fb2-8a56-78e3ee9fa3e4-tmp-dir" pod "0d9118e1-aff5-4fb2-8a56-78e3ee9fa3e4" (UID: "0d9118e1-aff5-4fb2-8a56-78e3ee9fa3e4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:06:08.776545 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:08.776065 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d9118e1-aff5-4fb2-8a56-78e3ee9fa3e4-home" pod "0d9118e1-aff5-4fb2-8a56-78e3ee9fa3e4" (UID: "0d9118e1-aff5-4fb2-8a56-78e3ee9fa3e4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:06:08.776545 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:08.776135 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1afb6cd-a9c5-43b9-ae3b-9929840fe019-tmp-dir" pod "c1afb6cd-a9c5-43b9-ae3b-9929840fe019" (UID: "c1afb6cd-a9c5-43b9-ae3b-9929840fe019"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:06:08.777203 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:08.777163 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1afb6cd-a9c5-43b9-ae3b-9929840fe019-model-cache" pod "c1afb6cd-a9c5-43b9-ae3b-9929840fe019" (UID: "c1afb6cd-a9c5-43b9-ae3b-9929840fe019"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:06:08.778311 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:08.778276 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d9118e1-aff5-4fb2-8a56-78e3ee9fa3e4-dshm" pod "0d9118e1-aff5-4fb2-8a56-78e3ee9fa3e4" (UID: "0d9118e1-aff5-4fb2-8a56-78e3ee9fa3e4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:06:08.779095 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:08.779069 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d9118e1-aff5-4fb2-8a56-78e3ee9fa3e4-tls-certs" pod "0d9118e1-aff5-4fb2-8a56-78e3ee9fa3e4" (UID: "0d9118e1-aff5-4fb2-8a56-78e3ee9fa3e4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 05:06:08.779209 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:08.779103 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1afb6cd-a9c5-43b9-ae3b-9929840fe019-kube-api-access-gnrsv" pod "c1afb6cd-a9c5-43b9-ae3b-9929840fe019" (UID: "c1afb6cd-a9c5-43b9-ae3b-9929840fe019"). InnerVolumeSpecName "kube-api-access-gnrsv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 05:06:08.779606 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:08.779579 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d9118e1-aff5-4fb2-8a56-78e3ee9fa3e4-kube-api-access-85wqk" pod "0d9118e1-aff5-4fb2-8a56-78e3ee9fa3e4" (UID: "0d9118e1-aff5-4fb2-8a56-78e3ee9fa3e4"). InnerVolumeSpecName "kube-api-access-85wqk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 05:06:08.779780 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:08.779760 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1afb6cd-a9c5-43b9-ae3b-9929840fe019-tls-certs" pod "c1afb6cd-a9c5-43b9-ae3b-9929840fe019" (UID: "c1afb6cd-a9c5-43b9-ae3b-9929840fe019"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 05:06:08.780769 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:08.780745 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1afb6cd-a9c5-43b9-ae3b-9929840fe019-dshm" pod "c1afb6cd-a9c5-43b9-ae3b-9929840fe019" (UID: "c1afb6cd-a9c5-43b9-ae3b-9929840fe019"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:06:08.875881 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:08.875840 4509 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d9118e1-aff5-4fb2-8a56-78e3ee9fa3e4-tls-certs\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:06:08.875881 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:08.875868 4509 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d9118e1-aff5-4fb2-8a56-78e3ee9fa3e4-dshm\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:06:08.875881 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:08.875879 4509 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d9118e1-aff5-4fb2-8a56-78e3ee9fa3e4-model-cache\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:06:08.875881 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:08.875889 4509 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1afb6cd-a9c5-43b9-ae3b-9929840fe019-tls-certs\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:06:08.876232 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:08.875898 4509 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1afb6cd-a9c5-43b9-ae3b-9929840fe019-dshm\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:06:08.876232 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:08.875906 4509 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d9118e1-aff5-4fb2-8a56-78e3ee9fa3e4-home\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:06:08.876232 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:08.875913 4509 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d9118e1-aff5-4fb2-8a56-78e3ee9fa3e4-tmp-dir\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:06:08.876232 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:08.875920 4509 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1afb6cd-a9c5-43b9-ae3b-9929840fe019-tmp-dir\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:06:08.876232 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:08.875928 4509 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gnrsv\" (UniqueName: \"kubernetes.io/projected/c1afb6cd-a9c5-43b9-ae3b-9929840fe019-kube-api-access-gnrsv\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:06:08.876232 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:08.875937 4509 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1afb6cd-a9c5-43b9-ae3b-9929840fe019-model-cache\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:06:08.876232 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:08.875945 4509 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85wqk\" (UniqueName: \"kubernetes.io/projected/0d9118e1-aff5-4fb2-8a56-78e3ee9fa3e4-kube-api-access-85wqk\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:06:08.876232 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:08.875953 4509 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1afb6cd-a9c5-43b9-ae3b-9929840fe019-home\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:06:09.370294 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:09.370260 4509 generic.go:356] "Generic (PLEG): container finished" podID="c1afb6cd-a9c5-43b9-ae3b-9929840fe019" containerID="23d7e72b6b5c50984a8aee1d10405a99f4d045dfb5933ebc93ea44560de5a49f" exitCode=0 Nov 04 05:06:09.370731 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:09.370306 4509 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-4cce33/distinct-w5-4cce33-kserve-68bf9d9bb7-x62f2" Nov 04 05:06:09.370731 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:09.370347 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-4cce33/distinct-w5-4cce33-kserve-68bf9d9bb7-x62f2" event={"ID":"c1afb6cd-a9c5-43b9-ae3b-9929840fe019","Type":"ContainerDied","Data":"23d7e72b6b5c50984a8aee1d10405a99f4d045dfb5933ebc93ea44560de5a49f"} Nov 04 05:06:09.370731 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:09.370395 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-4cce33/distinct-w5-4cce33-kserve-68bf9d9bb7-x62f2" event={"ID":"c1afb6cd-a9c5-43b9-ae3b-9929840fe019","Type":"ContainerDied","Data":"b9892b4933cd991ced00faac6931f1eef09b4bd95cf49c87755325a54f3bf218"} Nov 04 05:06:09.370731 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:09.370413 4509 scope.go:122] "RemoveContainer" containerID="23d7e72b6b5c50984a8aee1d10405a99f4d045dfb5933ebc93ea44560de5a49f" Nov 04 05:06:09.372339 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:09.372305 4509 generic.go:356] "Generic (PLEG): container finished" podID="0d9118e1-aff5-4fb2-8a56-78e3ee9fa3e4" containerID="9aabeaf1e4937576824210ad25e559b17f36aaf258476162d5bfb21b1f1c65cc" exitCode=0 Nov 04 05:06:09.372459 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:09.372348 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-4cce33/premium-w5-4cce33-kserve-5875cf9fc6-9k7jz" event={"ID":"0d9118e1-aff5-4fb2-8a56-78e3ee9fa3e4","Type":"ContainerDied","Data":"9aabeaf1e4937576824210ad25e559b17f36aaf258476162d5bfb21b1f1c65cc"} Nov 04 05:06:09.372546 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:09.372501 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-4cce33/premium-w5-4cce33-kserve-5875cf9fc6-9k7jz" event={"ID":"0d9118e1-aff5-4fb2-8a56-78e3ee9fa3e4","Type":"ContainerDied","Data":"9e9a3584deb1937e8b8b067aa39fbb87aa8e8fb5acfd53b42921d6e223d823c2"} Nov 04 05:06:09.372546 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:09.372377 4509 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-4cce33/premium-w5-4cce33-kserve-5875cf9fc6-9k7jz" Nov 04 05:06:09.381873 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:09.381854 4509 scope.go:122] "RemoveContainer" containerID="23d7e72b6b5c50984a8aee1d10405a99f4d045dfb5933ebc93ea44560de5a49f" Nov 04 05:06:09.382219 ip-10-0-133-180 kubenswrapper[4509]: E1104 05:06:09.382194 4509 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23d7e72b6b5c50984a8aee1d10405a99f4d045dfb5933ebc93ea44560de5a49f\": container with ID starting with 23d7e72b6b5c50984a8aee1d10405a99f4d045dfb5933ebc93ea44560de5a49f not found: ID does not exist" containerID="23d7e72b6b5c50984a8aee1d10405a99f4d045dfb5933ebc93ea44560de5a49f" Nov 04 05:06:09.382374 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:09.382227 4509 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23d7e72b6b5c50984a8aee1d10405a99f4d045dfb5933ebc93ea44560de5a49f"} err="failed to get container status \"23d7e72b6b5c50984a8aee1d10405a99f4d045dfb5933ebc93ea44560de5a49f\": rpc error: code = NotFound desc = could not find container \"23d7e72b6b5c50984a8aee1d10405a99f4d045dfb5933ebc93ea44560de5a49f\": container with ID starting with 23d7e72b6b5c50984a8aee1d10405a99f4d045dfb5933ebc93ea44560de5a49f not found: ID does not exist" Nov 04 05:06:09.382374 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:09.382244 4509 scope.go:122] "RemoveContainer" containerID="9aabeaf1e4937576824210ad25e559b17f36aaf258476162d5bfb21b1f1c65cc" Nov 04 05:06:09.391266 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:09.391247 4509 scope.go:122] "RemoveContainer" containerID="9aabeaf1e4937576824210ad25e559b17f36aaf258476162d5bfb21b1f1c65cc" Nov 04 05:06:09.391592 ip-10-0-133-180 kubenswrapper[4509]: E1104 05:06:09.391574 4509 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9aabeaf1e4937576824210ad25e559b17f36aaf258476162d5bfb21b1f1c65cc\": container with ID starting with 9aabeaf1e4937576824210ad25e559b17f36aaf258476162d5bfb21b1f1c65cc not found: ID does not exist" containerID="9aabeaf1e4937576824210ad25e559b17f36aaf258476162d5bfb21b1f1c65cc" Nov 04 05:06:09.391658 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:09.391601 4509 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9aabeaf1e4937576824210ad25e559b17f36aaf258476162d5bfb21b1f1c65cc"} err="failed to get container status \"9aabeaf1e4937576824210ad25e559b17f36aaf258476162d5bfb21b1f1c65cc\": rpc error: code = NotFound desc = could not find container \"9aabeaf1e4937576824210ad25e559b17f36aaf258476162d5bfb21b1f1c65cc\": container with ID starting with 9aabeaf1e4937576824210ad25e559b17f36aaf258476162d5bfb21b1f1c65cc not found: ID does not exist" Nov 04 05:06:09.534984 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:09.534952 4509 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-4cce33/premium-w5-4cce33-kserve-5875cf9fc6-9k7jz"] Nov 04 05:06:09.541211 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:09.541183 4509 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-4cce33/premium-w5-4cce33-kserve-5875cf9fc6-9k7jz"] Nov 04 05:06:09.650805 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:09.650732 4509 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-4cce33/distinct-w5-4cce33-kserve-68bf9d9bb7-x62f2"] Nov 04 05:06:09.654861 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:09.654833 4509 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-4cce33/distinct-w5-4cce33-kserve-68bf9d9bb7-x62f2"] Nov 04 05:06:09.965897 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:09.965863 4509 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-4cce33-56666d7678-r88jh" podUID="2a16804b-60cc-46b5-9879-163bda7bd069" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 05:06:10.260830 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:10.260750 4509 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d9118e1-aff5-4fb2-8a56-78e3ee9fa3e4" path="/var/lib/kubelet/pods/0d9118e1-aff5-4fb2-8a56-78e3ee9fa3e4/volumes" Nov 04 05:06:10.261167 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:10.261153 4509 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1afb6cd-a9c5-43b9-ae3b-9929840fe019" path="/var/lib/kubelet/pods/c1afb6cd-a9c5-43b9-ae3b-9929840fe019/volumes" Nov 04 05:06:10.867974 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:10.867939 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-42dnm"] Nov 04 05:06:14.456269 ip-10-0-133-180 kubenswrapper[4509]: E1104 05:06:14.456227 4509 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmdhl" containerName="extractor" Nov 04 05:06:15.999588 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:15.999545 4509 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-cedfca-67b95cc659-6v86t" podUID="5f041060-cb24-4d0a-be92-aabd9a536c92" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 05:06:17.644391 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:17.644359 4509 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-cedfca-67b95cc659-6v86t" Nov 04 05:06:17.784634 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:17.784602 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f041060-cb24-4d0a-be92-aabd9a536c92-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f041060-cb24-4d0a-be92-aabd9a536c92-plugins-config-volume\") pod \"5f041060-cb24-4d0a-be92-aabd9a536c92\" (UID: \"5f041060-cb24-4d0a-be92-aabd9a536c92\") " Nov 04 05:06:17.784842 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:17.784703 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f041060-cb24-4d0a-be92-aabd9a536c92-kube-api-access-2vs8t\" (UniqueName: \"kubernetes.io/projected/5f041060-cb24-4d0a-be92-aabd9a536c92-kube-api-access-2vs8t\") pod \"5f041060-cb24-4d0a-be92-aabd9a536c92\" (UID: \"5f041060-cb24-4d0a-be92-aabd9a536c92\") " Nov 04 05:06:17.785063 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:17.785039 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f041060-cb24-4d0a-be92-aabd9a536c92-plugins-config-volume" pod "5f041060-cb24-4d0a-be92-aabd9a536c92" (UID: "5f041060-cb24-4d0a-be92-aabd9a536c92"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 05:06:17.787069 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:17.787046 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f041060-cb24-4d0a-be92-aabd9a536c92-kube-api-access-2vs8t" pod "5f041060-cb24-4d0a-be92-aabd9a536c92" (UID: "5f041060-cb24-4d0a-be92-aabd9a536c92"). InnerVolumeSpecName "kube-api-access-2vs8t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 05:06:17.885579 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:17.885539 4509 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f041060-cb24-4d0a-be92-aabd9a536c92-plugins-config-volume\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:06:17.885579 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:17.885575 4509 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vs8t\" (UniqueName: \"kubernetes.io/projected/5f041060-cb24-4d0a-be92-aabd9a536c92-kube-api-access-2vs8t\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:06:18.443369 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:18.443308 4509 generic.go:356] "Generic (PLEG): container finished" podID="5f041060-cb24-4d0a-be92-aabd9a536c92" containerID="d683c478c107507a70e1739f169c06fcda658c6ea0c5ab0a729b212b953e95df" exitCode=0 Nov 04 05:06:18.443369 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:18.443357 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-cedfca-67b95cc659-6v86t" event={"ID":"5f041060-cb24-4d0a-be92-aabd9a536c92","Type":"ContainerDied","Data":"d683c478c107507a70e1739f169c06fcda658c6ea0c5ab0a729b212b953e95df"} Nov 04 05:06:18.443369 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:18.443376 4509 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-cedfca-67b95cc659-6v86t" Nov 04 05:06:18.443642 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:18.443416 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-cedfca-67b95cc659-6v86t" event={"ID":"5f041060-cb24-4d0a-be92-aabd9a536c92","Type":"ContainerDied","Data":"18289411500cfdefece7185753c92620a504329621c613d3068a0cf61da23f6e"} Nov 04 05:06:18.443642 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:18.443438 4509 scope.go:122] "RemoveContainer" containerID="d683c478c107507a70e1739f169c06fcda658c6ea0c5ab0a729b212b953e95df" Nov 04 05:06:18.452310 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:18.452295 4509 scope.go:122] "RemoveContainer" containerID="d683c478c107507a70e1739f169c06fcda658c6ea0c5ab0a729b212b953e95df" Nov 04 05:06:18.452622 ip-10-0-133-180 kubenswrapper[4509]: E1104 05:06:18.452602 4509 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d683c478c107507a70e1739f169c06fcda658c6ea0c5ab0a729b212b953e95df\": container with ID starting with d683c478c107507a70e1739f169c06fcda658c6ea0c5ab0a729b212b953e95df not found: ID does not exist" containerID="d683c478c107507a70e1739f169c06fcda658c6ea0c5ab0a729b212b953e95df" Nov 04 05:06:18.452686 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:18.452629 4509 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d683c478c107507a70e1739f169c06fcda658c6ea0c5ab0a729b212b953e95df"} err="failed to get container status \"d683c478c107507a70e1739f169c06fcda658c6ea0c5ab0a729b212b953e95df\": rpc error: code = NotFound desc = could not find container \"d683c478c107507a70e1739f169c06fcda658c6ea0c5ab0a729b212b953e95df\": container with ID starting with d683c478c107507a70e1739f169c06fcda658c6ea0c5ab0a729b212b953e95df not found: ID does not exist" Nov 04 05:06:18.608030 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:18.607944 4509 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-cedfca-67b95cc659-6v86t"] Nov 04 05:06:18.657868 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:18.657822 4509 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-cedfca-67b95cc659-6v86t"] Nov 04 05:06:19.964999 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:19.964963 4509 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-4cce33-56666d7678-r88jh" podUID="2a16804b-60cc-46b5-9879-163bda7bd069" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 05:06:20.261209 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:20.261119 4509 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f041060-cb24-4d0a-be92-aabd9a536c92" path="/var/lib/kubelet/pods/5f041060-cb24-4d0a-be92-aabd9a536c92/volumes" Nov 04 05:06:21.949490 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:21.949451 4509 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-tkn4g"] Nov 04 05:06:21.949881 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:21.949844 4509 cpu_manager.go:414] "containerMap: removing container" podUID="c1afb6cd-a9c5-43b9-ae3b-9929840fe019" containerName="main" Nov 04 05:06:21.949881 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:21.949855 4509 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1afb6cd-a9c5-43b9-ae3b-9929840fe019" containerName="main" Nov 04 05:06:21.949881 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:21.949878 4509 cpu_manager.go:414] "containerMap: removing container" podUID="83294360-4718-4139-8287-8ec8ee6f72ce" containerName="revoke-keys" Nov 04 05:06:21.949998 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:21.949883 4509 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83294360-4718-4139-8287-8ec8ee6f72ce" containerName="revoke-keys" Nov 04 05:06:21.949998 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:21.949890 4509 cpu_manager.go:414] "containerMap: removing container" podUID="5f041060-cb24-4d0a-be92-aabd9a536c92" containerName="payload-pre-processing" Nov 04 05:06:21.949998 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:21.949895 4509 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f041060-cb24-4d0a-be92-aabd9a536c92" containerName="payload-pre-processing" Nov 04 05:06:21.949998 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:21.949914 4509 cpu_manager.go:414] "containerMap: removing container" podUID="0d9118e1-aff5-4fb2-8a56-78e3ee9fa3e4" containerName="main" Nov 04 05:06:21.949998 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:21.949919 4509 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d9118e1-aff5-4fb2-8a56-78e3ee9fa3e4" containerName="main" Nov 04 05:06:21.949998 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:21.949967 4509 memory_manager.go:365] "RemoveStaleState removing state" podUID="83294360-4718-4139-8287-8ec8ee6f72ce" containerName="revoke-keys" Nov 04 05:06:21.949998 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:21.949975 4509 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f041060-cb24-4d0a-be92-aabd9a536c92" containerName="payload-pre-processing" Nov 04 05:06:21.949998 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:21.949985 4509 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d9118e1-aff5-4fb2-8a56-78e3ee9fa3e4" containerName="main" Nov 04 05:06:21.949998 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:21.949990 4509 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1afb6cd-a9c5-43b9-ae3b-9929840fe019" containerName="main" Nov 04 05:06:21.954390 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:21.954367 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-tkn4g" Nov 04 05:06:21.999740 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:21.999702 4509 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-85pc8"] Nov 04 05:06:22.003313 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:22.003278 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-85pc8" Nov 04 05:06:22.096019 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:22.095982 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-85pc8"] Nov 04 05:06:22.130028 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:22.129995 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80cd56a4-9db7-4770-b1cc-a10754e3de43-istio-envoy\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-tkn4g\" (UID: \"80cd56a4-9db7-4770-b1cc-a10754e3de43\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-tkn4g" Nov 04 05:06:22.130028 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:22.130030 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80cd56a4-9db7-4770-b1cc-a10754e3de43-credential-socket\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-tkn4g\" (UID: \"80cd56a4-9db7-4770-b1cc-a10754e3de43\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-tkn4g" Nov 04 05:06:22.130270 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:22.130113 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80cd56a4-9db7-4770-b1cc-a10754e3de43-istiod-ca-cert\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-tkn4g\" (UID: \"80cd56a4-9db7-4770-b1cc-a10754e3de43\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-tkn4g" Nov 04 05:06:22.130270 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:22.130147 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37a5cce6-297b-4825-8710-bcb023060bb4-istiod-ca-cert\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-85pc8\" (UID: \"37a5cce6-297b-4825-8710-bcb023060bb4\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-85pc8" Nov 04 05:06:22.130270 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:22.130164 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80cd56a4-9db7-4770-b1cc-a10754e3de43-istio-podinfo\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-tkn4g\" (UID: \"80cd56a4-9db7-4770-b1cc-a10754e3de43\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-tkn4g" Nov 04 05:06:22.130270 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:22.130180 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56rsm\" (UniqueName: \"kubernetes.io/projected/80cd56a4-9db7-4770-b1cc-a10754e3de43-kube-api-access-56rsm\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-tkn4g\" (UID: \"80cd56a4-9db7-4770-b1cc-a10754e3de43\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-tkn4g" Nov 04 05:06:22.130270 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:22.130236 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80cd56a4-9db7-4770-b1cc-a10754e3de43-workload-certs\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-tkn4g\" (UID: \"80cd56a4-9db7-4770-b1cc-a10754e3de43\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-tkn4g" Nov 04 05:06:22.130468 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:22.130276 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d58vv\" (UniqueName: \"kubernetes.io/projected/37a5cce6-297b-4825-8710-bcb023060bb4-kube-api-access-d58vv\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-85pc8\" (UID: \"37a5cce6-297b-4825-8710-bcb023060bb4\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-85pc8" Nov 04 05:06:22.130468 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:22.130298 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37a5cce6-297b-4825-8710-bcb023060bb4-workload-certs\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-85pc8\" (UID: \"37a5cce6-297b-4825-8710-bcb023060bb4\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-85pc8" Nov 04 05:06:22.130468 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:22.130314 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80cd56a4-9db7-4770-b1cc-a10754e3de43-istio-data\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-tkn4g\" (UID: \"80cd56a4-9db7-4770-b1cc-a10754e3de43\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-tkn4g" Nov 04 05:06:22.130468 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:22.130395 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80cd56a4-9db7-4770-b1cc-a10754e3de43-istio-token\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-tkn4g\" (UID: \"80cd56a4-9db7-4770-b1cc-a10754e3de43\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-tkn4g" Nov 04 05:06:22.130468 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:22.130443 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37a5cce6-297b-4825-8710-bcb023060bb4-istio-envoy\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-85pc8\" (UID: \"37a5cce6-297b-4825-8710-bcb023060bb4\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-85pc8" Nov 04 05:06:22.130672 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:22.130490 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80cd56a4-9db7-4770-b1cc-a10754e3de43-workload-socket\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-tkn4g\" (UID: \"80cd56a4-9db7-4770-b1cc-a10754e3de43\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-tkn4g" Nov 04 05:06:22.130672 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:22.130506 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37a5cce6-297b-4825-8710-bcb023060bb4-istio-data\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-85pc8\" (UID: \"37a5cce6-297b-4825-8710-bcb023060bb4\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-85pc8" Nov 04 05:06:22.130672 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:22.130531 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37a5cce6-297b-4825-8710-bcb023060bb4-istio-podinfo\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-85pc8\" (UID: \"37a5cce6-297b-4825-8710-bcb023060bb4\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-85pc8" Nov 04 05:06:22.130672 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:22.130551 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37a5cce6-297b-4825-8710-bcb023060bb4-istio-token\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-85pc8\" (UID: \"37a5cce6-297b-4825-8710-bcb023060bb4\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-85pc8" Nov 04 05:06:22.130672 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:22.130666 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37a5cce6-297b-4825-8710-bcb023060bb4-credential-socket\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-85pc8\" (UID: \"37a5cce6-297b-4825-8710-bcb023060bb4\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-85pc8" Nov 04 05:06:22.130842 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:22.130686 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37a5cce6-297b-4825-8710-bcb023060bb4-workload-socket\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-85pc8\" (UID: \"37a5cce6-297b-4825-8710-bcb023060bb4\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-85pc8" Nov 04 05:06:22.155843 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:22.155806 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-tkn4g"] Nov 04 05:06:22.232060 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:22.232028 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37a5cce6-297b-4825-8710-bcb023060bb4-istio-envoy\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-85pc8\" (UID: \"37a5cce6-297b-4825-8710-bcb023060bb4\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-85pc8" Nov 04 05:06:22.232060 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:22.232065 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80cd56a4-9db7-4770-b1cc-a10754e3de43-workload-socket\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-tkn4g\" (UID: \"80cd56a4-9db7-4770-b1cc-a10754e3de43\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-tkn4g" Nov 04 05:06:22.232308 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:22.232084 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37a5cce6-297b-4825-8710-bcb023060bb4-istio-data\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-85pc8\" (UID: \"37a5cce6-297b-4825-8710-bcb023060bb4\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-85pc8" Nov 04 05:06:22.232308 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:22.232112 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37a5cce6-297b-4825-8710-bcb023060bb4-istio-podinfo\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-85pc8\" (UID: \"37a5cce6-297b-4825-8710-bcb023060bb4\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-85pc8" Nov 04 05:06:22.232308 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:22.232144 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37a5cce6-297b-4825-8710-bcb023060bb4-istio-token\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-85pc8\" (UID: \"37a5cce6-297b-4825-8710-bcb023060bb4\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-85pc8" Nov 04 05:06:22.232308 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:22.232184 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37a5cce6-297b-4825-8710-bcb023060bb4-credential-socket\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-85pc8\" (UID: \"37a5cce6-297b-4825-8710-bcb023060bb4\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-85pc8" Nov 04 05:06:22.232568 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:22.232316 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37a5cce6-297b-4825-8710-bcb023060bb4-workload-socket\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-85pc8\" (UID: \"37a5cce6-297b-4825-8710-bcb023060bb4\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-85pc8" Nov 04 05:06:22.232568 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:22.232405 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80cd56a4-9db7-4770-b1cc-a10754e3de43-istio-envoy\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-tkn4g\" (UID: \"80cd56a4-9db7-4770-b1cc-a10754e3de43\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-tkn4g" Nov 04 05:06:22.232568 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:22.232436 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80cd56a4-9db7-4770-b1cc-a10754e3de43-credential-socket\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-tkn4g\" (UID: \"80cd56a4-9db7-4770-b1cc-a10754e3de43\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-tkn4g" Nov 04 05:06:22.232568 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:22.232516 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80cd56a4-9db7-4770-b1cc-a10754e3de43-istiod-ca-cert\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-tkn4g\" (UID: \"80cd56a4-9db7-4770-b1cc-a10754e3de43\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-tkn4g" Nov 04 05:06:22.232568 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:22.232549 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37a5cce6-297b-4825-8710-bcb023060bb4-istiod-ca-cert\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-85pc8\" (UID: \"37a5cce6-297b-4825-8710-bcb023060bb4\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-85pc8" Nov 04 05:06:22.232568 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:22.232550 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80cd56a4-9db7-4770-b1cc-a10754e3de43-workload-socket\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-tkn4g\" (UID: \"80cd56a4-9db7-4770-b1cc-a10754e3de43\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-tkn4g" Nov 04 05:06:22.232875 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:22.232575 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80cd56a4-9db7-4770-b1cc-a10754e3de43-istio-podinfo\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-tkn4g\" (UID: \"80cd56a4-9db7-4770-b1cc-a10754e3de43\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-tkn4g" Nov 04 05:06:22.232875 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:22.232615 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37a5cce6-297b-4825-8710-bcb023060bb4-istio-data\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-85pc8\" (UID: \"37a5cce6-297b-4825-8710-bcb023060bb4\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-85pc8" Nov 04 05:06:22.232875 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:22.232617 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56rsm\" (UniqueName: \"kubernetes.io/projected/80cd56a4-9db7-4770-b1cc-a10754e3de43-kube-api-access-56rsm\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-tkn4g\" (UID: \"80cd56a4-9db7-4770-b1cc-a10754e3de43\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-tkn4g" Nov 04 05:06:22.232875 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:22.232674 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37a5cce6-297b-4825-8710-bcb023060bb4-workload-socket\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-85pc8\" (UID: \"37a5cce6-297b-4825-8710-bcb023060bb4\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-85pc8" Nov 04 05:06:22.232875 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:22.232681 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80cd56a4-9db7-4770-b1cc-a10754e3de43-workload-certs\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-tkn4g\" (UID: \"80cd56a4-9db7-4770-b1cc-a10754e3de43\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-tkn4g" Nov 04 05:06:22.232875 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:22.232742 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d58vv\" (UniqueName: \"kubernetes.io/projected/37a5cce6-297b-4825-8710-bcb023060bb4-kube-api-access-d58vv\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-85pc8\" (UID: \"37a5cce6-297b-4825-8710-bcb023060bb4\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-85pc8" Nov 04 05:06:22.232875 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:22.232777 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37a5cce6-297b-4825-8710-bcb023060bb4-workload-certs\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-85pc8\" (UID: \"37a5cce6-297b-4825-8710-bcb023060bb4\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-85pc8" Nov 04 05:06:22.232875 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:22.232788 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80cd56a4-9db7-4770-b1cc-a10754e3de43-credential-socket\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-tkn4g\" (UID: \"80cd56a4-9db7-4770-b1cc-a10754e3de43\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-tkn4g" Nov 04 05:06:22.232875 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:22.232801 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80cd56a4-9db7-4770-b1cc-a10754e3de43-istio-data\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-tkn4g\" (UID: \"80cd56a4-9db7-4770-b1cc-a10754e3de43\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-tkn4g" Nov 04 05:06:22.232875 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:22.232827 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80cd56a4-9db7-4770-b1cc-a10754e3de43-istio-token\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-tkn4g\" (UID: \"80cd56a4-9db7-4770-b1cc-a10754e3de43\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-tkn4g" Nov 04 05:06:22.233573 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:22.233384 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37a5cce6-297b-4825-8710-bcb023060bb4-credential-socket\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-85pc8\" (UID: \"37a5cce6-297b-4825-8710-bcb023060bb4\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-85pc8" Nov 04 05:06:22.233573 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:22.233462 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80cd56a4-9db7-4770-b1cc-a10754e3de43-istiod-ca-cert\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-tkn4g\" (UID: \"80cd56a4-9db7-4770-b1cc-a10754e3de43\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-tkn4g" Nov 04 05:06:22.233703 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:22.233392 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80cd56a4-9db7-4770-b1cc-a10754e3de43-workload-certs\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-tkn4g\" (UID: \"80cd56a4-9db7-4770-b1cc-a10754e3de43\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-tkn4g" Nov 04 05:06:22.233703 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:22.233661 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37a5cce6-297b-4825-8710-bcb023060bb4-workload-certs\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-85pc8\" (UID: \"37a5cce6-297b-4825-8710-bcb023060bb4\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-85pc8" Nov 04 05:06:22.233703 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:22.233680 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37a5cce6-297b-4825-8710-bcb023060bb4-istiod-ca-cert\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-85pc8\" (UID: \"37a5cce6-297b-4825-8710-bcb023060bb4\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-85pc8" Nov 04 05:06:22.233867 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:22.233727 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80cd56a4-9db7-4770-b1cc-a10754e3de43-istio-data\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-tkn4g\" (UID: \"80cd56a4-9db7-4770-b1cc-a10754e3de43\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-tkn4g" Nov 04 05:06:22.234764 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:22.234743 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37a5cce6-297b-4825-8710-bcb023060bb4-istio-envoy\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-85pc8\" (UID: \"37a5cce6-297b-4825-8710-bcb023060bb4\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-85pc8" Nov 04 05:06:22.235170 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:22.235151 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80cd56a4-9db7-4770-b1cc-a10754e3de43-istio-podinfo\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-tkn4g\" (UID: \"80cd56a4-9db7-4770-b1cc-a10754e3de43\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-tkn4g" Nov 04 05:06:22.235422 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:22.235404 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37a5cce6-297b-4825-8710-bcb023060bb4-istio-podinfo\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-85pc8\" (UID: \"37a5cce6-297b-4825-8710-bcb023060bb4\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-85pc8" Nov 04 05:06:22.235603 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:22.235584 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80cd56a4-9db7-4770-b1cc-a10754e3de43-istio-envoy\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-tkn4g\" (UID: \"80cd56a4-9db7-4770-b1cc-a10754e3de43\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-tkn4g" Nov 04 05:06:22.290039 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:22.290003 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37a5cce6-297b-4825-8710-bcb023060bb4-istio-token\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-85pc8\" (UID: \"37a5cce6-297b-4825-8710-bcb023060bb4\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-85pc8" Nov 04 05:06:22.290474 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:22.290450 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d58vv\" (UniqueName: \"kubernetes.io/projected/37a5cce6-297b-4825-8710-bcb023060bb4-kube-api-access-d58vv\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-85pc8\" (UID: \"37a5cce6-297b-4825-8710-bcb023060bb4\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-85pc8" Nov 04 05:06:22.290706 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:22.290684 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56rsm\" (UniqueName: \"kubernetes.io/projected/80cd56a4-9db7-4770-b1cc-a10754e3de43-kube-api-access-56rsm\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-tkn4g\" (UID: \"80cd56a4-9db7-4770-b1cc-a10754e3de43\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-tkn4g" Nov 04 05:06:22.290839 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:22.290821 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80cd56a4-9db7-4770-b1cc-a10754e3de43-istio-token\") pod \"e2e-worker-w5-4cce33-openshift-default-747768654f-tkn4g\" (UID: \"80cd56a4-9db7-4770-b1cc-a10754e3de43\") " pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-tkn4g" Nov 04 05:06:22.316991 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:22.316952 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-85pc8" Nov 04 05:06:22.568703 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:22.568613 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-tkn4g" Nov 04 05:06:22.589206 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:22.585230 4509 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-lptr9"] Nov 04 05:06:22.593948 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:22.593917 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-lptr9" Nov 04 05:06:22.662564 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:22.661710 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-lptr9"] Nov 04 05:06:22.736613 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:22.736571 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff705a2e-8f20-49e5-aa26-69c7a8d1c31e-istio-podinfo\") pod \"e2e-worker-w6-cedfca-openshift-default-6b99d7696f-lptr9\" (UID: \"ff705a2e-8f20-49e5-aa26-69c7a8d1c31e\") " pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-lptr9" Nov 04 05:06:22.736796 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:22.736689 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff705a2e-8f20-49e5-aa26-69c7a8d1c31e-credential-socket\") pod \"e2e-worker-w6-cedfca-openshift-default-6b99d7696f-lptr9\" (UID: \"ff705a2e-8f20-49e5-aa26-69c7a8d1c31e\") " pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-lptr9" Nov 04 05:06:22.736796 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:22.736754 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff705a2e-8f20-49e5-aa26-69c7a8d1c31e-istio-token\") pod \"e2e-worker-w6-cedfca-openshift-default-6b99d7696f-lptr9\" (UID: \"ff705a2e-8f20-49e5-aa26-69c7a8d1c31e\") " pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-lptr9" Nov 04 05:06:22.736796 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:22.736781 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccmws\" (UniqueName: \"kubernetes.io/projected/ff705a2e-8f20-49e5-aa26-69c7a8d1c31e-kube-api-access-ccmws\") pod \"e2e-worker-w6-cedfca-openshift-default-6b99d7696f-lptr9\" (UID: \"ff705a2e-8f20-49e5-aa26-69c7a8d1c31e\") " pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-lptr9" Nov 04 05:06:22.736961 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:22.736839 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff705a2e-8f20-49e5-aa26-69c7a8d1c31e-istiod-ca-cert\") pod \"e2e-worker-w6-cedfca-openshift-default-6b99d7696f-lptr9\" (UID: \"ff705a2e-8f20-49e5-aa26-69c7a8d1c31e\") " pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-lptr9" Nov 04 05:06:22.736961 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:22.736885 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff705a2e-8f20-49e5-aa26-69c7a8d1c31e-istio-envoy\") pod \"e2e-worker-w6-cedfca-openshift-default-6b99d7696f-lptr9\" (UID: \"ff705a2e-8f20-49e5-aa26-69c7a8d1c31e\") " pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-lptr9" Nov 04 05:06:22.737035 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:22.736999 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff705a2e-8f20-49e5-aa26-69c7a8d1c31e-workload-certs\") pod \"e2e-worker-w6-cedfca-openshift-default-6b99d7696f-lptr9\" (UID: \"ff705a2e-8f20-49e5-aa26-69c7a8d1c31e\") " pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-lptr9" Nov 04 05:06:22.737067 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:22.737037 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff705a2e-8f20-49e5-aa26-69c7a8d1c31e-workload-socket\") pod \"e2e-worker-w6-cedfca-openshift-default-6b99d7696f-lptr9\" (UID: \"ff705a2e-8f20-49e5-aa26-69c7a8d1c31e\") " pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-lptr9" Nov 04 05:06:22.737067 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:22.737056 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff705a2e-8f20-49e5-aa26-69c7a8d1c31e-istio-data\") pod \"e2e-worker-w6-cedfca-openshift-default-6b99d7696f-lptr9\" (UID: \"ff705a2e-8f20-49e5-aa26-69c7a8d1c31e\") " pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-lptr9" Nov 04 05:06:22.747987 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:22.747959 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-85pc8"] Nov 04 05:06:22.749418 ip-10-0-133-180 kubenswrapper[4509]: W1104 05:06:22.749375 4509 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37a5cce6_297b_4825_8710_bcb023060bb4.slice/crio-08f5bd3bd688b8eb0b7c4eb0615a25856d0925bba9943403c5af8526e1b2fa45 WatchSource:0}: Error finding container 08f5bd3bd688b8eb0b7c4eb0615a25856d0925bba9943403c5af8526e1b2fa45: Status 404 returned error can't find the container with id 08f5bd3bd688b8eb0b7c4eb0615a25856d0925bba9943403c5af8526e1b2fa45 Nov 04 05:06:22.751553 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:22.751522 4509 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 04 05:06:22.751630 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:22.751617 4509 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 04 05:06:22.751669 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:22.751657 4509 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 04 05:06:22.837821 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:22.837734 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff705a2e-8f20-49e5-aa26-69c7a8d1c31e-istio-token\") pod \"e2e-worker-w6-cedfca-openshift-default-6b99d7696f-lptr9\" (UID: \"ff705a2e-8f20-49e5-aa26-69c7a8d1c31e\") " pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-lptr9" Nov 04 05:06:22.837821 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:22.837791 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccmws\" (UniqueName: \"kubernetes.io/projected/ff705a2e-8f20-49e5-aa26-69c7a8d1c31e-kube-api-access-ccmws\") pod \"e2e-worker-w6-cedfca-openshift-default-6b99d7696f-lptr9\" (UID: \"ff705a2e-8f20-49e5-aa26-69c7a8d1c31e\") " pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-lptr9" Nov 04 05:06:22.838011 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:22.837826 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff705a2e-8f20-49e5-aa26-69c7a8d1c31e-istiod-ca-cert\") pod \"e2e-worker-w6-cedfca-openshift-default-6b99d7696f-lptr9\" (UID: \"ff705a2e-8f20-49e5-aa26-69c7a8d1c31e\") " pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-lptr9" Nov 04 05:06:22.838011 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:22.837880 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff705a2e-8f20-49e5-aa26-69c7a8d1c31e-istio-envoy\") pod \"e2e-worker-w6-cedfca-openshift-default-6b99d7696f-lptr9\" (UID: \"ff705a2e-8f20-49e5-aa26-69c7a8d1c31e\") " pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-lptr9" Nov 04 05:06:22.838011 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:22.837925 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff705a2e-8f20-49e5-aa26-69c7a8d1c31e-workload-certs\") pod \"e2e-worker-w6-cedfca-openshift-default-6b99d7696f-lptr9\" (UID: \"ff705a2e-8f20-49e5-aa26-69c7a8d1c31e\") " pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-lptr9" Nov 04 05:06:22.838011 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:22.837972 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff705a2e-8f20-49e5-aa26-69c7a8d1c31e-workload-socket\") pod \"e2e-worker-w6-cedfca-openshift-default-6b99d7696f-lptr9\" (UID: \"ff705a2e-8f20-49e5-aa26-69c7a8d1c31e\") " pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-lptr9" Nov 04 05:06:22.838226 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:22.838148 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff705a2e-8f20-49e5-aa26-69c7a8d1c31e-istio-data\") pod \"e2e-worker-w6-cedfca-openshift-default-6b99d7696f-lptr9\" (UID: \"ff705a2e-8f20-49e5-aa26-69c7a8d1c31e\") " pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-lptr9" Nov 04 05:06:22.838226 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:22.838188 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff705a2e-8f20-49e5-aa26-69c7a8d1c31e-istio-podinfo\") pod \"e2e-worker-w6-cedfca-openshift-default-6b99d7696f-lptr9\" (UID: \"ff705a2e-8f20-49e5-aa26-69c7a8d1c31e\") " pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-lptr9" Nov 04 05:06:22.838340 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:22.838232 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff705a2e-8f20-49e5-aa26-69c7a8d1c31e-credential-socket\") pod \"e2e-worker-w6-cedfca-openshift-default-6b99d7696f-lptr9\" (UID: \"ff705a2e-8f20-49e5-aa26-69c7a8d1c31e\") " pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-lptr9" Nov 04 05:06:22.838779 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:22.838624 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff705a2e-8f20-49e5-aa26-69c7a8d1c31e-workload-certs\") pod \"e2e-worker-w6-cedfca-openshift-default-6b99d7696f-lptr9\" (UID: \"ff705a2e-8f20-49e5-aa26-69c7a8d1c31e\") " pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-lptr9" Nov 04 05:06:22.838779 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:22.838643 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff705a2e-8f20-49e5-aa26-69c7a8d1c31e-workload-socket\") pod \"e2e-worker-w6-cedfca-openshift-default-6b99d7696f-lptr9\" (UID: \"ff705a2e-8f20-49e5-aa26-69c7a8d1c31e\") " pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-lptr9" Nov 04 05:06:22.838779 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:22.838737 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff705a2e-8f20-49e5-aa26-69c7a8d1c31e-istio-data\") pod \"e2e-worker-w6-cedfca-openshift-default-6b99d7696f-lptr9\" (UID: \"ff705a2e-8f20-49e5-aa26-69c7a8d1c31e\") " pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-lptr9" Nov 04 05:06:22.838976 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:22.838789 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff705a2e-8f20-49e5-aa26-69c7a8d1c31e-credential-socket\") pod \"e2e-worker-w6-cedfca-openshift-default-6b99d7696f-lptr9\" (UID: \"ff705a2e-8f20-49e5-aa26-69c7a8d1c31e\") " pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-lptr9" Nov 04 05:06:22.839205 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:22.839180 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff705a2e-8f20-49e5-aa26-69c7a8d1c31e-istiod-ca-cert\") pod \"e2e-worker-w6-cedfca-openshift-default-6b99d7696f-lptr9\" (UID: \"ff705a2e-8f20-49e5-aa26-69c7a8d1c31e\") " pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-lptr9" Nov 04 05:06:22.840555 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:22.840536 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff705a2e-8f20-49e5-aa26-69c7a8d1c31e-istio-envoy\") pod \"e2e-worker-w6-cedfca-openshift-default-6b99d7696f-lptr9\" (UID: \"ff705a2e-8f20-49e5-aa26-69c7a8d1c31e\") " pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-lptr9" Nov 04 05:06:22.841479 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:22.841460 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff705a2e-8f20-49e5-aa26-69c7a8d1c31e-istio-podinfo\") pod \"e2e-worker-w6-cedfca-openshift-default-6b99d7696f-lptr9\" (UID: \"ff705a2e-8f20-49e5-aa26-69c7a8d1c31e\") " pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-lptr9" Nov 04 05:06:22.894783 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:22.894750 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff705a2e-8f20-49e5-aa26-69c7a8d1c31e-istio-token\") pod \"e2e-worker-w6-cedfca-openshift-default-6b99d7696f-lptr9\" (UID: \"ff705a2e-8f20-49e5-aa26-69c7a8d1c31e\") " pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-lptr9" Nov 04 05:06:22.895028 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:22.895006 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccmws\" (UniqueName: \"kubernetes.io/projected/ff705a2e-8f20-49e5-aa26-69c7a8d1c31e-kube-api-access-ccmws\") pod \"e2e-worker-w6-cedfca-openshift-default-6b99d7696f-lptr9\" (UID: \"ff705a2e-8f20-49e5-aa26-69c7a8d1c31e\") " pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-lptr9" Nov 04 05:06:22.908828 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:22.908790 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-lptr9" Nov 04 05:06:22.935484 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:22.935427 4509 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-m6qb5"] Nov 04 05:06:22.936400 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:22.936312 4509 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-m6qb5" podUID="d751b557-4e07-4657-96ec-59fc0e70fd57" containerName="istio-proxy" containerID="cri-o://33466972d2f1df00a526742b7a18372fa8aa2b7b68e9a3ccab257f6ed7aa65ca" gracePeriod=30 Nov 04 05:06:22.939934 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:22.939905 4509 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-lptr9"] Nov 04 05:06:22.952126 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:22.952093 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-tkn4g"] Nov 04 05:06:22.954054 ip-10-0-133-180 kubenswrapper[4509]: W1104 05:06:22.954014 4509 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80cd56a4_9db7_4770_b1cc_a10754e3de43.slice/crio-ffe126a2fa009e3b63a8ad7f9d23c807ce351682305e64db301b53783e0fd0e3 WatchSource:0}: Error finding container ffe126a2fa009e3b63a8ad7f9d23c807ce351682305e64db301b53783e0fd0e3: Status 404 returned error can't find the container with id ffe126a2fa009e3b63a8ad7f9d23c807ce351682305e64db301b53783e0fd0e3 Nov 04 05:06:22.956399 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:22.956313 4509 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 04 05:06:22.956611 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:22.956399 4509 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 04 05:06:22.956611 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:22.956435 4509 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 04 05:06:23.324939 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:23.324466 4509 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-lptr9"] Nov 04 05:06:23.326906 ip-10-0-133-180 kubenswrapper[4509]: W1104 05:06:23.326872 4509 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff705a2e_8f20_49e5_aa26_69c7a8d1c31e.slice/crio-4a0e0fed34f4bc7006622a2c5bcbca99cedfb3c0cfab8298d26c460a6fb40ee2 WatchSource:0}: Error finding container 4a0e0fed34f4bc7006622a2c5bcbca99cedfb3c0cfab8298d26c460a6fb40ee2: Status 404 returned error can't find the container with id 4a0e0fed34f4bc7006622a2c5bcbca99cedfb3c0cfab8298d26c460a6fb40ee2 Nov 04 05:06:23.329354 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:23.329290 4509 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 04 05:06:23.329452 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:23.329402 4509 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 04 05:06:23.329497 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:23.329453 4509 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 04 05:06:23.495825 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:23.495463 4509 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-lptr9" podUID="ff705a2e-8f20-49e5-aa26-69c7a8d1c31e" containerName="istio-proxy" containerID="cri-o://e2e35235e427b2ccb79c21ff7abb73521c066a44d17924ce426074e02f8d92a4" gracePeriod=30 Nov 04 05:06:23.496120 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:23.496031 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-lptr9" event={"ID":"ff705a2e-8f20-49e5-aa26-69c7a8d1c31e","Type":"ContainerStarted","Data":"e2e35235e427b2ccb79c21ff7abb73521c066a44d17924ce426074e02f8d92a4"} Nov 04 05:06:23.496120 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:23.496090 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-lptr9" event={"ID":"ff705a2e-8f20-49e5-aa26-69c7a8d1c31e","Type":"ContainerStarted","Data":"4a0e0fed34f4bc7006622a2c5bcbca99cedfb3c0cfab8298d26c460a6fb40ee2"} Nov 04 05:06:23.498825 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:23.498784 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-85pc8" event={"ID":"37a5cce6-297b-4825-8710-bcb023060bb4","Type":"ContainerStarted","Data":"f3a21b7077cf2e04603805d64e5da5e9e6cc969699a1f3bb23599100e0b047f6"} Nov 04 05:06:23.499014 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:23.499000 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-85pc8" event={"ID":"37a5cce6-297b-4825-8710-bcb023060bb4","Type":"ContainerStarted","Data":"08f5bd3bd688b8eb0b7c4eb0615a25856d0925bba9943403c5af8526e1b2fa45"} Nov 04 05:06:23.502297 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:23.502263 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-tkn4g" event={"ID":"80cd56a4-9db7-4770-b1cc-a10754e3de43","Type":"ContainerStarted","Data":"c1ba8d43f434b1dc83a07db47eb299158ad9ee6ab4ec8cb3599ffcfb91bd2dc5"} Nov 04 05:06:23.502462 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:23.502317 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-tkn4g" event={"ID":"80cd56a4-9db7-4770-b1cc-a10754e3de43","Type":"ContainerStarted","Data":"ffe126a2fa009e3b63a8ad7f9d23c807ce351682305e64db301b53783e0fd0e3"} Nov 04 05:06:23.571938 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:23.571901 4509 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-tkn4g" Nov 04 05:06:23.615267 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:23.615145 4509 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-lptr9" podStartSLOduration=1.615125897 podStartE2EDuration="1.615125897s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 05:06:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 05:06:23.612880538 +0000 UTC m=+1147.490604056" watchObservedRunningTime="2026-11-04 05:06:23.615125897 +0000 UTC m=+1147.492849417" Nov 04 05:06:23.753833 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:23.753777 4509 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-85pc8" podStartSLOduration=2.7537631510000002 podStartE2EDuration="2.753763151s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 05:06:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 05:06:23.751971548 +0000 UTC m=+1147.629695070" watchObservedRunningTime="2026-11-04 05:06:23.753763151 +0000 UTC m=+1147.631486672" Nov 04 05:06:24.317437 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:24.317398 4509 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-85pc8" Nov 04 05:06:24.318950 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:24.318927 4509 patch_prober.go:29] interesting pod/e2e-worker-w5-4cce33-openshift-default-747768654f-85pc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Nov 04 05:06:24.319078 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:24.318971 4509 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-85pc8" podUID="37a5cce6-297b-4825-8710-bcb023060bb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Nov 04 05:06:24.601420 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:24.601134 4509 patch_prober.go:29] interesting pod/e2e-worker-w5-4cce33-openshift-default-747768654f-tkn4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Nov 04 05:06:24.601420 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:24.601200 4509 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-tkn4g" podUID="80cd56a4-9db7-4770-b1cc-a10754e3de43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Nov 04 05:06:25.318120 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:25.318033 4509 patch_prober.go:29] interesting pod/e2e-worker-w5-4cce33-openshift-default-747768654f-85pc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Nov 04 05:06:25.318120 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:25.318088 4509 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-85pc8" podUID="37a5cce6-297b-4825-8710-bcb023060bb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Nov 04 05:06:25.572912 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:25.572813 4509 patch_prober.go:29] interesting pod/e2e-worker-w5-4cce33-openshift-default-747768654f-tkn4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Nov 04 05:06:25.573756 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:25.573725 4509 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-tkn4g" podUID="80cd56a4-9db7-4770-b1cc-a10754e3de43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Nov 04 05:06:26.317778 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:26.317739 4509 patch_prober.go:29] interesting pod/e2e-worker-w5-4cce33-openshift-default-747768654f-85pc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Nov 04 05:06:26.318026 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:26.317996 4509 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-85pc8" podUID="37a5cce6-297b-4825-8710-bcb023060bb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Nov 04 05:06:26.570859 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:26.570755 4509 patch_prober.go:29] interesting pod/e2e-worker-w5-4cce33-openshift-default-747768654f-tkn4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Nov 04 05:06:26.571470 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:26.571437 4509 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-tkn4g" podUID="80cd56a4-9db7-4770-b1cc-a10754e3de43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Nov 04 05:06:27.044394 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:27.044339 4509 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-tkn4g" podStartSLOduration=6.044307259 podStartE2EDuration="6.044307259s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 05:06:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 05:06:23.892277932 +0000 UTC m=+1147.770001466" watchObservedRunningTime="2026-11-04 05:06:27.044307259 +0000 UTC m=+1150.922030780" Nov 04 05:06:27.045801 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:27.045781 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-42dnm"] Nov 04 05:06:27.317737 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:27.317636 4509 patch_prober.go:29] interesting pod/e2e-worker-w5-4cce33-openshift-default-747768654f-85pc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Nov 04 05:06:27.317737 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:27.317696 4509 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-85pc8" podUID="37a5cce6-297b-4825-8710-bcb023060bb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Nov 04 05:06:27.571408 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:27.570741 4509 patch_prober.go:29] interesting pod/e2e-worker-w5-4cce33-openshift-default-747768654f-tkn4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Nov 04 05:06:27.571408 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:27.570809 4509 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-tkn4g" podUID="80cd56a4-9db7-4770-b1cc-a10754e3de43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Nov 04 05:06:28.150926 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:28.150587 4509 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-4cce33-56666d7678-r88jh" Nov 04 05:06:28.239374 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:28.239338 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a16804b-60cc-46b5-9879-163bda7bd069-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a16804b-60cc-46b5-9879-163bda7bd069-plugins-config-volume\") pod \"2a16804b-60cc-46b5-9879-163bda7bd069\" (UID: \"2a16804b-60cc-46b5-9879-163bda7bd069\") " Nov 04 05:06:28.239535 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:28.239453 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a16804b-60cc-46b5-9879-163bda7bd069-kube-api-access-xmpj2\" (UniqueName: \"kubernetes.io/projected/2a16804b-60cc-46b5-9879-163bda7bd069-kube-api-access-xmpj2\") pod \"2a16804b-60cc-46b5-9879-163bda7bd069\" (UID: \"2a16804b-60cc-46b5-9879-163bda7bd069\") " Nov 04 05:06:28.240341 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:28.240298 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a16804b-60cc-46b5-9879-163bda7bd069-plugins-config-volume" pod "2a16804b-60cc-46b5-9879-163bda7bd069" (UID: "2a16804b-60cc-46b5-9879-163bda7bd069"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 05:06:28.244072 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:28.244042 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a16804b-60cc-46b5-9879-163bda7bd069-kube-api-access-xmpj2" pod "2a16804b-60cc-46b5-9879-163bda7bd069" (UID: "2a16804b-60cc-46b5-9879-163bda7bd069"). InnerVolumeSpecName "kube-api-access-xmpj2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 05:06:28.323188 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:28.323149 4509 patch_prober.go:29] interesting pod/e2e-worker-w5-4cce33-openshift-default-747768654f-85pc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Nov 04 05:06:28.323426 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:28.323214 4509 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-85pc8" podUID="37a5cce6-297b-4825-8710-bcb023060bb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Nov 04 05:06:28.340142 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:28.340114 4509 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-m6qb5" Nov 04 05:06:28.340773 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:28.340755 4509 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmpj2\" (UniqueName: \"kubernetes.io/projected/2a16804b-60cc-46b5-9879-163bda7bd069-kube-api-access-xmpj2\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:06:28.340773 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:28.340773 4509 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a16804b-60cc-46b5-9879-163bda7bd069-plugins-config-volume\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:06:28.441816 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:28.441740 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d751b557-4e07-4657-96ec-59fc0e70fd57-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d751b557-4e07-4657-96ec-59fc0e70fd57-istio-envoy\") pod \"d751b557-4e07-4657-96ec-59fc0e70fd57\" (UID: \"d751b557-4e07-4657-96ec-59fc0e70fd57\") " Nov 04 05:06:28.441952 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:28.441913 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d751b557-4e07-4657-96ec-59fc0e70fd57-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d751b557-4e07-4657-96ec-59fc0e70fd57-istiod-ca-cert\") pod \"d751b557-4e07-4657-96ec-59fc0e70fd57\" (UID: \"d751b557-4e07-4657-96ec-59fc0e70fd57\") " Nov 04 05:06:28.441987 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:28.441962 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d751b557-4e07-4657-96ec-59fc0e70fd57-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d751b557-4e07-4657-96ec-59fc0e70fd57-workload-socket\") pod \"d751b557-4e07-4657-96ec-59fc0e70fd57\" (UID: \"d751b557-4e07-4657-96ec-59fc0e70fd57\") " Nov 04 05:06:28.441987 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:28.441984 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d751b557-4e07-4657-96ec-59fc0e70fd57-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d751b557-4e07-4657-96ec-59fc0e70fd57-istio-data\") pod \"d751b557-4e07-4657-96ec-59fc0e70fd57\" (UID: \"d751b557-4e07-4657-96ec-59fc0e70fd57\") " Nov 04 05:06:28.442055 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:28.442025 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d751b557-4e07-4657-96ec-59fc0e70fd57-istio-token\" (UniqueName: \"kubernetes.io/projected/d751b557-4e07-4657-96ec-59fc0e70fd57-istio-token\") pod \"d751b557-4e07-4657-96ec-59fc0e70fd57\" (UID: \"d751b557-4e07-4657-96ec-59fc0e70fd57\") " Nov 04 05:06:28.442094 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:28.442054 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d751b557-4e07-4657-96ec-59fc0e70fd57-kube-api-access-5b5qb\" (UniqueName: \"kubernetes.io/projected/d751b557-4e07-4657-96ec-59fc0e70fd57-kube-api-access-5b5qb\") pod \"d751b557-4e07-4657-96ec-59fc0e70fd57\" (UID: \"d751b557-4e07-4657-96ec-59fc0e70fd57\") " Nov 04 05:06:28.442094 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:28.442082 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d751b557-4e07-4657-96ec-59fc0e70fd57-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d751b557-4e07-4657-96ec-59fc0e70fd57-credential-socket\") pod \"d751b557-4e07-4657-96ec-59fc0e70fd57\" (UID: \"d751b557-4e07-4657-96ec-59fc0e70fd57\") " Nov 04 05:06:28.442152 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:28.442109 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d751b557-4e07-4657-96ec-59fc0e70fd57-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d751b557-4e07-4657-96ec-59fc0e70fd57-workload-certs\") pod \"d751b557-4e07-4657-96ec-59fc0e70fd57\" (UID: \"d751b557-4e07-4657-96ec-59fc0e70fd57\") " Nov 04 05:06:28.442152 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:28.442131 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d751b557-4e07-4657-96ec-59fc0e70fd57-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d751b557-4e07-4657-96ec-59fc0e70fd57-istio-podinfo\") pod \"d751b557-4e07-4657-96ec-59fc0e70fd57\" (UID: \"d751b557-4e07-4657-96ec-59fc0e70fd57\") " Nov 04 05:06:28.442206 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:28.442163 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d751b557-4e07-4657-96ec-59fc0e70fd57-workload-socket" pod "d751b557-4e07-4657-96ec-59fc0e70fd57" (UID: "d751b557-4e07-4657-96ec-59fc0e70fd57"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:06:28.442385 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:28.442369 4509 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d751b557-4e07-4657-96ec-59fc0e70fd57-workload-socket\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:06:28.442451 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:28.442413 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d751b557-4e07-4657-96ec-59fc0e70fd57-istiod-ca-cert" pod "d751b557-4e07-4657-96ec-59fc0e70fd57" (UID: "d751b557-4e07-4657-96ec-59fc0e70fd57"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 05:06:28.442916 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:28.442889 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d751b557-4e07-4657-96ec-59fc0e70fd57-istio-data" pod "d751b557-4e07-4657-96ec-59fc0e70fd57" (UID: "d751b557-4e07-4657-96ec-59fc0e70fd57"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:06:28.443118 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:28.442927 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d751b557-4e07-4657-96ec-59fc0e70fd57-workload-certs" pod "d751b557-4e07-4657-96ec-59fc0e70fd57" (UID: "d751b557-4e07-4657-96ec-59fc0e70fd57"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:06:28.443118 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:28.443072 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d751b557-4e07-4657-96ec-59fc0e70fd57-credential-socket" pod "d751b557-4e07-4657-96ec-59fc0e70fd57" (UID: "d751b557-4e07-4657-96ec-59fc0e70fd57"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:06:28.444853 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:28.444829 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d751b557-4e07-4657-96ec-59fc0e70fd57-istio-token" pod "d751b557-4e07-4657-96ec-59fc0e70fd57" (UID: "d751b557-4e07-4657-96ec-59fc0e70fd57"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 05:06:28.445080 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:28.445061 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d751b557-4e07-4657-96ec-59fc0e70fd57-istio-podinfo" pod "d751b557-4e07-4657-96ec-59fc0e70fd57" (UID: "d751b557-4e07-4657-96ec-59fc0e70fd57"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 05:06:28.445209 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:28.445183 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d751b557-4e07-4657-96ec-59fc0e70fd57-kube-api-access-5b5qb" pod "d751b557-4e07-4657-96ec-59fc0e70fd57" (UID: "d751b557-4e07-4657-96ec-59fc0e70fd57"). InnerVolumeSpecName "kube-api-access-5b5qb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 05:06:28.445291 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:28.445261 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d751b557-4e07-4657-96ec-59fc0e70fd57-istio-envoy" pod "d751b557-4e07-4657-96ec-59fc0e70fd57" (UID: "d751b557-4e07-4657-96ec-59fc0e70fd57"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:06:28.542017 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:28.541988 4509 generic.go:356] "Generic (PLEG): container finished" podID="d751b557-4e07-4657-96ec-59fc0e70fd57" containerID="33466972d2f1df00a526742b7a18372fa8aa2b7b68e9a3ccab257f6ed7aa65ca" exitCode=0 Nov 04 05:06:28.542179 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:28.542167 4509 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-m6qb5" Nov 04 05:06:28.543048 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:28.543016 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-m6qb5" event={"ID":"d751b557-4e07-4657-96ec-59fc0e70fd57","Type":"ContainerDied","Data":"33466972d2f1df00a526742b7a18372fa8aa2b7b68e9a3ccab257f6ed7aa65ca"} Nov 04 05:06:28.543175 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:28.543083 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-m6qb5" event={"ID":"d751b557-4e07-4657-96ec-59fc0e70fd57","Type":"ContainerDied","Data":"1012908b2358a73c7b071a9072d9df084e9e7de2a8691f5cf931dc031bd15128"} Nov 04 05:06:28.543175 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:28.543116 4509 scope.go:122] "RemoveContainer" containerID="33466972d2f1df00a526742b7a18372fa8aa2b7b68e9a3ccab257f6ed7aa65ca" Nov 04 05:06:28.543779 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:28.543756 4509 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d751b557-4e07-4657-96ec-59fc0e70fd57-istiod-ca-cert\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:06:28.543876 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:28.543782 4509 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d751b557-4e07-4657-96ec-59fc0e70fd57-istio-data\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:06:28.543876 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:28.543792 4509 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d751b557-4e07-4657-96ec-59fc0e70fd57-istio-token\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:06:28.543876 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:28.543800 4509 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5b5qb\" (UniqueName: \"kubernetes.io/projected/d751b557-4e07-4657-96ec-59fc0e70fd57-kube-api-access-5b5qb\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:06:28.543876 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:28.543811 4509 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d751b557-4e07-4657-96ec-59fc0e70fd57-credential-socket\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:06:28.543876 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:28.543820 4509 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d751b557-4e07-4657-96ec-59fc0e70fd57-workload-certs\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:06:28.543876 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:28.543830 4509 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d751b557-4e07-4657-96ec-59fc0e70fd57-istio-podinfo\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:06:28.543876 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:28.543843 4509 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d751b557-4e07-4657-96ec-59fc0e70fd57-istio-envoy\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:06:28.544719 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:28.544645 4509 generic.go:356] "Generic (PLEG): container finished" podID="2a16804b-60cc-46b5-9879-163bda7bd069" containerID="9f5a4844648e5c09869687b1f548d63b756b2a51a6ac37f646cf789559c04cbb" exitCode=0 Nov 04 05:06:28.544719 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:28.544694 4509 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-4cce33-56666d7678-r88jh" Nov 04 05:06:28.544947 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:28.544690 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-4cce33-56666d7678-r88jh" event={"ID":"2a16804b-60cc-46b5-9879-163bda7bd069","Type":"ContainerDied","Data":"9f5a4844648e5c09869687b1f548d63b756b2a51a6ac37f646cf789559c04cbb"} Nov 04 05:06:28.544947 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:28.544812 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-4cce33-56666d7678-r88jh" event={"ID":"2a16804b-60cc-46b5-9879-163bda7bd069","Type":"ContainerDied","Data":"745892b1d1b7f5b91bc3577b66fd5916eb975246a1503ac41bbda14f67c2c717"} Nov 04 05:06:28.612392 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:28.610043 4509 patch_prober.go:29] interesting pod/e2e-worker-w5-4cce33-openshift-default-747768654f-tkn4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Nov 04 05:06:28.612392 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:28.610149 4509 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-tkn4g" podUID="80cd56a4-9db7-4770-b1cc-a10754e3de43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Nov 04 05:06:28.652972 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:28.652391 4509 scope.go:122] "RemoveContainer" containerID="90901cf35b3a5c5999f8076892b42910a049c313e90fe487980febe641e77b6b" Nov 04 05:06:28.737464 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:28.737056 4509 scope.go:122] "RemoveContainer" containerID="33466972d2f1df00a526742b7a18372fa8aa2b7b68e9a3ccab257f6ed7aa65ca" Nov 04 05:06:28.738076 ip-10-0-133-180 kubenswrapper[4509]: E1104 05:06:28.737745 4509 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33466972d2f1df00a526742b7a18372fa8aa2b7b68e9a3ccab257f6ed7aa65ca\": container with ID starting with 33466972d2f1df00a526742b7a18372fa8aa2b7b68e9a3ccab257f6ed7aa65ca not found: ID does not exist" containerID="33466972d2f1df00a526742b7a18372fa8aa2b7b68e9a3ccab257f6ed7aa65ca" Nov 04 05:06:28.738076 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:28.737784 4509 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33466972d2f1df00a526742b7a18372fa8aa2b7b68e9a3ccab257f6ed7aa65ca"} err="failed to get container status \"33466972d2f1df00a526742b7a18372fa8aa2b7b68e9a3ccab257f6ed7aa65ca\": rpc error: code = NotFound desc = could not find container \"33466972d2f1df00a526742b7a18372fa8aa2b7b68e9a3ccab257f6ed7aa65ca\": container with ID starting with 33466972d2f1df00a526742b7a18372fa8aa2b7b68e9a3ccab257f6ed7aa65ca not found: ID does not exist" Nov 04 05:06:28.738076 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:28.737810 4509 scope.go:122] "RemoveContainer" containerID="90901cf35b3a5c5999f8076892b42910a049c313e90fe487980febe641e77b6b" Nov 04 05:06:28.739141 ip-10-0-133-180 kubenswrapper[4509]: E1104 05:06:28.738777 4509 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90901cf35b3a5c5999f8076892b42910a049c313e90fe487980febe641e77b6b\": container with ID starting with 90901cf35b3a5c5999f8076892b42910a049c313e90fe487980febe641e77b6b not found: ID does not exist" containerID="90901cf35b3a5c5999f8076892b42910a049c313e90fe487980febe641e77b6b" Nov 04 05:06:28.739141 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:28.738813 4509 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90901cf35b3a5c5999f8076892b42910a049c313e90fe487980febe641e77b6b"} err="failed to get container status \"90901cf35b3a5c5999f8076892b42910a049c313e90fe487980febe641e77b6b\": rpc error: code = NotFound desc = could not find container \"90901cf35b3a5c5999f8076892b42910a049c313e90fe487980febe641e77b6b\": container with ID starting with 90901cf35b3a5c5999f8076892b42910a049c313e90fe487980febe641e77b6b not found: ID does not exist" Nov 04 05:06:28.739141 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:28.738833 4509 scope.go:122] "RemoveContainer" containerID="9f5a4844648e5c09869687b1f548d63b756b2a51a6ac37f646cf789559c04cbb" Nov 04 05:06:28.874348 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:28.873429 4509 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-4cce33-56666d7678-r88jh"] Nov 04 05:06:28.884350 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:28.880233 4509 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-4cce33-56666d7678-r88jh"] Nov 04 05:06:28.921795 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:28.921773 4509 scope.go:122] "RemoveContainer" containerID="9f5a4844648e5c09869687b1f548d63b756b2a51a6ac37f646cf789559c04cbb" Nov 04 05:06:28.922170 ip-10-0-133-180 kubenswrapper[4509]: E1104 05:06:28.922146 4509 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f5a4844648e5c09869687b1f548d63b756b2a51a6ac37f646cf789559c04cbb\": container with ID starting with 9f5a4844648e5c09869687b1f548d63b756b2a51a6ac37f646cf789559c04cbb not found: ID does not exist" containerID="9f5a4844648e5c09869687b1f548d63b756b2a51a6ac37f646cf789559c04cbb" Nov 04 05:06:28.922245 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:28.922182 4509 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f5a4844648e5c09869687b1f548d63b756b2a51a6ac37f646cf789559c04cbb"} err="failed to get container status \"9f5a4844648e5c09869687b1f548d63b756b2a51a6ac37f646cf789559c04cbb\": rpc error: code = NotFound desc = could not find container \"9f5a4844648e5c09869687b1f548d63b756b2a51a6ac37f646cf789559c04cbb\": container with ID starting with 9f5a4844648e5c09869687b1f548d63b756b2a51a6ac37f646cf789559c04cbb not found: ID does not exist" Nov 04 05:06:28.994433 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:28.994273 4509 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-m6qb5"] Nov 04 05:06:28.996454 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:28.996372 4509 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-m6qb5"] Nov 04 05:06:29.055182 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:29.055150 4509 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-lptr9" Nov 04 05:06:29.198268 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:29.198219 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff705a2e-8f20-49e5-aa26-69c7a8d1c31e-istio-token\" (UniqueName: \"kubernetes.io/projected/ff705a2e-8f20-49e5-aa26-69c7a8d1c31e-istio-token\") pod \"ff705a2e-8f20-49e5-aa26-69c7a8d1c31e\" (UID: \"ff705a2e-8f20-49e5-aa26-69c7a8d1c31e\") " Nov 04 05:06:29.198509 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:29.198296 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff705a2e-8f20-49e5-aa26-69c7a8d1c31e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff705a2e-8f20-49e5-aa26-69c7a8d1c31e-credential-socket\") pod \"ff705a2e-8f20-49e5-aa26-69c7a8d1c31e\" (UID: \"ff705a2e-8f20-49e5-aa26-69c7a8d1c31e\") " Nov 04 05:06:29.198509 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:29.198360 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff705a2e-8f20-49e5-aa26-69c7a8d1c31e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff705a2e-8f20-49e5-aa26-69c7a8d1c31e-istiod-ca-cert\") pod \"ff705a2e-8f20-49e5-aa26-69c7a8d1c31e\" (UID: \"ff705a2e-8f20-49e5-aa26-69c7a8d1c31e\") " Nov 04 05:06:29.198509 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:29.198426 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ff705a2e-8f20-49e5-aa26-69c7a8d1c31e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff705a2e-8f20-49e5-aa26-69c7a8d1c31e-istio-podinfo\") pod \"ff705a2e-8f20-49e5-aa26-69c7a8d1c31e\" (UID: \"ff705a2e-8f20-49e5-aa26-69c7a8d1c31e\") " Nov 04 05:06:29.198509 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:29.198499 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff705a2e-8f20-49e5-aa26-69c7a8d1c31e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff705a2e-8f20-49e5-aa26-69c7a8d1c31e-workload-socket\") pod \"ff705a2e-8f20-49e5-aa26-69c7a8d1c31e\" (UID: \"ff705a2e-8f20-49e5-aa26-69c7a8d1c31e\") " Nov 04 05:06:29.198749 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:29.198526 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff705a2e-8f20-49e5-aa26-69c7a8d1c31e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff705a2e-8f20-49e5-aa26-69c7a8d1c31e-istio-data\") pod \"ff705a2e-8f20-49e5-aa26-69c7a8d1c31e\" (UID: \"ff705a2e-8f20-49e5-aa26-69c7a8d1c31e\") " Nov 04 05:06:29.198749 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:29.198576 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff705a2e-8f20-49e5-aa26-69c7a8d1c31e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff705a2e-8f20-49e5-aa26-69c7a8d1c31e-workload-certs\") pod \"ff705a2e-8f20-49e5-aa26-69c7a8d1c31e\" (UID: \"ff705a2e-8f20-49e5-aa26-69c7a8d1c31e\") " Nov 04 05:06:29.198749 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:29.198623 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff705a2e-8f20-49e5-aa26-69c7a8d1c31e-kube-api-access-ccmws\" (UniqueName: \"kubernetes.io/projected/ff705a2e-8f20-49e5-aa26-69c7a8d1c31e-kube-api-access-ccmws\") pod \"ff705a2e-8f20-49e5-aa26-69c7a8d1c31e\" (UID: \"ff705a2e-8f20-49e5-aa26-69c7a8d1c31e\") " Nov 04 05:06:29.198749 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:29.198649 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff705a2e-8f20-49e5-aa26-69c7a8d1c31e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff705a2e-8f20-49e5-aa26-69c7a8d1c31e-istio-envoy\") pod \"ff705a2e-8f20-49e5-aa26-69c7a8d1c31e\" (UID: \"ff705a2e-8f20-49e5-aa26-69c7a8d1c31e\") " Nov 04 05:06:29.199005 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:29.198978 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff705a2e-8f20-49e5-aa26-69c7a8d1c31e-istio-data" pod "ff705a2e-8f20-49e5-aa26-69c7a8d1c31e" (UID: "ff705a2e-8f20-49e5-aa26-69c7a8d1c31e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:06:29.199225 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:29.199202 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff705a2e-8f20-49e5-aa26-69c7a8d1c31e-workload-socket" pod "ff705a2e-8f20-49e5-aa26-69c7a8d1c31e" (UID: "ff705a2e-8f20-49e5-aa26-69c7a8d1c31e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:06:29.199436 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:29.199412 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff705a2e-8f20-49e5-aa26-69c7a8d1c31e-workload-certs" pod "ff705a2e-8f20-49e5-aa26-69c7a8d1c31e" (UID: "ff705a2e-8f20-49e5-aa26-69c7a8d1c31e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:06:29.199895 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:29.199872 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff705a2e-8f20-49e5-aa26-69c7a8d1c31e-istiod-ca-cert" pod "ff705a2e-8f20-49e5-aa26-69c7a8d1c31e" (UID: "ff705a2e-8f20-49e5-aa26-69c7a8d1c31e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 05:06:29.200123 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:29.200102 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff705a2e-8f20-49e5-aa26-69c7a8d1c31e-credential-socket" pod "ff705a2e-8f20-49e5-aa26-69c7a8d1c31e" (UID: "ff705a2e-8f20-49e5-aa26-69c7a8d1c31e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:06:29.201883 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:29.201847 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff705a2e-8f20-49e5-aa26-69c7a8d1c31e-kube-api-access-ccmws" pod "ff705a2e-8f20-49e5-aa26-69c7a8d1c31e" (UID: "ff705a2e-8f20-49e5-aa26-69c7a8d1c31e"). InnerVolumeSpecName "kube-api-access-ccmws". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 05:06:29.202280 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:29.202260 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ff705a2e-8f20-49e5-aa26-69c7a8d1c31e-istio-podinfo" pod "ff705a2e-8f20-49e5-aa26-69c7a8d1c31e" (UID: "ff705a2e-8f20-49e5-aa26-69c7a8d1c31e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 05:06:29.203274 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:29.203257 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff705a2e-8f20-49e5-aa26-69c7a8d1c31e-istio-token" pod "ff705a2e-8f20-49e5-aa26-69c7a8d1c31e" (UID: "ff705a2e-8f20-49e5-aa26-69c7a8d1c31e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 05:06:29.203503 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:29.203486 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff705a2e-8f20-49e5-aa26-69c7a8d1c31e-istio-envoy" pod "ff705a2e-8f20-49e5-aa26-69c7a8d1c31e" (UID: "ff705a2e-8f20-49e5-aa26-69c7a8d1c31e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:06:29.300279 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:29.300188 4509 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff705a2e-8f20-49e5-aa26-69c7a8d1c31e-workload-certs\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:06:29.300279 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:29.300221 4509 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ccmws\" (UniqueName: \"kubernetes.io/projected/ff705a2e-8f20-49e5-aa26-69c7a8d1c31e-kube-api-access-ccmws\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:06:29.300279 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:29.300234 4509 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff705a2e-8f20-49e5-aa26-69c7a8d1c31e-istio-envoy\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:06:29.300279 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:29.300248 4509 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff705a2e-8f20-49e5-aa26-69c7a8d1c31e-istio-token\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:06:29.300279 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:29.300259 4509 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff705a2e-8f20-49e5-aa26-69c7a8d1c31e-credential-socket\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:06:29.300279 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:29.300271 4509 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff705a2e-8f20-49e5-aa26-69c7a8d1c31e-istiod-ca-cert\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:06:29.300279 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:29.300283 4509 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff705a2e-8f20-49e5-aa26-69c7a8d1c31e-istio-podinfo\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:06:29.300694 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:29.300295 4509 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff705a2e-8f20-49e5-aa26-69c7a8d1c31e-workload-socket\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:06:29.300694 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:29.300305 4509 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff705a2e-8f20-49e5-aa26-69c7a8d1c31e-istio-data\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:06:29.317666 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:29.317629 4509 patch_prober.go:29] interesting pod/e2e-worker-w5-4cce33-openshift-default-747768654f-85pc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Nov 04 05:06:29.317835 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:29.317686 4509 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-85pc8" podUID="37a5cce6-297b-4825-8710-bcb023060bb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Nov 04 05:06:29.554167 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:29.554077 4509 generic.go:356] "Generic (PLEG): container finished" podID="ff705a2e-8f20-49e5-aa26-69c7a8d1c31e" containerID="e2e35235e427b2ccb79c21ff7abb73521c066a44d17924ce426074e02f8d92a4" exitCode=0 Nov 04 05:06:29.554167 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:29.554130 4509 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-lptr9" Nov 04 05:06:29.554167 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:29.554139 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-lptr9" event={"ID":"ff705a2e-8f20-49e5-aa26-69c7a8d1c31e","Type":"ContainerDied","Data":"e2e35235e427b2ccb79c21ff7abb73521c066a44d17924ce426074e02f8d92a4"} Nov 04 05:06:29.554459 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:29.554191 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-lptr9" event={"ID":"ff705a2e-8f20-49e5-aa26-69c7a8d1c31e","Type":"ContainerDied","Data":"4a0e0fed34f4bc7006622a2c5bcbca99cedfb3c0cfab8298d26c460a6fb40ee2"} Nov 04 05:06:29.554459 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:29.554221 4509 scope.go:122] "RemoveContainer" containerID="e2e35235e427b2ccb79c21ff7abb73521c066a44d17924ce426074e02f8d92a4" Nov 04 05:06:29.564678 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:29.564658 4509 scope.go:122] "RemoveContainer" containerID="e2e35235e427b2ccb79c21ff7abb73521c066a44d17924ce426074e02f8d92a4" Nov 04 05:06:29.564985 ip-10-0-133-180 kubenswrapper[4509]: E1104 05:06:29.564964 4509 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2e35235e427b2ccb79c21ff7abb73521c066a44d17924ce426074e02f8d92a4\": container with ID starting with e2e35235e427b2ccb79c21ff7abb73521c066a44d17924ce426074e02f8d92a4 not found: ID does not exist" containerID="e2e35235e427b2ccb79c21ff7abb73521c066a44d17924ce426074e02f8d92a4" Nov 04 05:06:29.565052 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:29.564993 4509 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2e35235e427b2ccb79c21ff7abb73521c066a44d17924ce426074e02f8d92a4"} err="failed to get container status \"e2e35235e427b2ccb79c21ff7abb73521c066a44d17924ce426074e02f8d92a4\": rpc error: code = NotFound desc = could not find container \"e2e35235e427b2ccb79c21ff7abb73521c066a44d17924ce426074e02f8d92a4\": container with ID starting with e2e35235e427b2ccb79c21ff7abb73521c066a44d17924ce426074e02f8d92a4 not found: ID does not exist" Nov 04 05:06:29.569486 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:29.569455 4509 patch_prober.go:29] interesting pod/e2e-worker-w5-4cce33-openshift-default-747768654f-tkn4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Nov 04 05:06:29.569558 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:29.569515 4509 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-tkn4g" podUID="80cd56a4-9db7-4770-b1cc-a10754e3de43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Nov 04 05:06:29.720374 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:29.720307 4509 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-lptr9"] Nov 04 05:06:29.723946 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:29.723918 4509 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-lptr9"] Nov 04 05:06:30.267481 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:30.267450 4509 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a16804b-60cc-46b5-9879-163bda7bd069" path="/var/lib/kubelet/pods/2a16804b-60cc-46b5-9879-163bda7bd069/volumes" Nov 04 05:06:30.268350 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:30.268308 4509 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d751b557-4e07-4657-96ec-59fc0e70fd57" path="/var/lib/kubelet/pods/d751b557-4e07-4657-96ec-59fc0e70fd57/volumes" Nov 04 05:06:30.269275 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:30.269254 4509 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff705a2e-8f20-49e5-aa26-69c7a8d1c31e" path="/var/lib/kubelet/pods/ff705a2e-8f20-49e5-aa26-69c7a8d1c31e/volumes" Nov 04 05:06:30.862632 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:30.860676 4509 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-85pc8" Nov 04 05:06:30.862632 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:30.862029 4509 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-85pc8" Nov 04 05:06:30.863627 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:30.863605 4509 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-85pc8" Nov 04 05:06:31.046365 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:31.044601 4509 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-tkn4g" Nov 04 05:06:31.046365 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:31.045361 4509 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-tkn4g" Nov 04 05:06:31.046365 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:31.046216 4509 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-tkn4g" Nov 04 05:06:42.937622 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:42.937569 4509 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-tkn4g"] Nov 04 05:06:42.938158 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:42.937803 4509 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-tkn4g" podUID="80cd56a4-9db7-4770-b1cc-a10754e3de43" containerName="istio-proxy" containerID="cri-o://c1ba8d43f434b1dc83a07db47eb299158ad9ee6ab4ec8cb3599ffcfb91bd2dc5" gracePeriod=30 Nov 04 05:06:42.951887 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:42.951853 4509 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-85pc8"] Nov 04 05:06:42.952103 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:42.952061 4509 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-85pc8" podUID="37a5cce6-297b-4825-8710-bcb023060bb4" containerName="istio-proxy" containerID="cri-o://f3a21b7077cf2e04603805d64e5da5e9e6cc969699a1f3bb23599100e0b047f6" gracePeriod=30 Nov 04 05:06:42.971139 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:42.971105 4509 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-znf89"] Nov 04 05:06:42.971358 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:42.971336 4509 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-znf89" podUID="67ec6f66-3a36-4d01-be83-4ea5d42ba52e" containerName="istio-proxy" containerID="cri-o://2e57e7a19e5469496225664baefc6298927614f7fc55b23797a2a3c0daba3706" gracePeriod=30 Nov 04 05:06:44.455446 ip-10-0-133-180 kubenswrapper[4509]: E1104 05:06:44.455410 4509 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmdhl" containerName="extractor" Nov 04 05:06:44.989105 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:44.989061 4509 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-775668dcd7-z2z29_d90081c2-08dc-4d6d-b1eb-876a5e27584a/authorino/0.log" Nov 04 05:06:45.864818 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:45.864779 4509 patch_prober.go:29] interesting pod/e2e-worker-w5-4cce33-openshift-default-747768654f-85pc8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused Nov 04 05:06:45.865206 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:45.864838 4509 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-85pc8" podUID="37a5cce6-297b-4825-8710-bcb023060bb4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 05:06:46.047692 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:46.047657 4509 patch_prober.go:29] interesting pod/e2e-worker-w5-4cce33-openshift-default-747768654f-tkn4g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused Nov 04 05:06:46.047868 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:46.047714 4509 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-tkn4g" podUID="80cd56a4-9db7-4770-b1cc-a10754e3de43" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 05:06:48.229652 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:48.229629 4509 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-85pc8" Nov 04 05:06:48.250189 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:48.250167 4509 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-tkn4g" Nov 04 05:06:48.260285 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:48.260261 4509 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-znf89" Nov 04 05:06:48.310281 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:48.310252 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/37a5cce6-297b-4825-8710-bcb023060bb4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37a5cce6-297b-4825-8710-bcb023060bb4-istio-podinfo\") pod \"37a5cce6-297b-4825-8710-bcb023060bb4\" (UID: \"37a5cce6-297b-4825-8710-bcb023060bb4\") " Nov 04 05:06:48.310449 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:48.310295 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67ec6f66-3a36-4d01-be83-4ea5d42ba52e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67ec6f66-3a36-4d01-be83-4ea5d42ba52e-istiod-ca-cert\") pod \"67ec6f66-3a36-4d01-be83-4ea5d42ba52e\" (UID: \"67ec6f66-3a36-4d01-be83-4ea5d42ba52e\") " Nov 04 05:06:48.310449 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:48.310343 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80cd56a4-9db7-4770-b1cc-a10754e3de43-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80cd56a4-9db7-4770-b1cc-a10754e3de43-credential-socket\") pod \"80cd56a4-9db7-4770-b1cc-a10754e3de43\" (UID: \"80cd56a4-9db7-4770-b1cc-a10754e3de43\") " Nov 04 05:06:48.310449 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:48.310384 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37a5cce6-297b-4825-8710-bcb023060bb4-istio-token\" (UniqueName: \"kubernetes.io/projected/37a5cce6-297b-4825-8710-bcb023060bb4-istio-token\") pod \"37a5cce6-297b-4825-8710-bcb023060bb4\" (UID: \"37a5cce6-297b-4825-8710-bcb023060bb4\") " Nov 04 05:06:48.310449 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:48.310412 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37a5cce6-297b-4825-8710-bcb023060bb4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37a5cce6-297b-4825-8710-bcb023060bb4-workload-socket\") pod \"37a5cce6-297b-4825-8710-bcb023060bb4\" (UID: \"37a5cce6-297b-4825-8710-bcb023060bb4\") " Nov 04 05:06:48.310449 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:48.310442 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37a5cce6-297b-4825-8710-bcb023060bb4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37a5cce6-297b-4825-8710-bcb023060bb4-credential-socket\") pod \"37a5cce6-297b-4825-8710-bcb023060bb4\" (UID: \"37a5cce6-297b-4825-8710-bcb023060bb4\") " Nov 04 05:06:48.310759 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:48.310469 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37a5cce6-297b-4825-8710-bcb023060bb4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37a5cce6-297b-4825-8710-bcb023060bb4-workload-certs\") pod \"37a5cce6-297b-4825-8710-bcb023060bb4\" (UID: \"37a5cce6-297b-4825-8710-bcb023060bb4\") " Nov 04 05:06:48.310759 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:48.310504 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67ec6f66-3a36-4d01-be83-4ea5d42ba52e-kube-api-access-gm95c\" (UniqueName: \"kubernetes.io/projected/67ec6f66-3a36-4d01-be83-4ea5d42ba52e-kube-api-access-gm95c\") pod \"67ec6f66-3a36-4d01-be83-4ea5d42ba52e\" (UID: \"67ec6f66-3a36-4d01-be83-4ea5d42ba52e\") " Nov 04 05:06:48.310759 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:48.310532 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/80cd56a4-9db7-4770-b1cc-a10754e3de43-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80cd56a4-9db7-4770-b1cc-a10754e3de43-istio-podinfo\") pod \"80cd56a4-9db7-4770-b1cc-a10754e3de43\" (UID: \"80cd56a4-9db7-4770-b1cc-a10754e3de43\") " Nov 04 05:06:48.310759 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:48.310562 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80cd56a4-9db7-4770-b1cc-a10754e3de43-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80cd56a4-9db7-4770-b1cc-a10754e3de43-istio-data\") pod \"80cd56a4-9db7-4770-b1cc-a10754e3de43\" (UID: \"80cd56a4-9db7-4770-b1cc-a10754e3de43\") " Nov 04 05:06:48.310759 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:48.310610 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80cd56a4-9db7-4770-b1cc-a10754e3de43-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80cd56a4-9db7-4770-b1cc-a10754e3de43-workload-certs\") pod \"80cd56a4-9db7-4770-b1cc-a10754e3de43\" (UID: \"80cd56a4-9db7-4770-b1cc-a10754e3de43\") " Nov 04 05:06:48.310759 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:48.310635 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/67ec6f66-3a36-4d01-be83-4ea5d42ba52e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67ec6f66-3a36-4d01-be83-4ea5d42ba52e-istio-podinfo\") pod \"67ec6f66-3a36-4d01-be83-4ea5d42ba52e\" (UID: \"67ec6f66-3a36-4d01-be83-4ea5d42ba52e\") " Nov 04 05:06:48.310759 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:48.310677 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80cd56a4-9db7-4770-b1cc-a10754e3de43-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80cd56a4-9db7-4770-b1cc-a10754e3de43-istiod-ca-cert\") pod \"80cd56a4-9db7-4770-b1cc-a10754e3de43\" (UID: \"80cd56a4-9db7-4770-b1cc-a10754e3de43\") " Nov 04 05:06:48.310759 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:48.310682 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80cd56a4-9db7-4770-b1cc-a10754e3de43-credential-socket" pod "80cd56a4-9db7-4770-b1cc-a10754e3de43" (UID: "80cd56a4-9db7-4770-b1cc-a10754e3de43"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:06:48.310759 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:48.310693 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37a5cce6-297b-4825-8710-bcb023060bb4-workload-socket" pod "37a5cce6-297b-4825-8710-bcb023060bb4" (UID: "37a5cce6-297b-4825-8710-bcb023060bb4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:06:48.310759 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:48.310725 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37a5cce6-297b-4825-8710-bcb023060bb4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37a5cce6-297b-4825-8710-bcb023060bb4-istio-envoy\") pod \"37a5cce6-297b-4825-8710-bcb023060bb4\" (UID: \"37a5cce6-297b-4825-8710-bcb023060bb4\") " Nov 04 05:06:48.310759 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:48.310744 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67ec6f66-3a36-4d01-be83-4ea5d42ba52e-istiod-ca-cert" pod "67ec6f66-3a36-4d01-be83-4ea5d42ba52e" (UID: "67ec6f66-3a36-4d01-be83-4ea5d42ba52e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 05:06:48.311395 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:48.310777 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80cd56a4-9db7-4770-b1cc-a10754e3de43-kube-api-access-56rsm\" (UniqueName: \"kubernetes.io/projected/80cd56a4-9db7-4770-b1cc-a10754e3de43-kube-api-access-56rsm\") pod \"80cd56a4-9db7-4770-b1cc-a10754e3de43\" (UID: \"80cd56a4-9db7-4770-b1cc-a10754e3de43\") " Nov 04 05:06:48.311395 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:48.310814 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80cd56a4-9db7-4770-b1cc-a10754e3de43-istio-token\" (UniqueName: \"kubernetes.io/projected/80cd56a4-9db7-4770-b1cc-a10754e3de43-istio-token\") pod \"80cd56a4-9db7-4770-b1cc-a10754e3de43\" (UID: \"80cd56a4-9db7-4770-b1cc-a10754e3de43\") " Nov 04 05:06:48.311395 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:48.310852 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67ec6f66-3a36-4d01-be83-4ea5d42ba52e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67ec6f66-3a36-4d01-be83-4ea5d42ba52e-istio-envoy\") pod \"67ec6f66-3a36-4d01-be83-4ea5d42ba52e\" (UID: \"67ec6f66-3a36-4d01-be83-4ea5d42ba52e\") " Nov 04 05:06:48.311395 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:48.310892 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37a5cce6-297b-4825-8710-bcb023060bb4-kube-api-access-d58vv\" (UniqueName: \"kubernetes.io/projected/37a5cce6-297b-4825-8710-bcb023060bb4-kube-api-access-d58vv\") pod \"37a5cce6-297b-4825-8710-bcb023060bb4\" (UID: \"37a5cce6-297b-4825-8710-bcb023060bb4\") " Nov 04 05:06:48.311395 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:48.310911 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37a5cce6-297b-4825-8710-bcb023060bb4-credential-socket" pod "37a5cce6-297b-4825-8710-bcb023060bb4" (UID: "37a5cce6-297b-4825-8710-bcb023060bb4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:06:48.311395 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:48.310919 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67ec6f66-3a36-4d01-be83-4ea5d42ba52e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67ec6f66-3a36-4d01-be83-4ea5d42ba52e-workload-socket\") pod \"67ec6f66-3a36-4d01-be83-4ea5d42ba52e\" (UID: \"67ec6f66-3a36-4d01-be83-4ea5d42ba52e\") " Nov 04 05:06:48.311395 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:48.310937 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80cd56a4-9db7-4770-b1cc-a10754e3de43-workload-certs" pod "80cd56a4-9db7-4770-b1cc-a10754e3de43" (UID: "80cd56a4-9db7-4770-b1cc-a10754e3de43"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:06:48.311395 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:48.310949 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37a5cce6-297b-4825-8710-bcb023060bb4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37a5cce6-297b-4825-8710-bcb023060bb4-istiod-ca-cert\") pod \"37a5cce6-297b-4825-8710-bcb023060bb4\" (UID: \"37a5cce6-297b-4825-8710-bcb023060bb4\") " Nov 04 05:06:48.311395 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:48.310984 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67ec6f66-3a36-4d01-be83-4ea5d42ba52e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67ec6f66-3a36-4d01-be83-4ea5d42ba52e-workload-certs\") pod \"67ec6f66-3a36-4d01-be83-4ea5d42ba52e\" (UID: \"67ec6f66-3a36-4d01-be83-4ea5d42ba52e\") " Nov 04 05:06:48.311395 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:48.310983 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37a5cce6-297b-4825-8710-bcb023060bb4-workload-certs" pod "37a5cce6-297b-4825-8710-bcb023060bb4" (UID: "37a5cce6-297b-4825-8710-bcb023060bb4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:06:48.311395 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:48.311212 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67ec6f66-3a36-4d01-be83-4ea5d42ba52e-workload-socket" pod "67ec6f66-3a36-4d01-be83-4ea5d42ba52e" (UID: "67ec6f66-3a36-4d01-be83-4ea5d42ba52e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:06:48.312874 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:48.311521 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37a5cce6-297b-4825-8710-bcb023060bb4-istiod-ca-cert" pod "37a5cce6-297b-4825-8710-bcb023060bb4" (UID: "37a5cce6-297b-4825-8710-bcb023060bb4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 05:06:48.312874 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:48.311618 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80cd56a4-9db7-4770-b1cc-a10754e3de43-istiod-ca-cert" pod "80cd56a4-9db7-4770-b1cc-a10754e3de43" (UID: "80cd56a4-9db7-4770-b1cc-a10754e3de43"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 05:06:48.312874 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:48.311758 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80cd56a4-9db7-4770-b1cc-a10754e3de43-istio-data" pod "80cd56a4-9db7-4770-b1cc-a10754e3de43" (UID: "80cd56a4-9db7-4770-b1cc-a10754e3de43"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:06:48.312874 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:48.311848 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67ec6f66-3a36-4d01-be83-4ea5d42ba52e-workload-certs" pod "67ec6f66-3a36-4d01-be83-4ea5d42ba52e" (UID: "67ec6f66-3a36-4d01-be83-4ea5d42ba52e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:06:48.312874 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:48.311873 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67ec6f66-3a36-4d01-be83-4ea5d42ba52e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67ec6f66-3a36-4d01-be83-4ea5d42ba52e-istio-data\") pod \"67ec6f66-3a36-4d01-be83-4ea5d42ba52e\" (UID: \"67ec6f66-3a36-4d01-be83-4ea5d42ba52e\") " Nov 04 05:06:48.312874 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:48.311924 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67ec6f66-3a36-4d01-be83-4ea5d42ba52e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67ec6f66-3a36-4d01-be83-4ea5d42ba52e-credential-socket\") pod \"67ec6f66-3a36-4d01-be83-4ea5d42ba52e\" (UID: \"67ec6f66-3a36-4d01-be83-4ea5d42ba52e\") " Nov 04 05:06:48.312874 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:48.311959 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67ec6f66-3a36-4d01-be83-4ea5d42ba52e-istio-token\" (UniqueName: \"kubernetes.io/projected/67ec6f66-3a36-4d01-be83-4ea5d42ba52e-istio-token\") pod \"67ec6f66-3a36-4d01-be83-4ea5d42ba52e\" (UID: \"67ec6f66-3a36-4d01-be83-4ea5d42ba52e\") " Nov 04 05:06:48.312874 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:48.311989 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80cd56a4-9db7-4770-b1cc-a10754e3de43-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80cd56a4-9db7-4770-b1cc-a10754e3de43-istio-envoy\") pod \"80cd56a4-9db7-4770-b1cc-a10754e3de43\" (UID: \"80cd56a4-9db7-4770-b1cc-a10754e3de43\") " Nov 04 05:06:48.312874 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:48.312032 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80cd56a4-9db7-4770-b1cc-a10754e3de43-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80cd56a4-9db7-4770-b1cc-a10754e3de43-workload-socket\") pod \"80cd56a4-9db7-4770-b1cc-a10754e3de43\" (UID: \"80cd56a4-9db7-4770-b1cc-a10754e3de43\") " Nov 04 05:06:48.312874 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:48.312067 4509 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37a5cce6-297b-4825-8710-bcb023060bb4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37a5cce6-297b-4825-8710-bcb023060bb4-istio-data\") pod \"37a5cce6-297b-4825-8710-bcb023060bb4\" (UID: \"37a5cce6-297b-4825-8710-bcb023060bb4\") " Nov 04 05:06:48.312874 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:48.312456 4509 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67ec6f66-3a36-4d01-be83-4ea5d42ba52e-istiod-ca-cert\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:06:48.312874 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:48.312479 4509 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80cd56a4-9db7-4770-b1cc-a10754e3de43-credential-socket\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:06:48.312874 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:48.312512 4509 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37a5cce6-297b-4825-8710-bcb023060bb4-workload-socket\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:06:48.312874 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:48.312527 4509 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37a5cce6-297b-4825-8710-bcb023060bb4-credential-socket\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:06:48.312874 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:48.312541 4509 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37a5cce6-297b-4825-8710-bcb023060bb4-workload-certs\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:06:48.312874 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:48.312580 4509 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80cd56a4-9db7-4770-b1cc-a10754e3de43-istio-data\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:06:48.312874 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:48.312595 4509 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80cd56a4-9db7-4770-b1cc-a10754e3de43-workload-certs\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:06:48.312874 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:48.312608 4509 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80cd56a4-9db7-4770-b1cc-a10754e3de43-istiod-ca-cert\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:06:48.312874 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:48.312622 4509 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67ec6f66-3a36-4d01-be83-4ea5d42ba52e-workload-socket\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:06:48.312874 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:48.312636 4509 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37a5cce6-297b-4825-8710-bcb023060bb4-istiod-ca-cert\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:06:48.312874 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:48.312649 4509 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67ec6f66-3a36-4d01-be83-4ea5d42ba52e-workload-certs\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:06:48.312874 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:48.312853 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37a5cce6-297b-4825-8710-bcb023060bb4-istio-data" pod "37a5cce6-297b-4825-8710-bcb023060bb4" (UID: "37a5cce6-297b-4825-8710-bcb023060bb4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:06:48.314507 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:48.313027 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67ec6f66-3a36-4d01-be83-4ea5d42ba52e-istio-data" pod "67ec6f66-3a36-4d01-be83-4ea5d42ba52e" (UID: "67ec6f66-3a36-4d01-be83-4ea5d42ba52e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:06:48.314507 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:48.313263 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67ec6f66-3a36-4d01-be83-4ea5d42ba52e-credential-socket" pod "67ec6f66-3a36-4d01-be83-4ea5d42ba52e" (UID: "67ec6f66-3a36-4d01-be83-4ea5d42ba52e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:06:48.314507 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:48.314189 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80cd56a4-9db7-4770-b1cc-a10754e3de43-workload-socket" pod "80cd56a4-9db7-4770-b1cc-a10754e3de43" (UID: "80cd56a4-9db7-4770-b1cc-a10754e3de43"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:06:48.314507 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:48.314499 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/37a5cce6-297b-4825-8710-bcb023060bb4-istio-podinfo" pod "37a5cce6-297b-4825-8710-bcb023060bb4" (UID: "37a5cce6-297b-4825-8710-bcb023060bb4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 05:06:48.314953 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:48.314522 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80cd56a4-9db7-4770-b1cc-a10754e3de43-istio-token" pod "80cd56a4-9db7-4770-b1cc-a10754e3de43" (UID: "80cd56a4-9db7-4770-b1cc-a10754e3de43"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 05:06:48.315442 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:48.315389 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67ec6f66-3a36-4d01-be83-4ea5d42ba52e-kube-api-access-gm95c" pod "67ec6f66-3a36-4d01-be83-4ea5d42ba52e" (UID: "67ec6f66-3a36-4d01-be83-4ea5d42ba52e"). InnerVolumeSpecName "kube-api-access-gm95c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 05:06:48.315442 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:48.315399 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80cd56a4-9db7-4770-b1cc-a10754e3de43-kube-api-access-56rsm" pod "80cd56a4-9db7-4770-b1cc-a10754e3de43" (UID: "80cd56a4-9db7-4770-b1cc-a10754e3de43"). InnerVolumeSpecName "kube-api-access-56rsm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 05:06:48.315799 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:48.315609 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37a5cce6-297b-4825-8710-bcb023060bb4-istio-envoy" pod "37a5cce6-297b-4825-8710-bcb023060bb4" (UID: "37a5cce6-297b-4825-8710-bcb023060bb4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:06:48.315799 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:48.315667 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/67ec6f66-3a36-4d01-be83-4ea5d42ba52e-istio-podinfo" pod "67ec6f66-3a36-4d01-be83-4ea5d42ba52e" (UID: "67ec6f66-3a36-4d01-be83-4ea5d42ba52e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 05:06:48.316054 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:48.316028 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37a5cce6-297b-4825-8710-bcb023060bb4-istio-token" pod "37a5cce6-297b-4825-8710-bcb023060bb4" (UID: "37a5cce6-297b-4825-8710-bcb023060bb4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 05:06:48.316054 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:48.316033 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37a5cce6-297b-4825-8710-bcb023060bb4-kube-api-access-d58vv" pod "37a5cce6-297b-4825-8710-bcb023060bb4" (UID: "37a5cce6-297b-4825-8710-bcb023060bb4"). InnerVolumeSpecName "kube-api-access-d58vv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 05:06:48.316162 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:48.316147 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67ec6f66-3a36-4d01-be83-4ea5d42ba52e-istio-envoy" pod "67ec6f66-3a36-4d01-be83-4ea5d42ba52e" (UID: "67ec6f66-3a36-4d01-be83-4ea5d42ba52e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:06:48.316390 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:48.316369 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80cd56a4-9db7-4770-b1cc-a10754e3de43-istio-envoy" pod "80cd56a4-9db7-4770-b1cc-a10754e3de43" (UID: "80cd56a4-9db7-4770-b1cc-a10754e3de43"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:06:48.317217 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:48.317199 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/80cd56a4-9db7-4770-b1cc-a10754e3de43-istio-podinfo" pod "80cd56a4-9db7-4770-b1cc-a10754e3de43" (UID: "80cd56a4-9db7-4770-b1cc-a10754e3de43"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 05:06:48.317282 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:48.317267 4509 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67ec6f66-3a36-4d01-be83-4ea5d42ba52e-istio-token" pod "67ec6f66-3a36-4d01-be83-4ea5d42ba52e" (UID: "67ec6f66-3a36-4d01-be83-4ea5d42ba52e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 05:06:48.413980 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:48.413948 4509 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67ec6f66-3a36-4d01-be83-4ea5d42ba52e-istio-podinfo\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:06:48.413980 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:48.413977 4509 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37a5cce6-297b-4825-8710-bcb023060bb4-istio-envoy\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:06:48.413980 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:48.413986 4509 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56rsm\" (UniqueName: \"kubernetes.io/projected/80cd56a4-9db7-4770-b1cc-a10754e3de43-kube-api-access-56rsm\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:06:48.414187 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:48.413996 4509 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80cd56a4-9db7-4770-b1cc-a10754e3de43-istio-token\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:06:48.414187 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:48.414005 4509 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67ec6f66-3a36-4d01-be83-4ea5d42ba52e-istio-envoy\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:06:48.414187 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:48.414013 4509 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d58vv\" (UniqueName: \"kubernetes.io/projected/37a5cce6-297b-4825-8710-bcb023060bb4-kube-api-access-d58vv\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:06:48.414187 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:48.414021 4509 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67ec6f66-3a36-4d01-be83-4ea5d42ba52e-istio-data\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:06:48.414187 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:48.414031 4509 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67ec6f66-3a36-4d01-be83-4ea5d42ba52e-credential-socket\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:06:48.414187 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:48.414040 4509 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67ec6f66-3a36-4d01-be83-4ea5d42ba52e-istio-token\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:06:48.414187 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:48.414050 4509 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80cd56a4-9db7-4770-b1cc-a10754e3de43-istio-envoy\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:06:48.414187 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:48.414058 4509 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80cd56a4-9db7-4770-b1cc-a10754e3de43-workload-socket\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:06:48.414187 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:48.414067 4509 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37a5cce6-297b-4825-8710-bcb023060bb4-istio-data\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:06:48.414187 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:48.414075 4509 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37a5cce6-297b-4825-8710-bcb023060bb4-istio-podinfo\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:06:48.414187 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:48.414083 4509 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37a5cce6-297b-4825-8710-bcb023060bb4-istio-token\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:06:48.414187 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:48.414091 4509 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gm95c\" (UniqueName: \"kubernetes.io/projected/67ec6f66-3a36-4d01-be83-4ea5d42ba52e-kube-api-access-gm95c\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:06:48.414187 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:48.414098 4509 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80cd56a4-9db7-4770-b1cc-a10754e3de43-istio-podinfo\") on node \"ip-10-0-133-180.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 05:06:48.663220 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:48.663117 4509 generic.go:356] "Generic (PLEG): container finished" podID="37a5cce6-297b-4825-8710-bcb023060bb4" containerID="f3a21b7077cf2e04603805d64e5da5e9e6cc969699a1f3bb23599100e0b047f6" exitCode=0 Nov 04 05:06:48.663220 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:48.663171 4509 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-85pc8" Nov 04 05:06:48.663455 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:48.663213 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-85pc8" event={"ID":"37a5cce6-297b-4825-8710-bcb023060bb4","Type":"ContainerDied","Data":"f3a21b7077cf2e04603805d64e5da5e9e6cc969699a1f3bb23599100e0b047f6"} Nov 04 05:06:48.663455 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:48.663267 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-85pc8" event={"ID":"37a5cce6-297b-4825-8710-bcb023060bb4","Type":"ContainerDied","Data":"08f5bd3bd688b8eb0b7c4eb0615a25856d0925bba9943403c5af8526e1b2fa45"} Nov 04 05:06:48.663455 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:48.663284 4509 scope.go:122] "RemoveContainer" containerID="f3a21b7077cf2e04603805d64e5da5e9e6cc969699a1f3bb23599100e0b047f6" Nov 04 05:06:48.666305 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:48.666271 4509 generic.go:356] "Generic (PLEG): container finished" podID="80cd56a4-9db7-4770-b1cc-a10754e3de43" containerID="c1ba8d43f434b1dc83a07db47eb299158ad9ee6ab4ec8cb3599ffcfb91bd2dc5" exitCode=0 Nov 04 05:06:48.666449 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:48.666355 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-tkn4g" event={"ID":"80cd56a4-9db7-4770-b1cc-a10754e3de43","Type":"ContainerDied","Data":"c1ba8d43f434b1dc83a07db47eb299158ad9ee6ab4ec8cb3599ffcfb91bd2dc5"} Nov 04 05:06:48.666449 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:48.666401 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-tkn4g" event={"ID":"80cd56a4-9db7-4770-b1cc-a10754e3de43","Type":"ContainerDied","Data":"ffe126a2fa009e3b63a8ad7f9d23c807ce351682305e64db301b53783e0fd0e3"} Nov 04 05:06:48.666568 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:48.666496 4509 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-tkn4g" Nov 04 05:06:48.668433 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:48.668407 4509 generic.go:356] "Generic (PLEG): container finished" podID="67ec6f66-3a36-4d01-be83-4ea5d42ba52e" containerID="2e57e7a19e5469496225664baefc6298927614f7fc55b23797a2a3c0daba3706" exitCode=0 Nov 04 05:06:48.668613 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:48.668436 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-znf89" event={"ID":"67ec6f66-3a36-4d01-be83-4ea5d42ba52e","Type":"ContainerDied","Data":"2e57e7a19e5469496225664baefc6298927614f7fc55b23797a2a3c0daba3706"} Nov 04 05:06:48.668613 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:48.668522 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-znf89" event={"ID":"67ec6f66-3a36-4d01-be83-4ea5d42ba52e","Type":"ContainerDied","Data":"ce995f29b532a90db65521bc5f44e3071c0c84984adbb40ffdab86f87ef089c0"} Nov 04 05:06:48.668613 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:48.668462 4509 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-znf89" Nov 04 05:06:48.675902 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:48.675884 4509 scope.go:122] "RemoveContainer" containerID="f3a21b7077cf2e04603805d64e5da5e9e6cc969699a1f3bb23599100e0b047f6" Nov 04 05:06:48.676168 ip-10-0-133-180 kubenswrapper[4509]: E1104 05:06:48.676149 4509 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3a21b7077cf2e04603805d64e5da5e9e6cc969699a1f3bb23599100e0b047f6\": container with ID starting with f3a21b7077cf2e04603805d64e5da5e9e6cc969699a1f3bb23599100e0b047f6 not found: ID does not exist" containerID="f3a21b7077cf2e04603805d64e5da5e9e6cc969699a1f3bb23599100e0b047f6" Nov 04 05:06:48.676250 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:48.676178 4509 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3a21b7077cf2e04603805d64e5da5e9e6cc969699a1f3bb23599100e0b047f6"} err="failed to get container status \"f3a21b7077cf2e04603805d64e5da5e9e6cc969699a1f3bb23599100e0b047f6\": rpc error: code = NotFound desc = could not find container \"f3a21b7077cf2e04603805d64e5da5e9e6cc969699a1f3bb23599100e0b047f6\": container with ID starting with f3a21b7077cf2e04603805d64e5da5e9e6cc969699a1f3bb23599100e0b047f6 not found: ID does not exist" Nov 04 05:06:48.676250 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:48.676195 4509 scope.go:122] "RemoveContainer" containerID="c1ba8d43f434b1dc83a07db47eb299158ad9ee6ab4ec8cb3599ffcfb91bd2dc5" Nov 04 05:06:48.685109 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:48.685095 4509 scope.go:122] "RemoveContainer" containerID="c1ba8d43f434b1dc83a07db47eb299158ad9ee6ab4ec8cb3599ffcfb91bd2dc5" Nov 04 05:06:48.685423 ip-10-0-133-180 kubenswrapper[4509]: E1104 05:06:48.685406 4509 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1ba8d43f434b1dc83a07db47eb299158ad9ee6ab4ec8cb3599ffcfb91bd2dc5\": container with ID starting with c1ba8d43f434b1dc83a07db47eb299158ad9ee6ab4ec8cb3599ffcfb91bd2dc5 not found: ID does not exist" containerID="c1ba8d43f434b1dc83a07db47eb299158ad9ee6ab4ec8cb3599ffcfb91bd2dc5" Nov 04 05:06:48.685471 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:48.685430 4509 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1ba8d43f434b1dc83a07db47eb299158ad9ee6ab4ec8cb3599ffcfb91bd2dc5"} err="failed to get container status \"c1ba8d43f434b1dc83a07db47eb299158ad9ee6ab4ec8cb3599ffcfb91bd2dc5\": rpc error: code = NotFound desc = could not find container \"c1ba8d43f434b1dc83a07db47eb299158ad9ee6ab4ec8cb3599ffcfb91bd2dc5\": container with ID starting with c1ba8d43f434b1dc83a07db47eb299158ad9ee6ab4ec8cb3599ffcfb91bd2dc5 not found: ID does not exist" Nov 04 05:06:48.685471 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:48.685447 4509 scope.go:122] "RemoveContainer" containerID="2e57e7a19e5469496225664baefc6298927614f7fc55b23797a2a3c0daba3706" Nov 04 05:06:48.693051 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:48.693035 4509 scope.go:122] "RemoveContainer" containerID="38fd3e3b00aad944d8c094dcef7e8eb6d1d7b4f72f2be1288d1c49e77bcd6097" Nov 04 05:06:48.700679 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:48.700662 4509 scope.go:122] "RemoveContainer" containerID="2e57e7a19e5469496225664baefc6298927614f7fc55b23797a2a3c0daba3706" Nov 04 05:06:48.700952 ip-10-0-133-180 kubenswrapper[4509]: E1104 05:06:48.700933 4509 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e57e7a19e5469496225664baefc6298927614f7fc55b23797a2a3c0daba3706\": container with ID starting with 2e57e7a19e5469496225664baefc6298927614f7fc55b23797a2a3c0daba3706 not found: ID does not exist" containerID="2e57e7a19e5469496225664baefc6298927614f7fc55b23797a2a3c0daba3706" Nov 04 05:06:48.701019 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:48.700961 4509 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e57e7a19e5469496225664baefc6298927614f7fc55b23797a2a3c0daba3706"} err="failed to get container status \"2e57e7a19e5469496225664baefc6298927614f7fc55b23797a2a3c0daba3706\": rpc error: code = NotFound desc = could not find container \"2e57e7a19e5469496225664baefc6298927614f7fc55b23797a2a3c0daba3706\": container with ID starting with 2e57e7a19e5469496225664baefc6298927614f7fc55b23797a2a3c0daba3706 not found: ID does not exist" Nov 04 05:06:48.701019 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:48.700979 4509 scope.go:122] "RemoveContainer" containerID="38fd3e3b00aad944d8c094dcef7e8eb6d1d7b4f72f2be1288d1c49e77bcd6097" Nov 04 05:06:48.701261 ip-10-0-133-180 kubenswrapper[4509]: E1104 05:06:48.701236 4509 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38fd3e3b00aad944d8c094dcef7e8eb6d1d7b4f72f2be1288d1c49e77bcd6097\": container with ID starting with 38fd3e3b00aad944d8c094dcef7e8eb6d1d7b4f72f2be1288d1c49e77bcd6097 not found: ID does not exist" containerID="38fd3e3b00aad944d8c094dcef7e8eb6d1d7b4f72f2be1288d1c49e77bcd6097" Nov 04 05:06:48.701304 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:48.701267 4509 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38fd3e3b00aad944d8c094dcef7e8eb6d1d7b4f72f2be1288d1c49e77bcd6097"} err="failed to get container status \"38fd3e3b00aad944d8c094dcef7e8eb6d1d7b4f72f2be1288d1c49e77bcd6097\": rpc error: code = NotFound desc = could not find container \"38fd3e3b00aad944d8c094dcef7e8eb6d1d7b4f72f2be1288d1c49e77bcd6097\": container with ID starting with 38fd3e3b00aad944d8c094dcef7e8eb6d1d7b4f72f2be1288d1c49e77bcd6097 not found: ID does not exist" Nov 04 05:06:48.828044 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:48.828003 4509 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-znf89"] Nov 04 05:06:48.831749 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:48.831725 4509 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-znf89"] Nov 04 05:06:48.940998 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:48.940925 4509 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-tkn4g"] Nov 04 05:06:48.944813 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:48.944789 4509 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-tkn4g"] Nov 04 05:06:49.056971 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:49.056933 4509 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-85pc8"] Nov 04 05:06:49.060636 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:49.060610 4509 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-85pc8"] Nov 04 05:06:50.260433 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:50.260404 4509 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37a5cce6-297b-4825-8710-bcb023060bb4" path="/var/lib/kubelet/pods/37a5cce6-297b-4825-8710-bcb023060bb4/volumes" Nov 04 05:06:50.260865 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:50.260851 4509 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67ec6f66-3a36-4d01-be83-4ea5d42ba52e" path="/var/lib/kubelet/pods/67ec6f66-3a36-4d01-be83-4ea5d42ba52e/volumes" Nov 04 05:06:50.261224 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:06:50.261213 4509 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80cd56a4-9db7-4770-b1cc-a10754e3de43" path="/var/lib/kubelet/pods/80cd56a4-9db7-4770-b1cc-a10754e3de43/volumes" Nov 04 05:07:14.453302 ip-10-0-133-180 kubenswrapper[4509]: E1104 05:07:14.453263 4509 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmdhl" containerName="extractor" Nov 04 05:07:17.412579 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:07:17.412550 4509 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-64466f8c8d-bg6dc_242c7b88-7028-4b0f-a9e5-2e9fd16bc446/manager/0.log" Nov 04 05:07:18.139645 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:07:18.139611 4509 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-glcvz_b26b2ef7-b5a7-4933-8393-f830db7d1387/manager/0.log" Nov 04 05:07:19.049310 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:07:19.049286 4509 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-7pj6b_b4f68592-75e2-4d18-9b72-f987c072af88/server/0.log" Nov 04 05:07:41.466007 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:07:41.465937 4509 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-39ccb86b-6j5fn_db69c455-c747-4c73-8c2f-752c26822262/revoke-keys/0.log" Nov 04 05:07:41.698017 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:07:41.697990 4509 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3b12986e-h6fr5_950a4971-bcd5-43b7-a2c6-7545c36a63ea/revoke-keys/0.log" Nov 04 05:07:41.931490 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:07:41.931411 4509 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3c065f20-4452q_6e5e0f05-a7c1-497c-8846-fc11b15d3e0c/revoke-keys/0.log" Nov 04 05:07:42.169608 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:07:42.169579 4509 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6c621778-k2mxk_b78d9eb7-681a-4f6e-a61b-790ee1e814fd/revoke-keys/0.log" Nov 04 05:07:42.399988 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:07:42.399957 4509 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-80b24922-sdsg5_0c9909c8-ffcb-4601-b38d-923c92a786d0/revoke-keys/0.log" Nov 04 05:07:42.614817 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:07:42.614789 4509 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-836dd023-qv9dl_e213f5c3-01ac-485d-a99b-59afd30ab150/revoke-keys/0.log" Nov 04 05:07:42.854219 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:07:42.854190 4509 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-85f49706-9jtsg_fdc76a64-cc07-4e07-b02d-3007a7f1e83f/revoke-keys/0.log" Nov 04 05:07:43.084502 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:07:43.084456 4509 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-9dc147b4-c8x2x_c76c7ef3-5e20-413d-9528-c0264ba54975/revoke-keys/0.log" Nov 04 05:07:43.314956 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:07:43.314934 4509 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-9ed9dfcd-sg78z_e13aa5f1-8d64-4ad9-86c0-1d6d3df9d98c/revoke-keys/0.log" Nov 04 05:07:43.548369 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:07:43.548341 4509 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-c3a13485-chkkp_83294360-4718-4139-8287-8ec8ee6f72ce/revoke-keys/0.log" Nov 04 05:07:44.236860 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:07:44.236830 4509 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-e9823838-jnppq_fab7a88a-df4b-41d2-b96a-ae4938a024c9/revoke-keys/0.log" Nov 04 05:07:44.455757 ip-10-0-133-180 kubenswrapper[4509]: E1104 05:07:44.455720 4509 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmdhl" containerName="extractor" Nov 04 05:07:44.466803 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:07:44.466781 4509 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-fdaae210-nptmk_b1179b74-131f-49ae-9e4b-da6c1ac617be/revoke-keys/0.log" Nov 04 05:07:46.848384 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:07:46.848352 4509 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-775668dcd7-z2z29_d90081c2-08dc-4d6d-b1eb-876a5e27584a/authorino/0.log" Nov 04 05:07:48.634924 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:07:48.634895 4509 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-42dnm_e0ffdeeb-4a8f-437b-80ab-c4c09c6e14bc/limitador/0.log" Nov 04 05:08:00.210395 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:08:00.210360 4509 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-f447m_a4d34feb-7fd9-4b77-a628-eb9d84adb93c/discovery/0.log" Nov 04 05:08:00.785545 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:08:00.785518 4509 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-546d78d5c-vbwrt_542ce577-5ec5-400e-a82b-d5d1f3fe8c97/kube-auth-proxy/0.log" Nov 04 05:08:01.016199 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:08:01.016168 4509 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-4sj9h_e057215d-2b26-47c6-8c61-3d1413ef6aec/istio-proxy/0.log" Nov 04 05:08:01.746804 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:08:01.746774 4509 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-n9sxt_dbe3a747-d280-4042-8cb9-62a148236fd2/payload-pre-processing/0.log" Nov 04 05:08:03.858197 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:08:03.858157 4509 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-vkcnr_03dca812-45a7-42e0-81ce-9dad9861776c/main/0.log" Nov 04 05:08:04.085044 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:08:04.085012 4509 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-wc5pz_5697f521-fbf5-4d8d-9e09-cfb64fb7827f/main/0.log" Nov 04 05:08:05.115625 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:08:05.115598 4509 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-qrvq5_d5162291-7c0d-4a1d-93bb-35b19fe7582b/main/0.log" Nov 04 05:08:05.400888 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:08:05.400816 4509 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-qsr5h_b59b3e50-b834-4624-b100-46a7df2e159f/main/0.log" Nov 04 05:08:06.650672 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:08:06.650639 4509 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-775668dcd7-z2z29_d90081c2-08dc-4d6d-b1eb-876a5e27584a/authorino/0.log" Nov 04 05:08:14.453933 ip-10-0-133-180 kubenswrapper[4509]: E1104 05:08:14.453895 4509 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmdhl" containerName="extractor" Nov 04 05:08:16.465297 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:08:16.465259 4509 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-64466f8c8d-bg6dc_242c7b88-7028-4b0f-a9e5-2e9fd16bc446/manager/0.log" Nov 04 05:08:17.029494 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:08:17.029465 4509 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-glcvz_b26b2ef7-b5a7-4933-8393-f830db7d1387/manager/0.log" Nov 04 05:08:17.931263 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:08:17.931234 4509 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-7pj6b_b4f68592-75e2-4d18-9b72-f987c072af88/server/0.log" Nov 04 05:08:19.684696 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:08:19.684667 4509 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-39ccb86b-6j5fn_db69c455-c747-4c73-8c2f-752c26822262/revoke-keys/0.log" Nov 04 05:08:19.936253 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:08:19.936171 4509 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3b12986e-h6fr5_950a4971-bcd5-43b7-a2c6-7545c36a63ea/revoke-keys/0.log" Nov 04 05:08:20.206418 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:08:20.206314 4509 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3c065f20-4452q_6e5e0f05-a7c1-497c-8846-fc11b15d3e0c/revoke-keys/0.log" Nov 04 05:08:20.457093 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:08:20.457005 4509 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6c621778-k2mxk_b78d9eb7-681a-4f6e-a61b-790ee1e814fd/revoke-keys/0.log" Nov 04 05:08:20.690006 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:08:20.689975 4509 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-80b24922-sdsg5_0c9909c8-ffcb-4601-b38d-923c92a786d0/revoke-keys/0.log" Nov 04 05:08:20.943708 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:08:20.943634 4509 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-836dd023-qv9dl_e213f5c3-01ac-485d-a99b-59afd30ab150/revoke-keys/0.log" Nov 04 05:08:21.201124 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:08:21.201029 4509 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-85f49706-9jtsg_fdc76a64-cc07-4e07-b02d-3007a7f1e83f/revoke-keys/0.log" Nov 04 05:08:21.464793 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:08:21.464761 4509 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-9dc147b4-c8x2x_c76c7ef3-5e20-413d-9528-c0264ba54975/revoke-keys/0.log" Nov 04 05:08:21.696501 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:08:21.696468 4509 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-9ed9dfcd-sg78z_e13aa5f1-8d64-4ad9-86c0-1d6d3df9d98c/revoke-keys/0.log" Nov 04 05:08:21.923408 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:08:21.923302 4509 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-c3a13485-chkkp_83294360-4718-4139-8287-8ec8ee6f72ce/revoke-keys/0.log" Nov 04 05:08:22.686836 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:08:22.686806 4509 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-e9823838-jnppq_fab7a88a-df4b-41d2-b96a-ae4938a024c9/revoke-keys/0.log" Nov 04 05:08:22.939760 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:08:22.939655 4509 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-fdaae210-nptmk_b1179b74-131f-49ae-9e4b-da6c1ac617be/revoke-keys/0.log" Nov 04 05:08:24.490936 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:08:24.490902 4509 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-775668dcd7-z2z29_d90081c2-08dc-4d6d-b1eb-876a5e27584a/authorino/0.log" Nov 04 05:08:26.080079 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:08:26.080053 4509 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-42dnm_e0ffdeeb-4a8f-437b-80ab-c4c09c6e14bc/limitador/0.log" Nov 04 05:08:37.277614 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:08:37.277576 4509 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-f447m_a4d34feb-7fd9-4b77-a628-eb9d84adb93c/discovery/0.log" Nov 04 05:08:37.861732 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:08:37.861702 4509 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-546d78d5c-vbwrt_542ce577-5ec5-400e-a82b-d5d1f3fe8c97/kube-auth-proxy/0.log" Nov 04 05:08:38.110956 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:08:38.110928 4509 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-4sj9h_e057215d-2b26-47c6-8c61-3d1413ef6aec/istio-proxy/0.log" Nov 04 05:08:38.926383 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:08:38.926352 4509 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-n9sxt_dbe3a747-d280-4042-8cb9-62a148236fd2/payload-pre-processing/0.log" Nov 04 05:08:40.277457 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:08:40.277426 4509 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-vkcnr_03dca812-45a7-42e0-81ce-9dad9861776c/main/0.log" Nov 04 05:08:40.541209 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:08:40.541126 4509 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-wc5pz_5697f521-fbf5-4d8d-9e09-cfb64fb7827f/main/0.log" Nov 04 05:08:41.596112 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:08:41.596080 4509 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-qrvq5_d5162291-7c0d-4a1d-93bb-35b19fe7582b/main/0.log" Nov 04 05:08:41.856140 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:08:41.856027 4509 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-qsr5h_b59b3e50-b834-4624-b100-46a7df2e159f/main/0.log" Nov 04 05:08:43.050072 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:08:43.050042 4509 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-836dd023-qv9dl"] Nov 04 05:08:43.099597 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:08:43.099562 4509 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-836dd023-qv9dl"] Nov 04 05:08:44.260472 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:08:44.260435 4509 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e213f5c3-01ac-485d-a99b-59afd30ab150" path="/var/lib/kubelet/pods/e213f5c3-01ac-485d-a99b-59afd30ab150/volumes" Nov 04 05:08:44.453864 ip-10-0-133-180 kubenswrapper[4509]: E1104 05:08:44.453832 4509 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmdhl" containerName="extractor" Nov 04 05:08:54.045278 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:08:54.045233 4509 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fdaae210-nptmk"] Nov 04 05:08:54.095018 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:08:54.094985 4509 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fdaae210-nptmk"] Nov 04 05:08:54.260862 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:08:54.260828 4509 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1179b74-131f-49ae-9e4b-da6c1ac617be" path="/var/lib/kubelet/pods/b1179b74-131f-49ae-9e4b-da6c1ac617be/volumes" Nov 04 05:08:55.045046 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:08:55.044999 4509 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85f49706-9jtsg"] Nov 04 05:08:55.094817 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:08:55.094784 4509 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85f49706-9jtsg"] Nov 04 05:08:56.260099 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:08:56.260067 4509 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fdc76a64-cc07-4e07-b02d-3007a7f1e83f" path="/var/lib/kubelet/pods/fdc76a64-cc07-4e07-b02d-3007a7f1e83f/volumes" Nov 04 05:09:11.046856 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:09:11.046806 4509 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c621778-k2mxk"] Nov 04 05:09:11.096279 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:09:11.096247 4509 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c621778-k2mxk"] Nov 04 05:09:12.260375 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:09:12.260342 4509 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b78d9eb7-681a-4f6e-a61b-790ee1e814fd" path="/var/lib/kubelet/pods/b78d9eb7-681a-4f6e-a61b-790ee1e814fd/volumes" Nov 04 05:09:14.453878 ip-10-0-133-180 kubenswrapper[4509]: E1104 05:09:14.453835 4509 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmdhl" containerName="extractor" Nov 04 05:09:27.045267 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:09:27.045226 4509 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ed9dfcd-sg78z"] Nov 04 05:09:27.094939 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:09:27.094893 4509 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ed9dfcd-sg78z"] Nov 04 05:09:28.260809 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:09:28.260767 4509 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e13aa5f1-8d64-4ad9-86c0-1d6d3df9d98c" path="/var/lib/kubelet/pods/e13aa5f1-8d64-4ad9-86c0-1d6d3df9d98c/volumes" Nov 04 05:09:44.045896 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:09:44.045854 4509 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3c065f20-4452q"] Nov 04 05:09:44.095473 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:09:44.095440 4509 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3c065f20-4452q"] Nov 04 05:09:44.261516 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:09:44.261473 4509 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e5e0f05-a7c1-497c-8846-fc11b15d3e0c" path="/var/lib/kubelet/pods/6e5e0f05-a7c1-497c-8846-fc11b15d3e0c/volumes" Nov 04 05:09:44.455109 ip-10-0-133-180 kubenswrapper[4509]: E1104 05:09:44.455028 4509 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmdhl" containerName="extractor" Nov 04 05:09:46.045476 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:09:46.045441 4509 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39ccb86b-6j5fn"] Nov 04 05:09:46.095235 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:09:46.095197 4509 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39ccb86b-6j5fn"] Nov 04 05:09:46.260733 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:09:46.260701 4509 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db69c455-c747-4c73-8c2f-752c26822262" path="/var/lib/kubelet/pods/db69c455-c747-4c73-8c2f-752c26822262/volumes" Nov 04 05:09:53.045178 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:09:53.045142 4509 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80b24922-sdsg5"] Nov 04 05:09:53.095068 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:09:53.095011 4509 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80b24922-sdsg5"] Nov 04 05:09:54.260433 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:09:54.260404 4509 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c9909c8-ffcb-4601-b38d-923c92a786d0" path="/var/lib/kubelet/pods/0c9909c8-ffcb-4601-b38d-923c92a786d0/volumes" Nov 04 05:10:02.044940 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:10:02.044907 4509 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9823838-jnppq"] Nov 04 05:10:02.094877 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:10:02.094839 4509 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9823838-jnppq"] Nov 04 05:10:02.260855 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:10:02.260822 4509 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fab7a88a-df4b-41d2-b96a-ae4938a024c9" path="/var/lib/kubelet/pods/fab7a88a-df4b-41d2-b96a-ae4938a024c9/volumes" Nov 04 05:10:10.054854 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:10:10.054820 4509 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b12986e-h6fr5"] Nov 04 05:10:10.104748 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:10:10.104717 4509 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b12986e-h6fr5"] Nov 04 05:10:10.261483 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:10:10.261451 4509 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="950a4971-bcd5-43b7-a2c6-7545c36a63ea" path="/var/lib/kubelet/pods/950a4971-bcd5-43b7-a2c6-7545c36a63ea/volumes" Nov 04 05:10:14.454223 ip-10-0-133-180 kubenswrapper[4509]: E1104 05:10:14.454183 4509 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmdhl" containerName="extractor" Nov 04 05:10:16.584130 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:10:16.584097 4509 scope.go:122] "RemoveContainer" containerID="39ee30ed5c2ddd85728e13268b7e011a7b8de794701c731d989e7c78e6e8f94b" Nov 04 05:10:16.592592 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:10:16.592570 4509 scope.go:122] "RemoveContainer" containerID="3ea9f98aa62c7295cd9e0db052eb0ed68db4851d10ac97bf62614efa8ba7775d" Nov 04 05:10:16.600582 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:10:16.600565 4509 scope.go:122] "RemoveContainer" containerID="9b0c3c9eb3d51eaeb33000115753efedda6be235980196e90013f85ac82561e0" Nov 04 05:10:16.609946 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:10:16.609925 4509 scope.go:122] "RemoveContainer" containerID="954f1b5cdc2f3ef29fd8139d89b5a36de54be8cb3ed0b05a15005bbd8e310ef1" Nov 04 05:10:44.453953 ip-10-0-133-180 kubenswrapper[4509]: E1104 05:10:44.453868 4509 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmdhl" containerName="extractor" Nov 04 05:10:56.045630 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:10:56.045594 4509 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9dc147b4-c8x2x"] Nov 04 05:10:56.095495 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:10:56.095460 4509 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9dc147b4-c8x2x"] Nov 04 05:10:56.260570 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:10:56.260535 4509 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c76c7ef3-5e20-413d-9528-c0264ba54975" path="/var/lib/kubelet/pods/c76c7ef3-5e20-413d-9528-c0264ba54975/volumes" Nov 04 05:11:05.044693 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:11:05.044642 4509 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3a13485-chkkp"] Nov 04 05:11:05.094515 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:11:05.094480 4509 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3a13485-chkkp"] Nov 04 05:11:05.996167 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:11:05.996138 4509 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-gfjqz_c79eff94-6543-4313-b59b-502ca4089e84/global-pull-secret-syncer/0.log" Nov 04 05:11:06.260525 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:11:06.260436 4509 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83294360-4718-4139-8287-8ec8ee6f72ce" path="/var/lib/kubelet/pods/83294360-4718-4139-8287-8ec8ee6f72ce/volumes" Nov 04 05:11:09.665180 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:11:09.665154 4509 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-7tw97_23ed4a39-18b5-4bc6-9c7b-8b60a7043ad2/konnectivity-agent/0.log" Nov 04 05:11:10.738271 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:11:10.738243 4509 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-133-180.us-west-1.compute.internal_6aa9201e961e4ed5a49af4c50453ef92/haproxy/0.log" Nov 04 05:11:14.453279 ip-10-0-133-180 kubenswrapper[4509]: E1104 05:11:14.453238 4509 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dmdhl" containerName="extractor" Nov 04 05:11:14.730463 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:11:14.730433 4509 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-775668dcd7-z2z29_d90081c2-08dc-4d6d-b1eb-876a5e27584a/authorino/0.log" Nov 04 05:11:16.234083 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:11:16.234049 4509 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-zx9qc/perf-node-gather-daemonset-z5cgh"] Nov 04 05:11:16.234564 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:11:16.234393 4509 cpu_manager.go:414] "containerMap: removing container" podUID="37a5cce6-297b-4825-8710-bcb023060bb4" containerName="istio-proxy" Nov 04 05:11:16.234564 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:11:16.234404 4509 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37a5cce6-297b-4825-8710-bcb023060bb4" containerName="istio-proxy" Nov 04 05:11:16.234564 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:11:16.234416 4509 cpu_manager.go:414] "containerMap: removing container" podUID="67ec6f66-3a36-4d01-be83-4ea5d42ba52e" containerName="istio-proxy" Nov 04 05:11:16.234564 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:11:16.234422 4509 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67ec6f66-3a36-4d01-be83-4ea5d42ba52e" containerName="istio-proxy" Nov 04 05:11:16.234564 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:11:16.234430 4509 cpu_manager.go:414] "containerMap: removing container" podUID="2a16804b-60cc-46b5-9879-163bda7bd069" containerName="payload-processing" Nov 04 05:11:16.234564 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:11:16.234435 4509 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a16804b-60cc-46b5-9879-163bda7bd069" containerName="payload-processing" Nov 04 05:11:16.234564 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:11:16.234448 4509 cpu_manager.go:414] "containerMap: removing container" podUID="d751b557-4e07-4657-96ec-59fc0e70fd57" containerName="istio-proxy" Nov 04 05:11:16.234564 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:11:16.234453 4509 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d751b557-4e07-4657-96ec-59fc0e70fd57" containerName="istio-proxy" Nov 04 05:11:16.234564 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:11:16.234467 4509 cpu_manager.go:414] "containerMap: removing container" podUID="ff705a2e-8f20-49e5-aa26-69c7a8d1c31e" containerName="istio-proxy" Nov 04 05:11:16.234564 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:11:16.234473 4509 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff705a2e-8f20-49e5-aa26-69c7a8d1c31e" containerName="istio-proxy" Nov 04 05:11:16.234564 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:11:16.234481 4509 cpu_manager.go:414] "containerMap: removing container" podUID="d751b557-4e07-4657-96ec-59fc0e70fd57" containerName="istio-proxy" Nov 04 05:11:16.234564 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:11:16.234487 4509 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d751b557-4e07-4657-96ec-59fc0e70fd57" containerName="istio-proxy" Nov 04 05:11:16.234564 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:11:16.234492 4509 cpu_manager.go:414] "containerMap: removing container" podUID="80cd56a4-9db7-4770-b1cc-a10754e3de43" containerName="istio-proxy" Nov 04 05:11:16.234564 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:11:16.234497 4509 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80cd56a4-9db7-4770-b1cc-a10754e3de43" containerName="istio-proxy" Nov 04 05:11:16.234564 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:11:16.234545 4509 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a16804b-60cc-46b5-9879-163bda7bd069" containerName="payload-processing" Nov 04 05:11:16.234564 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:11:16.234555 4509 memory_manager.go:365] "RemoveStaleState removing state" podUID="d751b557-4e07-4657-96ec-59fc0e70fd57" containerName="istio-proxy" Nov 04 05:11:16.234564 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:11:16.234562 4509 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff705a2e-8f20-49e5-aa26-69c7a8d1c31e" containerName="istio-proxy" Nov 04 05:11:16.234564 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:11:16.234568 4509 memory_manager.go:365] "RemoveStaleState removing state" podUID="67ec6f66-3a36-4d01-be83-4ea5d42ba52e" containerName="istio-proxy" Nov 04 05:11:16.234564 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:11:16.234574 4509 memory_manager.go:365] "RemoveStaleState removing state" podUID="80cd56a4-9db7-4770-b1cc-a10754e3de43" containerName="istio-proxy" Nov 04 05:11:16.235292 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:11:16.234580 4509 memory_manager.go:365] "RemoveStaleState removing state" podUID="37a5cce6-297b-4825-8710-bcb023060bb4" containerName="istio-proxy" Nov 04 05:11:16.237893 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:11:16.237878 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zx9qc/perf-node-gather-daemonset-z5cgh" Nov 04 05:11:16.293859 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:11:16.293832 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zx9qc/perf-node-gather-daemonset-z5cgh"] Nov 04 05:11:16.342088 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:11:16.342060 4509 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zx9qc\"/\"openshift-service-ca.crt\"" Nov 04 05:11:16.342256 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:11:16.342124 4509 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zx9qc\"/\"kube-root-ca.crt\"" Nov 04 05:11:16.342256 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:11:16.342139 4509 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-zx9qc\"/\"default-dockercfg-qh69v\"" Nov 04 05:11:16.410975 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:11:16.410950 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d1e8bfd2-f609-4941-ad70-6534c4143254-proc\") pod \"perf-node-gather-daemonset-z5cgh\" (UID: \"d1e8bfd2-f609-4941-ad70-6534c4143254\") " pod="openshift-must-gather-zx9qc/perf-node-gather-daemonset-z5cgh" Nov 04 05:11:16.411147 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:11:16.411004 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d1e8bfd2-f609-4941-ad70-6534c4143254-sys\") pod \"perf-node-gather-daemonset-z5cgh\" (UID: \"d1e8bfd2-f609-4941-ad70-6534c4143254\") " pod="openshift-must-gather-zx9qc/perf-node-gather-daemonset-z5cgh" Nov 04 05:11:16.411213 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:11:16.411137 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65q2z\" (UniqueName: \"kubernetes.io/projected/d1e8bfd2-f609-4941-ad70-6534c4143254-kube-api-access-65q2z\") pod \"perf-node-gather-daemonset-z5cgh\" (UID: \"d1e8bfd2-f609-4941-ad70-6534c4143254\") " pod="openshift-must-gather-zx9qc/perf-node-gather-daemonset-z5cgh" Nov 04 05:11:16.411213 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:11:16.411202 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d1e8bfd2-f609-4941-ad70-6534c4143254-lib-modules\") pod \"perf-node-gather-daemonset-z5cgh\" (UID: \"d1e8bfd2-f609-4941-ad70-6534c4143254\") " pod="openshift-must-gather-zx9qc/perf-node-gather-daemonset-z5cgh" Nov 04 05:11:16.411316 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:11:16.411228 4509 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d1e8bfd2-f609-4941-ad70-6534c4143254-podres\") pod \"perf-node-gather-daemonset-z5cgh\" (UID: \"d1e8bfd2-f609-4941-ad70-6534c4143254\") " pod="openshift-must-gather-zx9qc/perf-node-gather-daemonset-z5cgh" Nov 04 05:11:16.511907 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:11:16.511826 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d1e8bfd2-f609-4941-ad70-6534c4143254-proc\") pod \"perf-node-gather-daemonset-z5cgh\" (UID: \"d1e8bfd2-f609-4941-ad70-6534c4143254\") " pod="openshift-must-gather-zx9qc/perf-node-gather-daemonset-z5cgh" Nov 04 05:11:16.511907 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:11:16.511872 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d1e8bfd2-f609-4941-ad70-6534c4143254-sys\") pod \"perf-node-gather-daemonset-z5cgh\" (UID: \"d1e8bfd2-f609-4941-ad70-6534c4143254\") " pod="openshift-must-gather-zx9qc/perf-node-gather-daemonset-z5cgh" Nov 04 05:11:16.511907 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:11:16.511890 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65q2z\" (UniqueName: \"kubernetes.io/projected/d1e8bfd2-f609-4941-ad70-6534c4143254-kube-api-access-65q2z\") pod \"perf-node-gather-daemonset-z5cgh\" (UID: \"d1e8bfd2-f609-4941-ad70-6534c4143254\") " pod="openshift-must-gather-zx9qc/perf-node-gather-daemonset-z5cgh" Nov 04 05:11:16.512129 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:11:16.511918 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d1e8bfd2-f609-4941-ad70-6534c4143254-lib-modules\") pod \"perf-node-gather-daemonset-z5cgh\" (UID: \"d1e8bfd2-f609-4941-ad70-6534c4143254\") " pod="openshift-must-gather-zx9qc/perf-node-gather-daemonset-z5cgh" Nov 04 05:11:16.512129 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:11:16.511943 4509 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d1e8bfd2-f609-4941-ad70-6534c4143254-podres\") pod \"perf-node-gather-daemonset-z5cgh\" (UID: \"d1e8bfd2-f609-4941-ad70-6534c4143254\") " pod="openshift-must-gather-zx9qc/perf-node-gather-daemonset-z5cgh" Nov 04 05:11:16.512129 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:11:16.511965 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d1e8bfd2-f609-4941-ad70-6534c4143254-sys\") pod \"perf-node-gather-daemonset-z5cgh\" (UID: \"d1e8bfd2-f609-4941-ad70-6534c4143254\") " pod="openshift-must-gather-zx9qc/perf-node-gather-daemonset-z5cgh" Nov 04 05:11:16.512129 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:11:16.511966 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d1e8bfd2-f609-4941-ad70-6534c4143254-proc\") pod \"perf-node-gather-daemonset-z5cgh\" (UID: \"d1e8bfd2-f609-4941-ad70-6534c4143254\") " pod="openshift-must-gather-zx9qc/perf-node-gather-daemonset-z5cgh" Nov 04 05:11:16.512129 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:11:16.512006 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d1e8bfd2-f609-4941-ad70-6534c4143254-lib-modules\") pod \"perf-node-gather-daemonset-z5cgh\" (UID: \"d1e8bfd2-f609-4941-ad70-6534c4143254\") " pod="openshift-must-gather-zx9qc/perf-node-gather-daemonset-z5cgh" Nov 04 05:11:16.512129 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:11:16.512022 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d1e8bfd2-f609-4941-ad70-6534c4143254-podres\") pod \"perf-node-gather-daemonset-z5cgh\" (UID: \"d1e8bfd2-f609-4941-ad70-6534c4143254\") " pod="openshift-must-gather-zx9qc/perf-node-gather-daemonset-z5cgh" Nov 04 05:11:16.569763 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:11:16.569736 4509 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65q2z\" (UniqueName: \"kubernetes.io/projected/d1e8bfd2-f609-4941-ad70-6534c4143254-kube-api-access-65q2z\") pod \"perf-node-gather-daemonset-z5cgh\" (UID: \"d1e8bfd2-f609-4941-ad70-6534c4143254\") " pod="openshift-must-gather-zx9qc/perf-node-gather-daemonset-z5cgh" Nov 04 05:11:16.640916 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:11:16.640886 4509 scope.go:122] "RemoveContainer" containerID="abab568c99208ecfd7a1a88e5a76eef3f18d0d900c464ff8bedf216983d500b0" Nov 04 05:11:16.649091 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:11:16.649072 4509 scope.go:122] "RemoveContainer" containerID="88bada5ad8c02cea49b39b5215d84e478081a64b40529295d3305ee555a96db3" Nov 04 05:11:16.656367 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:11:16.656348 4509 scope.go:122] "RemoveContainer" containerID="626caa894d3c7a35491a635dd40d8fc60feb4d4f962c2b93affba03541ace5c1" Nov 04 05:11:16.663391 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:11:16.663375 4509 scope.go:122] "RemoveContainer" containerID="754c5488adafe136991f0b57e1b6ad9d7e7c05c6c83ee49ca5a6fbc1ada5b9c1" Nov 04 05:11:16.670246 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:11:16.670229 4509 scope.go:122] "RemoveContainer" containerID="3c6fdd633d94d7e06fef09d9acbf6cb236c9678a87c1811e3ce1d2ad34fbfd1c" Nov 04 05:11:16.677154 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:11:16.677140 4509 scope.go:122] "RemoveContainer" containerID="306b306d9b814119f5696f0ee54a385aa7988fd833d4b96f45099c1262173f48" Nov 04 05:11:16.850010 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:11:16.849964 4509 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zx9qc/perf-node-gather-daemonset-z5cgh" Nov 04 05:11:17.212645 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:11:17.212617 4509 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-42dnm_e0ffdeeb-4a8f-437b-80ab-c4c09c6e14bc/limitador/0.log" Nov 04 05:11:17.221310 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:11:17.221104 4509 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zx9qc/perf-node-gather-daemonset-z5cgh"] Nov 04 05:11:17.226853 ip-10-0-133-180 kubenswrapper[4509]: W1104 05:11:17.226825 4509 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podd1e8bfd2_f609_4941_ad70_6534c4143254.slice/crio-db128994bbdd4c7dd5dc810dbbce62e10451cb2bbf869472288104341d611f65 WatchSource:0}: Error finding container db128994bbdd4c7dd5dc810dbbce62e10451cb2bbf869472288104341d611f65: Status 404 returned error can't find the container with id db128994bbdd4c7dd5dc810dbbce62e10451cb2bbf869472288104341d611f65 Nov 04 05:11:17.976015 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:11:17.975980 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zx9qc/perf-node-gather-daemonset-z5cgh" event={"ID":"d1e8bfd2-f609-4941-ad70-6534c4143254","Type":"ContainerStarted","Data":"d34d0759f4e7b5258e94294f173c1cac5755cf7fcd289d127cf2dc696173c1b0"} Nov 04 05:11:17.976425 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:11:17.976030 4509 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zx9qc/perf-node-gather-daemonset-z5cgh" event={"ID":"d1e8bfd2-f609-4941-ad70-6534c4143254","Type":"ContainerStarted","Data":"db128994bbdd4c7dd5dc810dbbce62e10451cb2bbf869472288104341d611f65"} Nov 04 05:11:17.976425 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:11:17.976059 4509 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-zx9qc/perf-node-gather-daemonset-z5cgh" Nov 04 05:11:18.090023 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:11:18.089970 4509 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-zx9qc/perf-node-gather-daemonset-z5cgh" podStartSLOduration=2.089954858 podStartE2EDuration="2.089954858s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 05:11:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 05:11:18.088902075 +0000 UTC m=+1441.966625607" watchObservedRunningTime="2026-11-04 05:11:18.089954858 +0000 UTC m=+1441.967678396" Nov 04 05:11:23.991657 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:11:23.991630 4509 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-zx9qc/perf-node-gather-daemonset-z5cgh" Nov 04 05:11:25.436544 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:11:25.436513 4509 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-957r8_841efaa4-fa65-4d0e-887f-7390c27db3f5/monitoring-plugin/0.log" Nov 04 05:11:27.122534 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:11:27.122507 4509 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-l64n5_ca21ea05-78e7-4261-98a4-7025888ef362/node-exporter/0.log" Nov 04 05:11:27.538838 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:11:27.538798 4509 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-l64n5_ca21ea05-78e7-4261-98a4-7025888ef362/kube-rbac-proxy/0.log" Nov 04 05:11:27.954304 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:11:27.954234 4509 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-l64n5_ca21ea05-78e7-4261-98a4-7025888ef362/init-textfile/0.log" Nov 04 05:11:29.665817 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:11:29.665789 4509 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-cxhln_0c773e84-74ce-494f-8322-8d6204654e7f/kube-rbac-proxy-main/0.log" Nov 04 05:11:30.082060 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:11:30.082031 4509 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-cxhln_0c773e84-74ce-494f-8322-8d6204654e7f/kube-rbac-proxy-self/0.log" Nov 04 05:11:30.498029 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:11:30.497989 4509 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-cxhln_0c773e84-74ce-494f-8322-8d6204654e7f/openshift-state-metrics/0.log" Nov 04 05:11:34.741253 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:11:34.741224 4509 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-9k5vv_b4d26ce7-b503-4c3d-8e1e-02ff73858fbe/prometheus-operator-admission-webhook/0.log" Nov 04 05:11:35.164695 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:11:35.164600 4509 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-c656644f9-tk9wq_67b9e6c7-278b-47dc-bad0-72d05c2c6427/telemeter-client/0.log" Nov 04 05:11:35.582650 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:11:35.582617 4509 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-c656644f9-tk9wq_67b9e6c7-278b-47dc-bad0-72d05c2c6427/reload/0.log" Nov 04 05:11:36.000632 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:11:36.000597 4509 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-c656644f9-tk9wq_67b9e6c7-278b-47dc-bad0-72d05c2c6427/kube-rbac-proxy/0.log" Nov 04 05:11:39.741449 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:11:39.741411 4509 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-f447m_a4d34feb-7fd9-4b77-a628-eb9d84adb93c/discovery/0.log" Nov 04 05:11:40.067515 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:11:40.067429 4509 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-546d78d5c-vbwrt_542ce577-5ec5-400e-a82b-d5d1f3fe8c97/kube-auth-proxy/0.log" Nov 04 05:11:40.285260 ip-10-0-133-180 kubenswrapper[4509]: I1104 05:11:40.285229 4509 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-4sj9h_e057215d-2b26-47c6-8c61-3d1413ef6aec/istio-proxy/0.log"