Nov 02 06:13:29.689134 ip-10-0-130-93 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Nov 02 06:13:29.689350 ip-10-0-130-93 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Nov 02 06:13:29.689442 ip-10-0-130-93 systemd[1]: kubelet.service: Failed with result 'resources'. Nov 02 06:13:29.689991 ip-10-0-130-93 systemd[1]: Failed to start Kubernetes Kubelet. Nov 02 06:13:39.692731 ip-10-0-130-93 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Nov 02 06:13:39.692741 ip-10-0-130-93 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 0257ffa4a85a458293f8e3183ee282db -- Nov 02 06:15:38.006217 ip-10-0-130-93 systemd[1]: Starting Kubernetes Kubelet... Nov 02 06:15:38.403658 ip-10-0-130-93 kubenswrapper[2579]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 06:15:38.403658 ip-10-0-130-93 kubenswrapper[2579]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 02 06:15:38.403658 ip-10-0-130-93 kubenswrapper[2579]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 06:15:38.403658 ip-10-0-130-93 kubenswrapper[2579]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 06:15:38.409221 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.409129 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 06:15:38.409221 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.409218 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 06:15:38.409221 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.409222 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 06:15:38.409221 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.409225 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 06:15:38.409221 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.409227 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 06:15:38.409430 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.409230 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 06:15:38.409430 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.409234 2579 feature_gate.go:386] unrecognized feature gate: Example Nov 02 06:15:38.409430 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.409236 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 06:15:38.409430 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.409239 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 06:15:38.409430 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.409242 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 06:15:38.409430 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.409244 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 06:15:38.409430 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.409247 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 06:15:38.409430 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.409250 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 06:15:38.409430 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.409252 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 06:15:38.409430 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.409255 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 06:15:38.409430 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.409259 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 06:15:38.409430 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.409265 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 06:15:38.409430 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.409268 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 06:15:38.409430 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.409271 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 06:15:38.409430 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.409274 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 06:15:38.409430 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.409277 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 06:15:38.409430 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.409279 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 06:15:38.409430 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.409282 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 06:15:38.409430 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.409285 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 06:15:38.409928 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.409289 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 06:15:38.409928 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.409293 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 06:15:38.409928 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.409295 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 06:15:38.409928 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.409298 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 06:15:38.409928 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.409301 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 06:15:38.409928 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.409304 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 06:15:38.409928 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.409308 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 06:15:38.409928 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.409312 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 06:15:38.409928 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.409316 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 06:15:38.409928 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.409320 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 06:15:38.409928 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.409322 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 06:15:38.409928 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.409326 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 06:15:38.409928 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.409329 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 06:15:38.409928 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.409331 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 06:15:38.409928 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.409335 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 06:15:38.409928 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.409338 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 06:15:38.409928 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.409341 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 06:15:38.409928 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.409344 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 06:15:38.409928 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.409346 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 06:15:38.410373 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.409349 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 06:15:38.410373 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.409352 2579 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 06:15:38.410373 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.409354 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 06:15:38.410373 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.409357 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 06:15:38.410373 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.409360 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 06:15:38.410373 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.409362 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 06:15:38.410373 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.409364 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 06:15:38.410373 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.409367 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 06:15:38.410373 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.409370 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 06:15:38.410373 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.409373 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 06:15:38.410373 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.409376 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 06:15:38.410373 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.409378 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 06:15:38.410373 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.409381 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 06:15:38.410373 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.409383 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 06:15:38.410373 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.409386 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 06:15:38.410373 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.409388 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 06:15:38.410373 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.409391 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 06:15:38.410373 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.409394 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 06:15:38.410373 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.409396 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 06:15:38.410373 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.409398 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 06:15:38.410884 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.409401 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 06:15:38.410884 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.409403 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 06:15:38.410884 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.409406 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 06:15:38.410884 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.409408 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 06:15:38.410884 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.409411 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 06:15:38.410884 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.409414 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 06:15:38.410884 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.409416 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 06:15:38.410884 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.409419 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 06:15:38.410884 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.409422 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 06:15:38.410884 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.409424 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 06:15:38.410884 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.409427 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 06:15:38.410884 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.409430 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 06:15:38.410884 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.409433 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 06:15:38.410884 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.409435 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 06:15:38.410884 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.409438 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 06:15:38.410884 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.409440 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 06:15:38.410884 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.409443 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 06:15:38.410884 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.409446 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 06:15:38.410884 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.409449 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 06:15:38.411363 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.409451 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 06:15:38.411363 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.409455 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 06:15:38.411363 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.409458 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 06:15:38.411363 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.409461 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 06:15:38.411363 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.409463 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 06:15:38.411363 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.409466 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 06:15:38.411363 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.409469 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 06:15:38.411363 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.409471 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 06:15:38.411363 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.409474 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 06:15:38.411363 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.409476 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 06:15:38.411363 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.409479 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 06:15:38.411363 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.409481 2579 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 06:15:38.411363 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.409484 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 06:15:38.411363 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.409486 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 06:15:38.411363 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.409489 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 06:15:38.411363 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.409491 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 06:15:38.411363 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.409494 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 06:15:38.411363 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.409496 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 06:15:38.411363 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.409499 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 06:15:38.411363 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.409502 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 06:15:38.411845 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.409505 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 06:15:38.411845 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.409508 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 06:15:38.411845 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.409511 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 06:15:38.411845 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.409515 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 06:15:38.411845 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.410317 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 06:15:38.411845 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.410325 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 06:15:38.411845 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.410328 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 06:15:38.411845 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.410330 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 06:15:38.411845 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.410333 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 06:15:38.411845 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.410336 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 06:15:38.411845 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.410339 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 06:15:38.411845 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.410342 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 06:15:38.411845 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.410345 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 06:15:38.411845 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.410348 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 06:15:38.411845 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.410352 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 06:15:38.411845 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.410355 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 06:15:38.411845 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.410358 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 06:15:38.411845 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.410360 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 06:15:38.411845 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.410363 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 06:15:38.412321 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.410366 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 06:15:38.412321 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.410368 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 06:15:38.412321 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.410371 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 06:15:38.412321 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.410373 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 06:15:38.412321 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.410376 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 06:15:38.412321 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.410379 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 06:15:38.412321 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.410381 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 06:15:38.412321 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.410384 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 06:15:38.412321 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.410387 2579 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 06:15:38.412321 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.410389 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 06:15:38.412321 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.410392 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 06:15:38.412321 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.410394 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 06:15:38.412321 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.410397 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 06:15:38.412321 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.410401 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 06:15:38.412321 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.410403 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 06:15:38.412321 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.410407 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 06:15:38.412321 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.410411 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 06:15:38.412321 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.410415 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 06:15:38.412321 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.410418 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 06:15:38.412773 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.410421 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 06:15:38.412773 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.410424 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 06:15:38.412773 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.410426 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 06:15:38.412773 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.410429 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 06:15:38.412773 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.410431 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 06:15:38.412773 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.410434 2579 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 06:15:38.412773 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.410438 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 06:15:38.412773 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.410441 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 06:15:38.412773 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.410444 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 06:15:38.412773 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.410448 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 06:15:38.412773 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.410451 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 06:15:38.412773 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.410453 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 06:15:38.412773 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.410456 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 06:15:38.412773 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.410459 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 06:15:38.412773 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.410462 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 06:15:38.412773 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.410464 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 06:15:38.412773 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.410466 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 06:15:38.412773 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.410469 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 06:15:38.412773 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.410471 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 06:15:38.413237 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.410474 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 06:15:38.413237 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.410476 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 06:15:38.413237 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.410479 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 06:15:38.413237 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.410482 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 06:15:38.413237 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.410484 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 06:15:38.413237 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.410487 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 06:15:38.413237 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.410489 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 06:15:38.413237 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.410492 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 06:15:38.413237 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.410495 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 06:15:38.413237 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.410498 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 06:15:38.413237 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.410500 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 06:15:38.413237 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.410503 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 06:15:38.413237 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.410505 2579 feature_gate.go:386] unrecognized feature gate: Example Nov 02 06:15:38.413237 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.410508 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 06:15:38.413237 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.410511 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 06:15:38.413237 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.410513 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 06:15:38.413237 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.410516 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 06:15:38.413237 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.410518 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 06:15:38.413237 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.410521 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 06:15:38.413237 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.410523 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 06:15:38.413748 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.410526 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 06:15:38.413748 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.410528 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 06:15:38.413748 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.410531 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 06:15:38.413748 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.410534 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 06:15:38.413748 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.410537 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 06:15:38.413748 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.410539 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 06:15:38.413748 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.410542 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 06:15:38.413748 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.410544 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 06:15:38.413748 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.410547 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 06:15:38.413748 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.410550 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 06:15:38.413748 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.410552 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 06:15:38.413748 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.410555 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 06:15:38.413748 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.410557 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 06:15:38.413748 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.410560 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 06:15:38.413748 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.410563 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 06:15:38.413748 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.410565 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 06:15:38.413748 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.410568 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 06:15:38.413748 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.410570 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 06:15:38.413748 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.410573 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 06:15:38.414215 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.410576 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 06:15:38.414215 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.410579 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 06:15:38.414215 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.410583 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 06:15:38.414215 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.410585 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 06:15:38.414215 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.410588 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 06:15:38.414215 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.410590 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 06:15:38.414215 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.410593 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 06:15:38.414215 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.410596 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 06:15:38.414215 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.410598 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 06:15:38.414215 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.410600 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 06:15:38.414215 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.410603 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 06:15:38.414215 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.410606 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 06:15:38.414215 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.410608 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 06:15:38.414215 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.410611 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 06:15:38.414215 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.411114 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 06:15:38.414215 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.411120 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 06:15:38.414215 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.411123 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 06:15:38.414215 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.411126 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 06:15:38.414215 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.411129 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 06:15:38.414215 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.411132 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 06:15:38.414687 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.411135 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 06:15:38.414687 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.411137 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 06:15:38.414687 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.411141 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 06:15:38.414687 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.411144 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 06:15:38.414687 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.411146 2579 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 06:15:38.414687 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.411149 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 06:15:38.414687 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.411151 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 06:15:38.414687 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.411154 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 06:15:38.414687 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.411157 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 06:15:38.414687 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.411160 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 06:15:38.414687 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.411162 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 06:15:38.414687 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.411165 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 06:15:38.414687 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.411168 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 06:15:38.414687 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.411171 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 06:15:38.414687 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.411173 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 06:15:38.414687 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.411176 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 06:15:38.414687 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.411179 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 06:15:38.414687 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.411181 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 06:15:38.414687 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.411184 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 06:15:38.415147 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.411187 2579 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 06:15:38.415147 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.411189 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 06:15:38.415147 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.411192 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 06:15:38.415147 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.411194 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 06:15:38.415147 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.411197 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 06:15:38.415147 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.411199 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 06:15:38.415147 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.411202 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 06:15:38.415147 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.411204 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 06:15:38.415147 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.411207 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 06:15:38.415147 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.411209 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 06:15:38.415147 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.411212 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 06:15:38.415147 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.411215 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 06:15:38.415147 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.411217 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 06:15:38.415147 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.411220 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 06:15:38.415147 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.411222 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 06:15:38.415147 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.411225 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 06:15:38.415147 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.411228 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 06:15:38.415147 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.411230 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 06:15:38.415147 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.411233 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 06:15:38.415595 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.411236 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 06:15:38.415595 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.411239 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 06:15:38.415595 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.411242 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 06:15:38.415595 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.411244 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 06:15:38.415595 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.411246 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 06:15:38.415595 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.411249 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 06:15:38.415595 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.411252 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 06:15:38.415595 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.411254 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 06:15:38.415595 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.411256 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 06:15:38.415595 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.411259 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 06:15:38.415595 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.411261 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 06:15:38.415595 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.411264 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 06:15:38.415595 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.411266 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 06:15:38.415595 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.411269 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 06:15:38.415595 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.411271 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 06:15:38.415595 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.411274 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 06:15:38.415595 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.411276 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 06:15:38.415595 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.411279 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 06:15:38.415595 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.411281 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 06:15:38.415595 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.411284 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 06:15:38.415595 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.411286 2579 feature_gate.go:386] unrecognized feature gate: Example Nov 02 06:15:38.416103 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.411289 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 06:15:38.416103 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.411291 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 06:15:38.416103 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.411293 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 06:15:38.416103 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.411297 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 06:15:38.416103 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.411299 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 06:15:38.416103 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.411302 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 06:15:38.416103 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.411304 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 06:15:38.416103 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.411307 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 06:15:38.416103 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.411309 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 06:15:38.416103 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.411312 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 06:15:38.416103 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.411314 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 06:15:38.416103 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.411317 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 06:15:38.416103 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.411320 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 06:15:38.416103 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.411322 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 06:15:38.416103 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.411325 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 06:15:38.416103 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.411327 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 06:15:38.416103 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.411330 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 06:15:38.416103 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.411335 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 06:15:38.416103 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.411337 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 06:15:38.416103 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.411348 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 06:15:38.416580 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.411351 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 06:15:38.416580 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.411354 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 06:15:38.416580 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.411357 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 06:15:38.416580 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.411359 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 06:15:38.416580 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.411362 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 06:15:38.416580 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.411365 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 06:15:38.416580 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.411367 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 06:15:38.416580 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.411372 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 06:15:38.416580 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.411377 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 06:15:38.416580 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.411380 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 06:15:38.416580 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.411383 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 06:15:38.416580 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.411386 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 06:15:38.416580 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.411388 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 06:15:38.416580 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.411391 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 06:15:38.416580 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.411393 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 06:15:38.416580 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.411395 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 06:15:38.416580 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.411399 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 06:15:38.416580 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.411405 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 06:15:38.416580 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.411408 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 06:15:38.417044 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.411411 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 06:15:38.417044 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.411414 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 06:15:38.417044 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.412550 2579 flags.go:64] FLAG: --address="0.0.0.0" Nov 02 06:15:38.417044 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.412560 2579 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 02 06:15:38.417044 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.412567 2579 flags.go:64] FLAG: --anonymous-auth="true" Nov 02 06:15:38.417044 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.412572 2579 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 02 06:15:38.417044 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.412576 2579 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 02 06:15:38.417044 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.412581 2579 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 02 06:15:38.417044 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.412585 2579 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 02 06:15:38.417044 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.412590 2579 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 02 06:15:38.417044 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.412593 2579 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 02 06:15:38.417044 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.412597 2579 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 02 06:15:38.417044 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.412600 2579 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 02 06:15:38.417044 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.412604 2579 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 02 06:15:38.417044 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.412607 2579 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 02 06:15:38.417044 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.412610 2579 flags.go:64] FLAG: --cgroup-root="" Nov 02 06:15:38.417044 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.412613 2579 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 02 06:15:38.417044 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.412616 2579 flags.go:64] FLAG: --client-ca-file="" Nov 02 06:15:38.417044 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.412619 2579 flags.go:64] FLAG: --cloud-provider="external" Nov 02 06:15:38.417044 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.412622 2579 flags.go:64] FLAG: --cluster-dns="[]" Nov 02 06:15:38.417044 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.412626 2579 flags.go:64] FLAG: --cluster-domain="" Nov 02 06:15:38.417044 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.412629 2579 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 02 06:15:38.417044 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.412632 2579 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 02 06:15:38.417663 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.412635 2579 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 02 06:15:38.417663 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.412638 2579 flags.go:64] FLAG: --container-log-max-files="5" Nov 02 06:15:38.417663 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.412642 2579 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 02 06:15:38.417663 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.412645 2579 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 02 06:15:38.417663 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.412648 2579 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 02 06:15:38.417663 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.412651 2579 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 02 06:15:38.417663 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.412654 2579 flags.go:64] FLAG: --contention-profiling="false" Nov 02 06:15:38.417663 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.412658 2579 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 02 06:15:38.417663 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.412661 2579 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 02 06:15:38.417663 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.412664 2579 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 02 06:15:38.417663 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.412667 2579 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 02 06:15:38.417663 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.412672 2579 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 02 06:15:38.417663 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.412675 2579 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 02 06:15:38.417663 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.412678 2579 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 02 06:15:38.417663 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.412681 2579 flags.go:64] FLAG: --enable-load-reader="false" Nov 02 06:15:38.417663 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.412684 2579 flags.go:64] FLAG: --enable-server="true" Nov 02 06:15:38.417663 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.412687 2579 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 02 06:15:38.417663 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.412692 2579 flags.go:64] FLAG: --event-burst="100" Nov 02 06:15:38.417663 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.412695 2579 flags.go:64] FLAG: --event-qps="50" Nov 02 06:15:38.417663 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.412698 2579 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 02 06:15:38.417663 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.412701 2579 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 02 06:15:38.417663 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.412704 2579 flags.go:64] FLAG: --eviction-hard="" Nov 02 06:15:38.417663 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.412709 2579 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 02 06:15:38.417663 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.412712 2579 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 02 06:15:38.418292 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.412715 2579 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 02 06:15:38.418292 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.412719 2579 flags.go:64] FLAG: --eviction-soft="" Nov 02 06:15:38.418292 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.412722 2579 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 02 06:15:38.418292 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.412725 2579 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 02 06:15:38.418292 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.412728 2579 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 02 06:15:38.418292 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.412731 2579 flags.go:64] FLAG: --experimental-mounter-path="" Nov 02 06:15:38.418292 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.412734 2579 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 02 06:15:38.418292 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.412737 2579 flags.go:64] FLAG: --fail-swap-on="true" Nov 02 06:15:38.418292 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.412740 2579 flags.go:64] FLAG: --feature-gates="" Nov 02 06:15:38.418292 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.412744 2579 flags.go:64] FLAG: --file-check-frequency="20s" Nov 02 06:15:38.418292 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.412748 2579 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 02 06:15:38.418292 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.412751 2579 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 02 06:15:38.418292 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.412754 2579 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 02 06:15:38.418292 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.412757 2579 flags.go:64] FLAG: --healthz-port="10248" Nov 02 06:15:38.418292 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.412760 2579 flags.go:64] FLAG: --help="false" Nov 02 06:15:38.418292 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.412764 2579 flags.go:64] FLAG: --hostname-override="ip-10-0-130-93.us-west-2.compute.internal" Nov 02 06:15:38.418292 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.412767 2579 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 02 06:15:38.418292 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.412774 2579 flags.go:64] FLAG: --http-check-frequency="20s" Nov 02 06:15:38.418292 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.412778 2579 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 02 06:15:38.418292 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.412781 2579 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 02 06:15:38.418292 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.412785 2579 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 02 06:15:38.418292 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.412788 2579 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 02 06:15:38.418292 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.412790 2579 flags.go:64] FLAG: --image-service-endpoint="" Nov 02 06:15:38.418830 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.412793 2579 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 02 06:15:38.418830 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.412797 2579 flags.go:64] FLAG: --kube-api-burst="100" Nov 02 06:15:38.418830 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.412799 2579 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 02 06:15:38.418830 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.412802 2579 flags.go:64] FLAG: --kube-api-qps="50" Nov 02 06:15:38.418830 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.412805 2579 flags.go:64] FLAG: --kube-reserved="" Nov 02 06:15:38.418830 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.412808 2579 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 02 06:15:38.418830 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.412811 2579 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 02 06:15:38.418830 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.412815 2579 flags.go:64] FLAG: --kubelet-cgroups="" Nov 02 06:15:38.418830 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.412817 2579 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 02 06:15:38.418830 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.412820 2579 flags.go:64] FLAG: --lock-file="" Nov 02 06:15:38.418830 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.412823 2579 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 02 06:15:38.418830 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.412826 2579 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 02 06:15:38.418830 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.412829 2579 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 02 06:15:38.418830 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.412835 2579 flags.go:64] FLAG: --log-json-split-stream="false" Nov 02 06:15:38.418830 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.412838 2579 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 02 06:15:38.418830 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.412841 2579 flags.go:64] FLAG: --log-text-split-stream="false" Nov 02 06:15:38.418830 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.412845 2579 flags.go:64] FLAG: --logging-format="text" Nov 02 06:15:38.418830 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.412848 2579 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 02 06:15:38.418830 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.412852 2579 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 02 06:15:38.418830 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.412854 2579 flags.go:64] FLAG: --manifest-url="" Nov 02 06:15:38.418830 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.412857 2579 flags.go:64] FLAG: --manifest-url-header="" Nov 02 06:15:38.418830 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.412862 2579 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 02 06:15:38.418830 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.412865 2579 flags.go:64] FLAG: --max-open-files="1000000" Nov 02 06:15:38.418830 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.412869 2579 flags.go:64] FLAG: --max-pods="110" Nov 02 06:15:38.418830 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.412872 2579 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 02 06:15:38.419457 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.412875 2579 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 02 06:15:38.419457 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.412883 2579 flags.go:64] FLAG: --memory-manager-policy="None" Nov 02 06:15:38.419457 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.412886 2579 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 02 06:15:38.419457 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.412889 2579 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 02 06:15:38.419457 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.412892 2579 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 02 06:15:38.419457 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.412895 2579 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 02 06:15:38.419457 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.412914 2579 flags.go:64] FLAG: --node-status-max-images="50" Nov 02 06:15:38.419457 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.412918 2579 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 02 06:15:38.419457 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.412921 2579 flags.go:64] FLAG: --oom-score-adj="-999" Nov 02 06:15:38.419457 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.412924 2579 flags.go:64] FLAG: --pod-cidr="" Nov 02 06:15:38.419457 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.412927 2579 flags.go:64] FLAG: --pod-manifest-path="" Nov 02 06:15:38.419457 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.412930 2579 flags.go:64] FLAG: --pod-max-pids="-1" Nov 02 06:15:38.419457 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.412933 2579 flags.go:64] FLAG: --pods-per-core="0" Nov 02 06:15:38.419457 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.412936 2579 flags.go:64] FLAG: --port="10250" Nov 02 06:15:38.419457 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.412939 2579 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 02 06:15:38.419457 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.412942 2579 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-01095da248285f49f" Nov 02 06:15:38.419457 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.412945 2579 flags.go:64] FLAG: --qos-reserved="" Nov 02 06:15:38.419457 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.412949 2579 flags.go:64] FLAG: --read-only-port="10255" Nov 02 06:15:38.419457 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.412952 2579 flags.go:64] FLAG: --register-node="true" Nov 02 06:15:38.419457 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.412955 2579 flags.go:64] FLAG: --register-with-taints="" Nov 02 06:15:38.419457 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.412959 2579 flags.go:64] FLAG: --registry-burst="10" Nov 02 06:15:38.419457 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.412962 2579 flags.go:64] FLAG: --registry-qps="5" Nov 02 06:15:38.419457 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.412964 2579 flags.go:64] FLAG: --reserved-cpus="" Nov 02 06:15:38.419457 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.412968 2579 flags.go:64] FLAG: --reserved-memory="" Nov 02 06:15:38.419457 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.412972 2579 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 02 06:15:38.419457 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.412975 2579 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 02 06:15:38.420130 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.412978 2579 flags.go:64] FLAG: --rotate-certificates="false" Nov 02 06:15:38.420130 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.412981 2579 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 02 06:15:38.420130 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.412984 2579 flags.go:64] FLAG: --runonce="false" Nov 02 06:15:38.420130 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.412987 2579 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 02 06:15:38.420130 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.412990 2579 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 02 06:15:38.420130 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.412993 2579 flags.go:64] FLAG: --seccomp-default="false" Nov 02 06:15:38.420130 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.412996 2579 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 02 06:15:38.420130 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.413001 2579 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 02 06:15:38.420130 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.413005 2579 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 02 06:15:38.420130 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.413008 2579 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 02 06:15:38.420130 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.413011 2579 flags.go:64] FLAG: --storage-driver-password="root" Nov 02 06:15:38.420130 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.413014 2579 flags.go:64] FLAG: --storage-driver-secure="false" Nov 02 06:15:38.420130 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.413017 2579 flags.go:64] FLAG: --storage-driver-table="stats" Nov 02 06:15:38.420130 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.413020 2579 flags.go:64] FLAG: --storage-driver-user="root" Nov 02 06:15:38.420130 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.413023 2579 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 02 06:15:38.420130 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.413026 2579 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 02 06:15:38.420130 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.413029 2579 flags.go:64] FLAG: --system-cgroups="" Nov 02 06:15:38.420130 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.413033 2579 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 02 06:15:38.420130 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.413039 2579 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 02 06:15:38.420130 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.413042 2579 flags.go:64] FLAG: --tls-cert-file="" Nov 02 06:15:38.420130 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.413045 2579 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 02 06:15:38.420130 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.413048 2579 flags.go:64] FLAG: --tls-min-version="" Nov 02 06:15:38.420130 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.413051 2579 flags.go:64] FLAG: --tls-private-key-file="" Nov 02 06:15:38.420130 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.413054 2579 flags.go:64] FLAG: --topology-manager-policy="none" Nov 02 06:15:38.420130 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.413057 2579 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 02 06:15:38.420712 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.413060 2579 flags.go:64] FLAG: --topology-manager-scope="container" Nov 02 06:15:38.420712 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.413063 2579 flags.go:64] FLAG: --v="2" Nov 02 06:15:38.420712 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.413068 2579 flags.go:64] FLAG: --version="false" Nov 02 06:15:38.420712 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.413072 2579 flags.go:64] FLAG: --vmodule="" Nov 02 06:15:38.420712 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.413076 2579 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 02 06:15:38.420712 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.413080 2579 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 02 06:15:38.420712 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413238 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 06:15:38.420712 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413243 2579 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 06:15:38.420712 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413247 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 06:15:38.420712 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413250 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 06:15:38.420712 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413253 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 06:15:38.420712 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413256 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 06:15:38.420712 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413259 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 06:15:38.420712 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413262 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 06:15:38.420712 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413265 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 06:15:38.420712 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413267 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 06:15:38.420712 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413271 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 06:15:38.420712 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413273 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 06:15:38.420712 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413276 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 06:15:38.420712 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413279 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 06:15:38.420712 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413281 2579 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 06:15:38.421224 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413285 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 06:15:38.421224 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413289 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 06:15:38.421224 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413292 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 06:15:38.421224 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413296 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 06:15:38.421224 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413299 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 06:15:38.421224 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413301 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 06:15:38.421224 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413304 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 06:15:38.421224 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413307 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 06:15:38.421224 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413309 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 06:15:38.421224 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413312 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 06:15:38.421224 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413314 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 06:15:38.421224 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413317 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 06:15:38.421224 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413320 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 06:15:38.421224 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413323 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 06:15:38.421224 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413325 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 06:15:38.421224 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413328 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 06:15:38.421224 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413331 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 06:15:38.421224 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413333 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 06:15:38.421224 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413336 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 06:15:38.421684 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413338 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 06:15:38.421684 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413341 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 06:15:38.421684 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413343 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 06:15:38.421684 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413346 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 06:15:38.421684 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413348 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 06:15:38.421684 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413351 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 06:15:38.421684 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413354 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 06:15:38.421684 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413357 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 06:15:38.421684 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413359 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 06:15:38.421684 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413363 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 06:15:38.421684 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413366 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 06:15:38.421684 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413369 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 06:15:38.421684 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413371 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 06:15:38.421684 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413374 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 06:15:38.421684 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413377 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 06:15:38.421684 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413379 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 06:15:38.421684 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413383 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 06:15:38.421684 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413386 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 06:15:38.421684 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413388 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 06:15:38.422142 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413391 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 06:15:38.422142 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413395 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 06:15:38.422142 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413399 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 06:15:38.422142 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413402 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 06:15:38.422142 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413405 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 06:15:38.422142 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413407 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 06:15:38.422142 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413410 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 06:15:38.422142 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413412 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 06:15:38.422142 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413415 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 06:15:38.422142 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413418 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 06:15:38.422142 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413420 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 06:15:38.422142 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413423 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 06:15:38.422142 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413425 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 06:15:38.422142 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413428 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 06:15:38.422142 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413430 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 06:15:38.422142 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413433 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 06:15:38.422142 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413435 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 06:15:38.422142 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413438 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 06:15:38.422142 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413440 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 06:15:38.422626 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413443 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 06:15:38.422626 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413445 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 06:15:38.422626 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413448 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 06:15:38.422626 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413450 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 06:15:38.422626 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413453 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 06:15:38.422626 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413456 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 06:15:38.422626 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413458 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 06:15:38.422626 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413461 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 06:15:38.422626 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413463 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 06:15:38.422626 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413466 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 06:15:38.422626 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413470 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 06:15:38.422626 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413472 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 06:15:38.422626 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413475 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 06:15:38.422626 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413477 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 06:15:38.422626 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413480 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 06:15:38.422626 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413482 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 06:15:38.422626 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413485 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 06:15:38.422626 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413487 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 06:15:38.422626 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413490 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 06:15:38.422626 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413492 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 06:15:38.423123 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413495 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 06:15:38.423123 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413497 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 06:15:38.423123 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413500 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 06:15:38.423123 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413502 2579 feature_gate.go:386] unrecognized feature gate: Example Nov 02 06:15:38.423123 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413505 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 06:15:38.423123 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413507 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 06:15:38.423123 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413510 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 06:15:38.423123 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413512 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 06:15:38.423123 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413515 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 06:15:38.423123 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413518 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 06:15:38.423123 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413520 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 06:15:38.423123 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413523 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 06:15:38.423123 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413525 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 06:15:38.423123 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413528 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 06:15:38.423123 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.413554 2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 06:15:38.423123 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413589 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 06:15:38.423515 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413593 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 06:15:38.423515 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413597 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 06:15:38.423515 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413600 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 06:15:38.423515 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413603 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 06:15:38.423515 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413605 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 06:15:38.423515 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413608 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 06:15:38.423515 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413612 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 06:15:38.423515 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413614 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 06:15:38.423515 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413617 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 06:15:38.423515 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413619 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 06:15:38.423515 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413622 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 06:15:38.423515 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413625 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 06:15:38.423515 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413627 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 06:15:38.423515 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413630 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 06:15:38.423515 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413632 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 06:15:38.423515 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413635 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 06:15:38.423515 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413637 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 06:15:38.423515 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413640 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 06:15:38.423515 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413642 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 06:15:38.423515 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413645 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 06:15:38.424067 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413647 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 06:15:38.424067 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413650 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 06:15:38.424067 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413652 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 06:15:38.424067 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413655 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 06:15:38.424067 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413657 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 06:15:38.424067 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413660 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 06:15:38.424067 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413662 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 06:15:38.424067 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413665 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 06:15:38.424067 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413667 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 06:15:38.424067 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413672 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 06:15:38.424067 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413675 2579 feature_gate.go:386] unrecognized feature gate: Example Nov 02 06:15:38.424067 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413678 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 06:15:38.424067 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413681 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 06:15:38.424067 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413683 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 06:15:38.424067 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413687 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 06:15:38.424067 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413690 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 06:15:38.424067 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413693 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 06:15:38.424067 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413695 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 06:15:38.424067 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413699 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 06:15:38.424506 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413701 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 06:15:38.424506 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413704 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 06:15:38.424506 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413707 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 06:15:38.424506 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413709 2579 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 06:15:38.424506 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413712 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 06:15:38.424506 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413715 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 06:15:38.424506 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413717 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 06:15:38.424506 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413719 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 06:15:38.424506 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413722 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 06:15:38.424506 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413724 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 06:15:38.424506 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413727 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 06:15:38.424506 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413730 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 06:15:38.424506 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413732 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 06:15:38.424506 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413735 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 06:15:38.424506 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413737 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 06:15:38.424506 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413740 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 06:15:38.424506 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413743 2579 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 06:15:38.424506 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413747 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 06:15:38.424506 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413750 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 06:15:38.424961 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413754 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 06:15:38.424961 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413757 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 06:15:38.424961 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413760 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 06:15:38.424961 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413764 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 06:15:38.424961 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413767 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 06:15:38.424961 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413770 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 06:15:38.424961 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413773 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 06:15:38.424961 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413775 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 06:15:38.424961 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413778 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 06:15:38.424961 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413781 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 06:15:38.424961 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413784 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 06:15:38.424961 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413786 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 06:15:38.424961 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413790 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 06:15:38.424961 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413793 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 06:15:38.424961 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413796 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 06:15:38.424961 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413798 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 06:15:38.424961 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413801 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 06:15:38.424961 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413803 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 06:15:38.424961 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413806 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 06:15:38.424961 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413808 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 06:15:38.425426 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413811 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 06:15:38.425426 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413813 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 06:15:38.425426 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413816 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 06:15:38.425426 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413818 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 06:15:38.425426 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413821 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 06:15:38.425426 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413823 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 06:15:38.425426 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413827 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 06:15:38.425426 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413830 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 06:15:38.425426 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413833 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 06:15:38.425426 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413835 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 06:15:38.425426 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413838 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 06:15:38.425426 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413840 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 06:15:38.425426 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413843 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 06:15:38.425426 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413846 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 06:15:38.425426 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413848 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 06:15:38.425426 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413852 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 06:15:38.425426 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413854 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 06:15:38.425426 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413858 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 06:15:38.425426 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413861 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 06:15:38.425864 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413865 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 06:15:38.425864 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413868 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 06:15:38.425864 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413870 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 06:15:38.425864 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413874 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 06:15:38.425864 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413877 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 06:15:38.425864 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413881 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 06:15:38.425864 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413884 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 06:15:38.425864 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.413887 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 06:15:38.425864 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.421189 2579 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 02 06:15:38.425864 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.421207 2579 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 02 06:15:38.425864 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.421931 2579 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 02 06:15:38.425864 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.421938 2579 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 02 06:15:38.425864 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422021 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 06:15:38.425864 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422027 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 06:15:38.425864 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422030 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 06:15:38.425864 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422033 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 06:15:38.425864 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422036 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 06:15:38.425864 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422038 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 06:15:38.425864 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422042 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 06:15:38.426323 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422044 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 06:15:38.426323 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422047 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 06:15:38.426323 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422050 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 06:15:38.426323 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422052 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 06:15:38.426323 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422055 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 06:15:38.426323 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422058 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 06:15:38.426323 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422062 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 06:15:38.426323 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422066 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 06:15:38.426323 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422069 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 06:15:38.426323 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422072 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 06:15:38.426323 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422074 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 06:15:38.426323 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422078 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 06:15:38.426323 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422081 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 06:15:38.426323 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422083 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 06:15:38.426323 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422086 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 06:15:38.426323 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422089 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 06:15:38.426323 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422092 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 06:15:38.426323 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422094 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 06:15:38.426323 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422097 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 06:15:38.426771 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422099 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 06:15:38.426771 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422102 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 06:15:38.426771 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422104 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 06:15:38.426771 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422109 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 06:15:38.426771 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422112 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 06:15:38.426771 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422114 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 06:15:38.426771 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422117 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 06:15:38.426771 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422120 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 06:15:38.426771 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422122 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 06:15:38.426771 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422125 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 06:15:38.426771 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422128 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 06:15:38.426771 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422130 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 06:15:38.426771 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422133 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 06:15:38.426771 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422135 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 06:15:38.426771 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422138 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 06:15:38.426771 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422141 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 06:15:38.426771 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422143 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 06:15:38.426771 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422146 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 06:15:38.426771 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422148 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 06:15:38.426771 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422151 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 06:15:38.427275 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422154 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 06:15:38.427275 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422156 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 06:15:38.427275 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422159 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 06:15:38.427275 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422162 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 06:15:38.427275 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422164 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 06:15:38.427275 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422167 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 06:15:38.427275 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422169 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 06:15:38.427275 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422172 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 06:15:38.427275 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422174 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 06:15:38.427275 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422177 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 06:15:38.427275 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422180 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 06:15:38.427275 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422182 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 06:15:38.427275 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422184 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 06:15:38.427275 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422187 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 06:15:38.427275 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422190 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 06:15:38.427275 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422193 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 06:15:38.427275 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422196 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 06:15:38.427275 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422199 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 06:15:38.427275 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422201 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 06:15:38.427275 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422204 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 06:15:38.427809 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422206 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 06:15:38.427809 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422209 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 06:15:38.427809 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422211 2579 feature_gate.go:386] unrecognized feature gate: Example Nov 02 06:15:38.427809 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422214 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 06:15:38.427809 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422216 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 06:15:38.427809 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422219 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 06:15:38.427809 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422221 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 06:15:38.427809 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422224 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 06:15:38.427809 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422227 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 06:15:38.427809 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422229 2579 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 06:15:38.427809 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422231 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 06:15:38.427809 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422234 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 06:15:38.427809 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422236 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 06:15:38.427809 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422239 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 06:15:38.427809 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422241 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 06:15:38.427809 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422244 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 06:15:38.427809 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422246 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 06:15:38.427809 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422249 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 06:15:38.427809 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422251 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 06:15:38.427809 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422254 2579 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 06:15:38.427809 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422256 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 06:15:38.428296 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422259 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 06:15:38.428296 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422261 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 06:15:38.428296 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422264 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 06:15:38.428296 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422266 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 06:15:38.428296 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422269 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 06:15:38.428296 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422271 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 06:15:38.428296 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422274 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 06:15:38.428296 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422276 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 06:15:38.428296 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422279 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 06:15:38.428296 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422282 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 06:15:38.428296 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422285 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 06:15:38.428296 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422287 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 06:15:38.428296 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422290 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 06:15:38.428296 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422292 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 06:15:38.428296 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422295 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 06:15:38.428296 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422299 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 06:15:38.428296 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422302 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 06:15:38.428296 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422305 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 06:15:38.428296 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422308 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 06:15:38.428763 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.422336 2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 06:15:38.428763 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422472 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 06:15:38.428763 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422477 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 06:15:38.428763 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422480 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 06:15:38.428763 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422483 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 06:15:38.428763 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422486 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 06:15:38.428763 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422488 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 06:15:38.428763 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422491 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 06:15:38.428763 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422493 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 06:15:38.428763 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422496 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 06:15:38.428763 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422498 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 06:15:38.428763 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422501 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 06:15:38.428763 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422504 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 06:15:38.428763 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422507 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 06:15:38.428763 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422510 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 06:15:38.428763 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422512 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 06:15:38.429149 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422515 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 06:15:38.429149 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422518 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 06:15:38.429149 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422520 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 06:15:38.429149 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422522 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 06:15:38.429149 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422525 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 06:15:38.429149 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422528 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 06:15:38.429149 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422531 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 06:15:38.429149 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422533 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 06:15:38.429149 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422535 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 06:15:38.429149 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422538 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 06:15:38.429149 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422541 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 06:15:38.429149 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422543 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 06:15:38.429149 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422545 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 06:15:38.429149 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422548 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 06:15:38.429149 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422552 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 06:15:38.429149 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422555 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 06:15:38.429149 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422558 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 06:15:38.429149 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422561 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 06:15:38.429149 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422563 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 06:15:38.429602 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422566 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 06:15:38.429602 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422568 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 06:15:38.429602 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422571 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 06:15:38.429602 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422573 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 06:15:38.429602 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422576 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 06:15:38.429602 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422578 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 06:15:38.429602 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422581 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 06:15:38.429602 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422583 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 06:15:38.429602 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422586 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 06:15:38.429602 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422588 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 06:15:38.429602 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422591 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 06:15:38.429602 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422593 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 06:15:38.429602 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422596 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 06:15:38.429602 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422598 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 06:15:38.429602 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422601 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 06:15:38.429602 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422603 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 06:15:38.429602 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422606 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 06:15:38.429602 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422608 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 06:15:38.429602 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422611 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 06:15:38.429602 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422613 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 06:15:38.430075 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422616 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 06:15:38.430075 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422619 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 06:15:38.430075 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422621 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 06:15:38.430075 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422624 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 06:15:38.430075 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422626 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 06:15:38.430075 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422629 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 06:15:38.430075 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422631 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 06:15:38.430075 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422634 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 06:15:38.430075 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422637 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 06:15:38.430075 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422639 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 06:15:38.430075 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422642 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 06:15:38.430075 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422645 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 06:15:38.430075 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422647 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 06:15:38.430075 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422650 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 06:15:38.430075 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422652 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 06:15:38.430075 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422655 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 06:15:38.430075 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422657 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 06:15:38.430075 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422661 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 06:15:38.430075 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422665 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 06:15:38.430519 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422668 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 06:15:38.430519 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422671 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 06:15:38.430519 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422673 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 06:15:38.430519 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422676 2579 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 06:15:38.430519 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422679 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 06:15:38.430519 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422682 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 06:15:38.430519 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422684 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 06:15:38.430519 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422686 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 06:15:38.430519 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422689 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 06:15:38.430519 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422693 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 06:15:38.430519 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422696 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 06:15:38.430519 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422698 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 06:15:38.430519 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422701 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 06:15:38.430519 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422703 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 06:15:38.430519 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422706 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 06:15:38.430519 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422709 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 06:15:38.430519 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422711 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 06:15:38.430519 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422714 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 06:15:38.430519 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422716 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 06:15:38.430519 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422719 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 06:15:38.431170 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422721 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 06:15:38.431170 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422724 2579 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 06:15:38.431170 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422726 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 06:15:38.431170 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422729 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 06:15:38.431170 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422731 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 06:15:38.431170 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422734 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 06:15:38.431170 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422736 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 06:15:38.431170 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422739 2579 feature_gate.go:386] unrecognized feature gate: Example Nov 02 06:15:38.431170 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422741 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 06:15:38.431170 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422743 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 06:15:38.431170 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422746 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 06:15:38.431170 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422748 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 06:15:38.431170 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422753 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 06:15:38.431170 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.422776 2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 06:15:38.431170 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422806 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 06:15:38.431170 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422810 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 06:15:38.431548 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422813 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 06:15:38.431548 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422816 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 06:15:38.431548 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422819 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 06:15:38.431548 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422821 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 06:15:38.431548 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422824 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 06:15:38.431548 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422828 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 06:15:38.431548 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422830 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 06:15:38.431548 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422833 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 06:15:38.431548 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422835 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 06:15:38.431548 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422838 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 06:15:38.431548 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422841 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 06:15:38.431548 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422844 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 06:15:38.431548 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422846 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 06:15:38.431548 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422849 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 06:15:38.431548 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422852 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 06:15:38.431548 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422854 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 06:15:38.431548 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422857 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 06:15:38.431548 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422859 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 06:15:38.431548 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422861 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 06:15:38.431548 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422864 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 06:15:38.432027 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422867 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 06:15:38.432027 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422869 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 06:15:38.432027 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422872 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 06:15:38.432027 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422875 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 06:15:38.432027 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422877 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 06:15:38.432027 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422880 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 06:15:38.432027 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422883 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 06:15:38.432027 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422886 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 06:15:38.432027 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422890 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 06:15:38.432027 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422895 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 06:15:38.432027 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422898 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 06:15:38.432027 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422901 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 06:15:38.432027 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422916 2579 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 06:15:38.432027 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422919 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 06:15:38.432027 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422921 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 06:15:38.432027 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422924 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 06:15:38.432027 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422927 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 06:15:38.432027 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422931 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 06:15:38.432027 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422933 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 06:15:38.432475 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422937 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 06:15:38.432475 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422939 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 06:15:38.432475 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422942 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 06:15:38.432475 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422945 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 06:15:38.432475 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422948 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 06:15:38.432475 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422951 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 06:15:38.432475 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422953 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 06:15:38.432475 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422956 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 06:15:38.432475 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422959 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 06:15:38.432475 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422961 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 06:15:38.432475 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422964 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 06:15:38.432475 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422966 2579 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 06:15:38.432475 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422969 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 06:15:38.432475 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422971 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 06:15:38.432475 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422974 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 06:15:38.432475 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422977 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 06:15:38.432475 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422979 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 06:15:38.432475 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422982 2579 feature_gate.go:386] unrecognized feature gate: Example Nov 02 06:15:38.432475 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422984 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 06:15:38.432475 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422987 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 06:15:38.432947 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422989 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 06:15:38.432947 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422992 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 06:15:38.432947 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422996 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 06:15:38.432947 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.422999 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 06:15:38.432947 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.423001 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 06:15:38.432947 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.423004 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 06:15:38.432947 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.423006 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 06:15:38.432947 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.423009 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 06:15:38.432947 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.423011 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 06:15:38.432947 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.423014 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 06:15:38.432947 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.423018 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 06:15:38.432947 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.423020 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 06:15:38.432947 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.423023 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 06:15:38.432947 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.423026 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 06:15:38.432947 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.423028 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 06:15:38.432947 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.423031 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 06:15:38.432947 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.423033 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 06:15:38.432947 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.423036 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 06:15:38.432947 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.423039 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 06:15:38.433639 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.423042 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 06:15:38.433639 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.423044 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 06:15:38.433639 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.423047 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 06:15:38.433639 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.423049 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 06:15:38.433639 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.423052 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 06:15:38.433639 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.423055 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 06:15:38.433639 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.423057 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 06:15:38.433639 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.423060 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 06:15:38.433639 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.423062 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 06:15:38.433639 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.423065 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 06:15:38.433639 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.423067 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 06:15:38.433639 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.423070 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 06:15:38.433639 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.423072 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 06:15:38.433639 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.423076 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 06:15:38.433639 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.423079 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 06:15:38.433639 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.423084 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 06:15:38.433639 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.423086 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 06:15:38.433639 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.423089 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 06:15:38.433639 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.423092 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 06:15:38.434429 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.423094 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 06:15:38.434429 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.423097 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 06:15:38.434429 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.423099 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 06:15:38.434429 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.423102 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 06:15:38.434429 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.423107 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 06:15:38.434429 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.423109 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 06:15:38.434429 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:38.423112 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 06:15:38.434429 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.423162 2579 server.go:957] "Client rotation is on, will bootstrap in background" Nov 02 06:15:38.434429 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.427237 2579 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Nov 02 06:15:38.434429 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.428369 2579 server.go:1014] "Starting client certificate rotation" Nov 02 06:15:38.434429 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.428470 2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 02 06:15:38.434429 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.428501 2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 02 06:15:38.434429 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.430782 2579 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 06:15:38.435808 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.435786 2579 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 06:15:38.447559 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.447541 2579 log.go:25] "Validated CRI v1 runtime API" Nov 02 06:15:38.451368 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.451350 2579 log.go:25] "Validated CRI v1 image API" Nov 02 06:15:38.452286 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.452260 2579 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 02 06:15:38.459364 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.459335 2579 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 a3052cc1-5d54-4510-a252-8750d465bd42:/dev/nvme0n1p4 b74f8c3a-d9d7-412c-b5ea-ebb494b55370:/dev/nvme0n1p3] Nov 02 06:15:38.459420 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.459363 2579 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Nov 02 06:15:38.464664 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.464547 2579 manager.go:217] Machine: {Timestamp:2026-11-02 06:15:38.463117874 +0000 UTC m=+0.354936128 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3113107 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2f38e4cdb73e9d854a14e852006bb4 SystemUUID:ec2f38e4-cdb7-3e9d-854a-14e852006bb4 BootID:0257ffa4-a85a-4582-93f8-e3183ee282db Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:06:ff:ca:67:ab:dd Speed:0 Mtu:9001} {Name:ens5 MacAddress:06:ff:ca:67:ab:dd Speed:0 Mtu:9001} {Name:ovs-system MacAddress:7a:78:2b:b9:3d:6b Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 02 06:15:38.465432 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.465420 2579 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 02 06:15:38.465572 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.465554 2579 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 02 06:15:38.468448 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.468416 2579 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 02 06:15:38.468600 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.468449 2579 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-93.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 02 06:15:38.468650 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.468602 2579 topology_manager.go:143] "Creating topology manager with none policy" Nov 02 06:15:38.468650 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.468611 2579 container_manager_linux.go:311] "Creating device plugin manager" Nov 02 06:15:38.468650 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.468648 2579 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 06:15:38.468732 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.468669 2579 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 06:15:38.469926 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.469915 2579 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 02 06:15:38.474415 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.474401 2579 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 02 06:15:38.474544 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.474534 2579 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 02 06:15:38.475369 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.475359 2579 kubelet.go:493] "Attempting to sync node with API server" Nov 02 06:15:38.475406 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.475374 2579 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 02 06:15:38.475406 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.475385 2579 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 02 06:15:38.475406 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.475396 2579 kubelet.go:405] "Adding apiserver pod source" Nov 02 06:15:38.475406 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.475406 2579 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 02 06:15:38.475775 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.475763 2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 06:15:38.478317 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.478301 2579 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 02 06:15:38.485052 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.485036 2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 06:15:38.486004 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.485992 2579 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 02 06:15:38.486047 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.486022 2579 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 02 06:15:38.486185 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.486174 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 02 06:15:38.486229 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.486192 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 02 06:15:38.486229 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.486198 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 02 06:15:38.486229 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.486205 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 02 06:15:38.486229 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.486210 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 02 06:15:38.486229 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.486216 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 02 06:15:38.487190 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.487180 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 02 06:15:38.487229 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.487191 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 02 06:15:38.487229 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.487199 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 02 06:15:38.487229 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.487205 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 02 06:15:38.487229 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.487219 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 02 06:15:38.487229 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.487228 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 02 06:15:38.488958 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.488939 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 02 06:15:38.489013 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.488966 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 02 06:15:38.494791 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.494773 2579 server.go:1263] "Started kubelet" Nov 02 06:15:38.494879 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.494835 2579 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 02 06:15:38.494988 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.494933 2579 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 02 06:15:38.495032 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.495013 2579 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 02 06:15:38.495603 ip-10-0-130-93 systemd[1]: Started Kubernetes Kubelet. Nov 02 06:15:38.496575 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.496559 2579 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 02 06:15:38.496949 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.496928 2579 server.go:317] "Adding debug handlers to kubelet server" Nov 02 06:15:38.502400 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:15:38.502384 2579 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Nov 02 06:15:38.504891 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.504873 2579 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 02 06:15:38.505010 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.504877 2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 02 06:15:38.505075 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:15:38.505034 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-93.us-west-2.compute.internal\" not found" Nov 02 06:15:38.505623 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.505605 2579 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 02 06:15:38.505973 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.505954 2579 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 02 06:15:38.505973 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.505976 2579 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 02 06:15:38.506102 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.506089 2579 reconstruct.go:97] "Volume reconstruction finished" Nov 02 06:15:38.506102 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.506101 2579 reconciler.go:29] "Reconciler: start to sync state" Nov 02 06:15:38.506248 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.506234 2579 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 02 06:15:38.506310 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.506249 2579 factory.go:55] Registering systemd factory Nov 02 06:15:38.506310 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.506258 2579 factory.go:223] Registration of the systemd container factory successfully Nov 02 06:15:38.506483 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.506468 2579 factory.go:153] Registering CRI-O factory Nov 02 06:15:38.506483 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.506483 2579 factory.go:223] Registration of the crio container factory successfully Nov 02 06:15:38.506587 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.506497 2579 factory.go:103] Registering Raw factory Nov 02 06:15:38.506587 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.506506 2579 manager.go:1199] Started watching for new ooms in manager Nov 02 06:15:38.507498 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.507484 2579 manager.go:319] Starting recovery of all containers Nov 02 06:15:38.517836 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.517805 2579 manager.go:324] Recovery completed Nov 02 06:15:38.523509 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.523322 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 06:15:38.525683 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.525663 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-93.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 06:15:38.525772 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.525700 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-93.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 06:15:38.525772 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.525716 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-93.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 06:15:38.526328 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.526308 2579 cpu_manager.go:226] "Starting" policy="none" Nov 02 06:15:38.526328 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.526321 2579 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 02 06:15:38.526465 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.526342 2579 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 02 06:15:38.526465 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.526354 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 06:15:38.527492 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.527472 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-93.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 06:15:38.527572 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.527509 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-93.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 06:15:38.527572 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.527525 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-93.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 06:15:38.528489 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.528475 2579 policy_none.go:50] "Start" Nov 02 06:15:38.528550 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.528495 2579 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 02 06:15:38.528550 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.528508 2579 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 02 06:15:38.530439 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.530426 2579 policy_none.go:44] "Start" Nov 02 06:15:38.569506 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.569474 2579 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 02 06:15:38.571948 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.571932 2579 manager.go:341] "Starting Device Plugin manager" Nov 02 06:15:38.572054 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:15:38.571963 2579 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 02 06:15:38.572054 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.571971 2579 server.go:87] "Starting device plugin registration server" Nov 02 06:15:38.572268 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.572254 2579 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 02 06:15:38.572313 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.572271 2579 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 02 06:15:38.572375 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.572359 2579 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 02 06:15:38.572459 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.572448 2579 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 02 06:15:38.572459 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.572459 2579 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 02 06:15:38.574845 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:15:38.574818 2579 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 02 06:15:38.574957 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:15:38.574888 2579 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-130-93.us-west-2.compute.internal\" not found" Nov 02 06:15:38.626642 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.626588 2579 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-130-93.us-west-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Nov 02 06:15:38.626642 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:15:38.626598 2579 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-130-93.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Nov 02 06:15:38.628672 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:15:38.626583 2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-93.us-west-2.compute.internal.18e425d2b4120dcb default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-93.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-130-93.us-west-2.compute.internal,},FirstTimestamp:2026-11-02 06:15:38.494741963 +0000 UTC m=+0.386560217,LastTimestamp:2026-11-02 06:15:38.494741963 +0000 UTC m=+0.386560217,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-93.us-west-2.compute.internal,}" Nov 02 06:15:38.646300 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.646264 2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 02 06:15:38.647546 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.647525 2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 02 06:15:38.647636 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.647553 2579 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 02 06:15:38.647636 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.647575 2579 kubelet.go:2521] "Starting kubelet main sync loop" Nov 02 06:15:38.647636 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:15:38.647618 2579 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Nov 02 06:15:38.673335 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.673272 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 06:15:38.678141 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.678124 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-93.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 06:15:38.678227 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.678152 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-93.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 06:15:38.678227 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.678165 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-93.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 06:15:38.678227 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.678199 2579 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-130-93.us-west-2.compute.internal" Nov 02 06:15:38.688767 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.688743 2579 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 02 06:15:38.688896 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.688801 2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-lnsvb" Nov 02 06:15:38.748086 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.748043 2579 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-93.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-130-93.us-west-2.compute.internal"] Nov 02 06:15:38.748209 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.748124 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 06:15:38.749391 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.749377 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-93.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 06:15:38.749466 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.749404 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-93.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 06:15:38.749466 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.749413 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-93.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 06:15:38.751581 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.751568 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 06:15:38.751728 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.751713 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-93.us-west-2.compute.internal" Nov 02 06:15:38.751761 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.751746 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 06:15:38.754813 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.754792 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-93.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 06:15:38.754876 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.754821 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-93.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 06:15:38.754876 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.754831 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-93.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 06:15:38.754876 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.754792 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-93.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 06:15:38.754999 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.754897 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-93.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 06:15:38.754999 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.754924 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-93.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 06:15:38.756964 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.756948 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-93.us-west-2.compute.internal" Nov 02 06:15:38.757057 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.756982 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 06:15:38.758568 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.758549 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-93.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 06:15:38.758654 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.758584 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-93.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 06:15:38.758654 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.758599 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-93.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 06:15:38.777561 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.777544 2579 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 06:15:38.779191 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.779174 2579 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 06:15:38.783600 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:15:38.783582 2579 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-93.us-west-2.compute.internal\" not found" node="ip-10-0-130-93.us-west-2.compute.internal" Nov 02 06:15:38.786110 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:15:38.786093 2579 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-93.us-west-2.compute.internal\" not found" node="ip-10-0-130-93.us-west-2.compute.internal" Nov 02 06:15:38.807451 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.807428 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/71bd052da0df40177024b26f7f0347a4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-93.us-west-2.compute.internal\" (UID: \"71bd052da0df40177024b26f7f0347a4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-93.us-west-2.compute.internal" Nov 02 06:15:38.807540 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.807454 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/71bd052da0df40177024b26f7f0347a4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-93.us-west-2.compute.internal\" (UID: \"71bd052da0df40177024b26f7f0347a4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-93.us-west-2.compute.internal" Nov 02 06:15:38.807540 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.807473 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/df7856ce915c3ef5bc71a7a412d369e5-config\") pod \"kube-apiserver-proxy-ip-10-0-130-93.us-west-2.compute.internal\" (UID: \"df7856ce915c3ef5bc71a7a412d369e5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-93.us-west-2.compute.internal" Nov 02 06:15:38.872686 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.872655 2579 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 06:15:38.873145 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.873129 2579 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 06:15:38.874278 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.874252 2579 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-130-93.us-west-2.compute.internal" Nov 02 06:15:38.905186 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.905135 2579 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-93.us-west-2.compute.internal" Nov 02 06:15:38.908406 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.908386 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/df7856ce915c3ef5bc71a7a412d369e5-config\") pod \"kube-apiserver-proxy-ip-10-0-130-93.us-west-2.compute.internal\" (UID: \"df7856ce915c3ef5bc71a7a412d369e5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-93.us-west-2.compute.internal" Nov 02 06:15:38.908473 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.908414 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/71bd052da0df40177024b26f7f0347a4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-93.us-west-2.compute.internal\" (UID: \"71bd052da0df40177024b26f7f0347a4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-93.us-west-2.compute.internal" Nov 02 06:15:38.908473 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.908433 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/71bd052da0df40177024b26f7f0347a4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-93.us-west-2.compute.internal\" (UID: \"71bd052da0df40177024b26f7f0347a4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-93.us-west-2.compute.internal" Nov 02 06:15:38.908536 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.908486 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/71bd052da0df40177024b26f7f0347a4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-93.us-west-2.compute.internal\" (UID: \"71bd052da0df40177024b26f7f0347a4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-93.us-west-2.compute.internal" Nov 02 06:15:38.908536 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.908501 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/71bd052da0df40177024b26f7f0347a4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-93.us-west-2.compute.internal\" (UID: \"71bd052da0df40177024b26f7f0347a4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-93.us-west-2.compute.internal" Nov 02 06:15:38.908536 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.908501 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/df7856ce915c3ef5bc71a7a412d369e5-config\") pod \"kube-apiserver-proxy-ip-10-0-130-93.us-west-2.compute.internal\" (UID: \"df7856ce915c3ef5bc71a7a412d369e5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-93.us-west-2.compute.internal" Nov 02 06:15:38.995531 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.995454 2579 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 06:15:38.995669 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:38.995559 2579 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-93.us-west-2.compute.internal" Nov 02 06:15:39.050099 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.050072 2579 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 06:15:39.087282 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.087258 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-93.us-west-2.compute.internal" Nov 02 06:15:39.090383 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.090365 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-93.us-west-2.compute.internal" Nov 02 06:15:39.428089 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.428000 2579 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Nov 02 06:15:39.428711 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.428181 2579 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 02 06:15:39.428711 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.428210 2579 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 02 06:15:39.428711 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:15:39.428274 2579 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://af4ae981bd91148c598c3a76754e16e4-db7866749802e67b.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-130-93.us-west-2.compute.internal.18e425d2b5ea2927\": read tcp 10.0.130.93:40206->3.140.45.243:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-130-93.us-west-2.compute.internal.18e425d2b5ea2927 default 4632 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-93.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-130-93.us-west-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-130-93.us-west-2.compute.internal,},FirstTimestamp:2026-11-02 06:15:38 +0000 UTC,LastTimestamp:2026-11-02 06:15:38.749390047 +0000 UTC m=+0.641208300,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-93.us-west-2.compute.internal,}" Nov 02 06:15:39.475794 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.475768 2579 apiserver.go:52] "Watching apiserver" Nov 02 06:15:39.505251 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.505229 2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Nov 02 06:15:39.694791 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.694694 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 06:10:38 +0000 UTC" deadline="2028-06-01 06:55:29.498950531 +0000 UTC" Nov 02 06:15:39.694791 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.694735 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13848h39m49.804218484s" Nov 02 06:15:39.712098 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.712073 2579 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 02 06:15:39.712365 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.712349 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cbzvd","openshift-multus/multus-additional-cni-plugins-ckddw","openshift-multus/network-metrics-daemon-sb4nt","openshift-network-operator/iptables-alerter-6fnx8","kube-system/konnectivity-agent-hwc99","openshift-dns/node-resolver-2p95f","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-93.us-west-2.compute.internal","openshift-ovn-kubernetes/ovnkube-node-7wb4x","openshift-cluster-node-tuning-operator/tuned-6nk8l","openshift-multus/multus-rf7qp","openshift-network-diagnostics/network-check-target-ft7gs","kube-system/global-pull-secret-syncer-ggpsr","kube-system/kube-apiserver-proxy-ip-10-0-130-93.us-west-2.compute.internal","openshift-image-registry/node-ca-qm6wn"] Nov 02 06:15:39.714760 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.714740 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-rf7qp" Nov 02 06:15:39.716739 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.716723 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sb4nt" Nov 02 06:15:39.716811 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:15:39.716793 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sb4nt" podUID="3b6432a8-b0ce-4954-8e3d-8eb8782557eb" Nov 02 06:15:39.718855 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.718841 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-7wb4x" Nov 02 06:15:39.720840 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.720819 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-hwc99" Nov 02 06:15:39.722754 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.722742 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-2p95f" Nov 02 06:15:39.724703 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.724690 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ft7gs" Nov 02 06:15:39.724758 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:15:39.724743 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ft7gs" podUID="3ced47b0-9cd0-4710-8f75-2a333437ed01" Nov 02 06:15:39.726748 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.726734 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-ckddw" Nov 02 06:15:39.729116 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.728941 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cbzvd" Nov 02 06:15:39.731206 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.731188 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-6nk8l" Nov 02 06:15:39.733397 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.733384 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ggpsr" Nov 02 06:15:39.733444 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:15:39.733431 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ggpsr" podUID="b65bb247-83f7-4749-9aa1-9b2677dd1b3d" Nov 02 06:15:39.735471 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.735458 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-6fnx8" Nov 02 06:15:39.737627 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.737612 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-qm6wn" Nov 02 06:15:39.764778 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.764752 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 02 06:15:39.764881 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.764756 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 02 06:15:39.764881 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.764756 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 02 06:15:39.764881 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.764827 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-bpw78\"" Nov 02 06:15:39.764881 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.764833 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 02 06:15:39.767343 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.767319 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 02 06:15:39.767446 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.767412 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 02 06:15:39.767446 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.767429 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 02 06:15:39.767529 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.767455 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 02 06:15:39.767529 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.767468 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 02 06:15:39.767529 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.767506 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-flhk8\"" Nov 02 06:15:39.767727 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.767641 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 02 06:15:39.768892 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.768870 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 02 06:15:39.769073 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.769060 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-zth7z\"" Nov 02 06:15:39.769128 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.769064 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 02 06:15:39.770632 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.770616 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 02 06:15:39.770765 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.770752 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 02 06:15:39.771294 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.771281 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-d57td\"" Nov 02 06:15:39.806637 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.806614 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 02 06:15:39.806637 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.806620 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 02 06:15:39.806756 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.806621 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-rwgrc\"" Nov 02 06:15:39.806756 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.806704 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 02 06:15:39.806756 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.806746 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-xffwp\"" Nov 02 06:15:39.807029 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.807017 2579 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 02 06:15:39.809544 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.809528 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-d5zsv\"" Nov 02 06:15:39.809624 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.809544 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-4lqj8\"" Nov 02 06:15:39.809624 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.809597 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 02 06:15:39.809719 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.809674 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 02 06:15:39.809872 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.809858 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 02 06:15:39.809961 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.809945 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 02 06:15:39.810019 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.809957 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 02 06:15:39.810019 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.809989 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 02 06:15:39.810019 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.809993 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 02 06:15:39.810019 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.810013 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 02 06:15:39.810165 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.810036 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 02 06:15:39.810165 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.810055 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 02 06:15:39.810165 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.810063 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-77l82\"" Nov 02 06:15:39.810165 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.810152 2579 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 02 06:15:39.810276 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.810219 2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-rnhkz" Nov 02 06:15:39.813300 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.813284 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f14e8781-bf82-4810-9909-c126ccdc1ce1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ckddw\" (UID: \"f14e8781-bf82-4810-9909-c126ccdc1ce1\") " pod="openshift-multus/multus-additional-cni-plugins-ckddw" Nov 02 06:15:39.813376 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.813313 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a2ef8269-c442-42a8-8041-6c1b6aaf0746-log-socket\") pod \"ovnkube-node-7wb4x\" (UID: \"a2ef8269-c442-42a8-8041-6c1b6aaf0746\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wb4x" Nov 02 06:15:39.813376 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.813334 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8lqt\" (UniqueName: \"kubernetes.io/projected/3500e682-2f15-45a2-92d6-bbc35dae3231-kube-api-access-v8lqt\") pod \"aws-ebs-csi-driver-node-cbzvd\" (UID: \"3500e682-2f15-45a2-92d6-bbc35dae3231\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cbzvd" Nov 02 06:15:39.813376 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.813359 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b65bb247-83f7-4749-9aa1-9b2677dd1b3d-dbus\") pod \"global-pull-secret-syncer-ggpsr\" (UID: \"b65bb247-83f7-4749-9aa1-9b2677dd1b3d\") " pod="kube-system/global-pull-secret-syncer-ggpsr" Nov 02 06:15:39.813475 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.813401 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/87e797a0-1da6-41ca-af8d-7382afaf3987-cni-binary-copy\") pod \"multus-rf7qp\" (UID: \"87e797a0-1da6-41ca-af8d-7382afaf3987\") " pod="openshift-multus/multus-rf7qp" Nov 02 06:15:39.813475 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.813426 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3419bd74-22aa-4841-87d2-c4340e01dae9-host-slash\") pod \"iptables-alerter-6fnx8\" (UID: \"3419bd74-22aa-4841-87d2-c4340e01dae9\") " pod="openshift-network-operator/iptables-alerter-6fnx8" Nov 02 06:15:39.813475 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.813460 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a2ef8269-c442-42a8-8041-6c1b6aaf0746-host-slash\") pod \"ovnkube-node-7wb4x\" (UID: \"a2ef8269-c442-42a8-8041-6c1b6aaf0746\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wb4x" Nov 02 06:15:39.813602 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.813480 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a2ef8269-c442-42a8-8041-6c1b6aaf0746-host-run-netns\") pod \"ovnkube-node-7wb4x\" (UID: \"a2ef8269-c442-42a8-8041-6c1b6aaf0746\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wb4x" Nov 02 06:15:39.813602 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.813505 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a2ef8269-c442-42a8-8041-6c1b6aaf0746-var-lib-openvswitch\") pod \"ovnkube-node-7wb4x\" (UID: \"a2ef8269-c442-42a8-8041-6c1b6aaf0746\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wb4x" Nov 02 06:15:39.813602 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.813535 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a2ef8269-c442-42a8-8041-6c1b6aaf0746-node-log\") pod \"ovnkube-node-7wb4x\" (UID: \"a2ef8269-c442-42a8-8041-6c1b6aaf0746\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wb4x" Nov 02 06:15:39.813602 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.813561 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzxsj\" (UniqueName: \"kubernetes.io/projected/a2ef8269-c442-42a8-8041-6c1b6aaf0746-kube-api-access-qzxsj\") pod \"ovnkube-node-7wb4x\" (UID: \"a2ef8269-c442-42a8-8041-6c1b6aaf0746\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wb4x" Nov 02 06:15:39.813602 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.813588 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8d87317-3f41-46d4-a58e-3c7601e0d445-tmp-dir\") pod \"node-resolver-2p95f\" (UID: \"d8d87317-3f41-46d4-a58e-3c7601e0d445\") " pod="openshift-dns/node-resolver-2p95f" Nov 02 06:15:39.813806 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.813612 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3500e682-2f15-45a2-92d6-bbc35dae3231-kubelet-dir\") pod \"aws-ebs-csi-driver-node-cbzvd\" (UID: \"3500e682-2f15-45a2-92d6-bbc35dae3231\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cbzvd" Nov 02 06:15:39.813806 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.813627 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f14e8781-bf82-4810-9909-c126ccdc1ce1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ckddw\" (UID: \"f14e8781-bf82-4810-9909-c126ccdc1ce1\") " pod="openshift-multus/multus-additional-cni-plugins-ckddw" Nov 02 06:15:39.813806 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.813643 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3419bd74-22aa-4841-87d2-c4340e01dae9-iptables-alerter-script\") pod \"iptables-alerter-6fnx8\" (UID: \"3419bd74-22aa-4841-87d2-c4340e01dae9\") " pod="openshift-network-operator/iptables-alerter-6fnx8" Nov 02 06:15:39.813806 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.813657 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3500e682-2f15-45a2-92d6-bbc35dae3231-device-dir\") pod \"aws-ebs-csi-driver-node-cbzvd\" (UID: \"3500e682-2f15-45a2-92d6-bbc35dae3231\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cbzvd" Nov 02 06:15:39.813806 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.813675 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/81b393f2-42de-4a99-8114-1445de4d6dc8-etc-sysctl-conf\") pod \"tuned-6nk8l\" (UID: \"81b393f2-42de-4a99-8114-1445de4d6dc8\") " pod="openshift-cluster-node-tuning-operator/tuned-6nk8l" Nov 02 06:15:39.813806 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.813726 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/87e797a0-1da6-41ca-af8d-7382afaf3987-multus-socket-dir-parent\") pod \"multus-rf7qp\" (UID: \"87e797a0-1da6-41ca-af8d-7382afaf3987\") " pod="openshift-multus/multus-rf7qp" Nov 02 06:15:39.813806 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.813752 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/87e797a0-1da6-41ca-af8d-7382afaf3987-hostroot\") pod \"multus-rf7qp\" (UID: \"87e797a0-1da6-41ca-af8d-7382afaf3987\") " pod="openshift-multus/multus-rf7qp" Nov 02 06:15:39.813806 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.813779 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/87e797a0-1da6-41ca-af8d-7382afaf3987-multus-daemon-config\") pod \"multus-rf7qp\" (UID: \"87e797a0-1da6-41ca-af8d-7382afaf3987\") " pod="openshift-multus/multus-rf7qp" Nov 02 06:15:39.813806 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.813805 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d4615a9e-da82-4f8b-8225-d60836b1ba80-agent-certs\") pod \"konnectivity-agent-hwc99\" (UID: \"d4615a9e-da82-4f8b-8225-d60836b1ba80\") " pod="kube-system/konnectivity-agent-hwc99" Nov 02 06:15:39.814199 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.813825 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a2ef8269-c442-42a8-8041-6c1b6aaf0746-host-kubelet\") pod \"ovnkube-node-7wb4x\" (UID: \"a2ef8269-c442-42a8-8041-6c1b6aaf0746\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wb4x" Nov 02 06:15:39.814199 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.813840 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kcrm\" (UniqueName: \"kubernetes.io/projected/d8d87317-3f41-46d4-a58e-3c7601e0d445-kube-api-access-5kcrm\") pod \"node-resolver-2p95f\" (UID: \"d8d87317-3f41-46d4-a58e-3c7601e0d445\") " pod="openshift-dns/node-resolver-2p95f" Nov 02 06:15:39.814199 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.813857 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrq9j\" (UniqueName: \"kubernetes.io/projected/81b393f2-42de-4a99-8114-1445de4d6dc8-kube-api-access-hrq9j\") pod \"tuned-6nk8l\" (UID: \"81b393f2-42de-4a99-8114-1445de4d6dc8\") " pod="openshift-cluster-node-tuning-operator/tuned-6nk8l" Nov 02 06:15:39.814199 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.813881 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b65bb247-83f7-4749-9aa1-9b2677dd1b3d-kubelet-config\") pod \"global-pull-secret-syncer-ggpsr\" (UID: \"b65bb247-83f7-4749-9aa1-9b2677dd1b3d\") " pod="kube-system/global-pull-secret-syncer-ggpsr" Nov 02 06:15:39.814199 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.813896 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvdp9\" (UniqueName: \"kubernetes.io/projected/3b6432a8-b0ce-4954-8e3d-8eb8782557eb-kube-api-access-pvdp9\") pod \"network-metrics-daemon-sb4nt\" (UID: \"3b6432a8-b0ce-4954-8e3d-8eb8782557eb\") " pod="openshift-multus/network-metrics-daemon-sb4nt" Nov 02 06:15:39.814199 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.813934 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vz9mf\" (UniqueName: \"kubernetes.io/projected/3419bd74-22aa-4841-87d2-c4340e01dae9-kube-api-access-vz9mf\") pod \"iptables-alerter-6fnx8\" (UID: \"3419bd74-22aa-4841-87d2-c4340e01dae9\") " pod="openshift-network-operator/iptables-alerter-6fnx8" Nov 02 06:15:39.814199 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.813950 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3500e682-2f15-45a2-92d6-bbc35dae3231-registration-dir\") pod \"aws-ebs-csi-driver-node-cbzvd\" (UID: \"3500e682-2f15-45a2-92d6-bbc35dae3231\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cbzvd" Nov 02 06:15:39.814199 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.813967 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3eec1082-df0b-420b-b4ff-5a1c0fab1d1d-serviceca\") pod \"node-ca-qm6wn\" (UID: \"3eec1082-df0b-420b-b4ff-5a1c0fab1d1d\") " pod="openshift-image-registry/node-ca-qm6wn" Nov 02 06:15:39.814199 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.813985 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/87e797a0-1da6-41ca-af8d-7382afaf3987-host-var-lib-kubelet\") pod \"multus-rf7qp\" (UID: \"87e797a0-1da6-41ca-af8d-7382afaf3987\") " pod="openshift-multus/multus-rf7qp" Nov 02 06:15:39.814199 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.814028 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnq6k\" (UniqueName: \"kubernetes.io/projected/3ced47b0-9cd0-4710-8f75-2a333437ed01-kube-api-access-gnq6k\") pod \"network-check-target-ft7gs\" (UID: \"3ced47b0-9cd0-4710-8f75-2a333437ed01\") " pod="openshift-network-diagnostics/network-check-target-ft7gs" Nov 02 06:15:39.814199 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.814065 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a2ef8269-c442-42a8-8041-6c1b6aaf0746-host-run-ovn-kubernetes\") pod \"ovnkube-node-7wb4x\" (UID: \"a2ef8269-c442-42a8-8041-6c1b6aaf0746\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wb4x" Nov 02 06:15:39.814199 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.814082 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f14e8781-bf82-4810-9909-c126ccdc1ce1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ckddw\" (UID: \"f14e8781-bf82-4810-9909-c126ccdc1ce1\") " pod="openshift-multus/multus-additional-cni-plugins-ckddw" Nov 02 06:15:39.814199 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.814114 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7xzh\" (UniqueName: \"kubernetes.io/projected/f14e8781-bf82-4810-9909-c126ccdc1ce1-kube-api-access-v7xzh\") pod \"multus-additional-cni-plugins-ckddw\" (UID: \"f14e8781-bf82-4810-9909-c126ccdc1ce1\") " pod="openshift-multus/multus-additional-cni-plugins-ckddw" Nov 02 06:15:39.814199 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.814163 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3500e682-2f15-45a2-92d6-bbc35dae3231-etc-selinux\") pod \"aws-ebs-csi-driver-node-cbzvd\" (UID: \"3500e682-2f15-45a2-92d6-bbc35dae3231\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cbzvd" Nov 02 06:15:39.814199 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.814204 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/81b393f2-42de-4a99-8114-1445de4d6dc8-host\") pod \"tuned-6nk8l\" (UID: \"81b393f2-42de-4a99-8114-1445de4d6dc8\") " pod="openshift-cluster-node-tuning-operator/tuned-6nk8l" Nov 02 06:15:39.814668 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.814237 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/87e797a0-1da6-41ca-af8d-7382afaf3987-multus-cni-dir\") pod \"multus-rf7qp\" (UID: \"87e797a0-1da6-41ca-af8d-7382afaf3987\") " pod="openshift-multus/multus-rf7qp" Nov 02 06:15:39.814668 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.814291 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a2ef8269-c442-42a8-8041-6c1b6aaf0746-systemd-units\") pod \"ovnkube-node-7wb4x\" (UID: \"a2ef8269-c442-42a8-8041-6c1b6aaf0746\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wb4x" Nov 02 06:15:39.814668 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.814336 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a2ef8269-c442-42a8-8041-6c1b6aaf0746-run-ovn\") pod \"ovnkube-node-7wb4x\" (UID: \"a2ef8269-c442-42a8-8041-6c1b6aaf0746\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wb4x" Nov 02 06:15:39.814668 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.814371 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a2ef8269-c442-42a8-8041-6c1b6aaf0746-ovnkube-config\") pod \"ovnkube-node-7wb4x\" (UID: \"a2ef8269-c442-42a8-8041-6c1b6aaf0746\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wb4x" Nov 02 06:15:39.814668 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.814397 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3eec1082-df0b-420b-b4ff-5a1c0fab1d1d-host\") pod \"node-ca-qm6wn\" (UID: \"3eec1082-df0b-420b-b4ff-5a1c0fab1d1d\") " pod="openshift-image-registry/node-ca-qm6wn" Nov 02 06:15:39.814668 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.814415 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/87e797a0-1da6-41ca-af8d-7382afaf3987-cnibin\") pod \"multus-rf7qp\" (UID: \"87e797a0-1da6-41ca-af8d-7382afaf3987\") " pod="openshift-multus/multus-rf7qp" Nov 02 06:15:39.814668 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.814437 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/81b393f2-42de-4a99-8114-1445de4d6dc8-etc-modprobe-d\") pod \"tuned-6nk8l\" (UID: \"81b393f2-42de-4a99-8114-1445de4d6dc8\") " pod="openshift-cluster-node-tuning-operator/tuned-6nk8l" Nov 02 06:15:39.814668 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.814473 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b65bb247-83f7-4749-9aa1-9b2677dd1b3d-original-pull-secret\") pod \"global-pull-secret-syncer-ggpsr\" (UID: \"b65bb247-83f7-4749-9aa1-9b2677dd1b3d\") " pod="kube-system/global-pull-secret-syncer-ggpsr" Nov 02 06:15:39.814668 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.814489 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/87e797a0-1da6-41ca-af8d-7382afaf3987-host-var-lib-cni-bin\") pod \"multus-rf7qp\" (UID: \"87e797a0-1da6-41ca-af8d-7382afaf3987\") " pod="openshift-multus/multus-rf7qp" Nov 02 06:15:39.814668 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.814504 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/87e797a0-1da6-41ca-af8d-7382afaf3987-host-run-multus-certs\") pod \"multus-rf7qp\" (UID: \"87e797a0-1da6-41ca-af8d-7382afaf3987\") " pod="openshift-multus/multus-rf7qp" Nov 02 06:15:39.814668 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.814531 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f14e8781-bf82-4810-9909-c126ccdc1ce1-cnibin\") pod \"multus-additional-cni-plugins-ckddw\" (UID: \"f14e8781-bf82-4810-9909-c126ccdc1ce1\") " pod="openshift-multus/multus-additional-cni-plugins-ckddw" Nov 02 06:15:39.814668 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.814561 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a2ef8269-c442-42a8-8041-6c1b6aaf0746-run-openvswitch\") pod \"ovnkube-node-7wb4x\" (UID: \"a2ef8269-c442-42a8-8041-6c1b6aaf0746\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wb4x" Nov 02 06:15:39.814668 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.814583 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/81b393f2-42de-4a99-8114-1445de4d6dc8-etc-kubernetes\") pod \"tuned-6nk8l\" (UID: \"81b393f2-42de-4a99-8114-1445de4d6dc8\") " pod="openshift-cluster-node-tuning-operator/tuned-6nk8l" Nov 02 06:15:39.814668 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.814597 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/81b393f2-42de-4a99-8114-1445de4d6dc8-run\") pod \"tuned-6nk8l\" (UID: \"81b393f2-42de-4a99-8114-1445de4d6dc8\") " pod="openshift-cluster-node-tuning-operator/tuned-6nk8l" Nov 02 06:15:39.814668 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.814618 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/81b393f2-42de-4a99-8114-1445de4d6dc8-var-lib-kubelet\") pod \"tuned-6nk8l\" (UID: \"81b393f2-42de-4a99-8114-1445de4d6dc8\") " pod="openshift-cluster-node-tuning-operator/tuned-6nk8l" Nov 02 06:15:39.814668 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.814645 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f14e8781-bf82-4810-9909-c126ccdc1ce1-system-cni-dir\") pod \"multus-additional-cni-plugins-ckddw\" (UID: \"f14e8781-bf82-4810-9909-c126ccdc1ce1\") " pod="openshift-multus/multus-additional-cni-plugins-ckddw" Nov 02 06:15:39.814668 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.814664 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f14e8781-bf82-4810-9909-c126ccdc1ce1-cni-binary-copy\") pod \"multus-additional-cni-plugins-ckddw\" (UID: \"f14e8781-bf82-4810-9909-c126ccdc1ce1\") " pod="openshift-multus/multus-additional-cni-plugins-ckddw" Nov 02 06:15:39.815189 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.814677 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a2ef8269-c442-42a8-8041-6c1b6aaf0746-etc-openvswitch\") pod \"ovnkube-node-7wb4x\" (UID: \"a2ef8269-c442-42a8-8041-6c1b6aaf0746\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wb4x" Nov 02 06:15:39.815189 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.814708 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a2ef8269-c442-42a8-8041-6c1b6aaf0746-host-cni-bin\") pod \"ovnkube-node-7wb4x\" (UID: \"a2ef8269-c442-42a8-8041-6c1b6aaf0746\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wb4x" Nov 02 06:15:39.815189 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.814730 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/81b393f2-42de-4a99-8114-1445de4d6dc8-etc-sysconfig\") pod \"tuned-6nk8l\" (UID: \"81b393f2-42de-4a99-8114-1445de4d6dc8\") " pod="openshift-cluster-node-tuning-operator/tuned-6nk8l" Nov 02 06:15:39.815189 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.814753 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/87e797a0-1da6-41ca-af8d-7382afaf3987-host-run-k8s-cni-cncf-io\") pod \"multus-rf7qp\" (UID: \"87e797a0-1da6-41ca-af8d-7382afaf3987\") " pod="openshift-multus/multus-rf7qp" Nov 02 06:15:39.815189 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.814771 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6vx8\" (UniqueName: \"kubernetes.io/projected/87e797a0-1da6-41ca-af8d-7382afaf3987-kube-api-access-l6vx8\") pod \"multus-rf7qp\" (UID: \"87e797a0-1da6-41ca-af8d-7382afaf3987\") " pod="openshift-multus/multus-rf7qp" Nov 02 06:15:39.815189 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.814788 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a2ef8269-c442-42a8-8041-6c1b6aaf0746-env-overrides\") pod \"ovnkube-node-7wb4x\" (UID: \"a2ef8269-c442-42a8-8041-6c1b6aaf0746\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wb4x" Nov 02 06:15:39.815189 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.814804 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d8d87317-3f41-46d4-a58e-3c7601e0d445-hosts-file\") pod \"node-resolver-2p95f\" (UID: \"d8d87317-3f41-46d4-a58e-3c7601e0d445\") " pod="openshift-dns/node-resolver-2p95f" Nov 02 06:15:39.815189 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.814822 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/81b393f2-42de-4a99-8114-1445de4d6dc8-etc-sysctl-d\") pod \"tuned-6nk8l\" (UID: \"81b393f2-42de-4a99-8114-1445de4d6dc8\") " pod="openshift-cluster-node-tuning-operator/tuned-6nk8l" Nov 02 06:15:39.815189 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.814835 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/81b393f2-42de-4a99-8114-1445de4d6dc8-lib-modules\") pod \"tuned-6nk8l\" (UID: \"81b393f2-42de-4a99-8114-1445de4d6dc8\") " pod="openshift-cluster-node-tuning-operator/tuned-6nk8l" Nov 02 06:15:39.815189 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.814857 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/87e797a0-1da6-41ca-af8d-7382afaf3987-host-var-lib-cni-multus\") pod \"multus-rf7qp\" (UID: \"87e797a0-1da6-41ca-af8d-7382afaf3987\") " pod="openshift-multus/multus-rf7qp" Nov 02 06:15:39.815189 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.814899 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/87e797a0-1da6-41ca-af8d-7382afaf3987-etc-kubernetes\") pod \"multus-rf7qp\" (UID: \"87e797a0-1da6-41ca-af8d-7382afaf3987\") " pod="openshift-multus/multus-rf7qp" Nov 02 06:15:39.815189 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.814941 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a2ef8269-c442-42a8-8041-6c1b6aaf0746-run-systemd\") pod \"ovnkube-node-7wb4x\" (UID: \"a2ef8269-c442-42a8-8041-6c1b6aaf0746\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wb4x" Nov 02 06:15:39.815189 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.814964 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a2ef8269-c442-42a8-8041-6c1b6aaf0746-ovn-node-metrics-cert\") pod \"ovnkube-node-7wb4x\" (UID: \"a2ef8269-c442-42a8-8041-6c1b6aaf0746\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wb4x" Nov 02 06:15:39.815189 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.814982 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/81b393f2-42de-4a99-8114-1445de4d6dc8-etc-systemd\") pod \"tuned-6nk8l\" (UID: \"81b393f2-42de-4a99-8114-1445de4d6dc8\") " pod="openshift-cluster-node-tuning-operator/tuned-6nk8l" Nov 02 06:15:39.815189 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.815014 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/81b393f2-42de-4a99-8114-1445de4d6dc8-sys\") pod \"tuned-6nk8l\" (UID: \"81b393f2-42de-4a99-8114-1445de4d6dc8\") " pod="openshift-cluster-node-tuning-operator/tuned-6nk8l" Nov 02 06:15:39.815189 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.815043 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/81b393f2-42de-4a99-8114-1445de4d6dc8-etc-tuned\") pod \"tuned-6nk8l\" (UID: \"81b393f2-42de-4a99-8114-1445de4d6dc8\") " pod="openshift-cluster-node-tuning-operator/tuned-6nk8l" Nov 02 06:15:39.815189 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.815079 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8vwr\" (UniqueName: \"kubernetes.io/projected/3eec1082-df0b-420b-b4ff-5a1c0fab1d1d-kube-api-access-k8vwr\") pod \"node-ca-qm6wn\" (UID: \"3eec1082-df0b-420b-b4ff-5a1c0fab1d1d\") " pod="openshift-image-registry/node-ca-qm6wn" Nov 02 06:15:39.815661 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.815103 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a2ef8269-c442-42a8-8041-6c1b6aaf0746-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7wb4x\" (UID: \"a2ef8269-c442-42a8-8041-6c1b6aaf0746\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wb4x" Nov 02 06:15:39.815661 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.815129 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b6432a8-b0ce-4954-8e3d-8eb8782557eb-metrics-certs\") pod \"network-metrics-daemon-sb4nt\" (UID: \"3b6432a8-b0ce-4954-8e3d-8eb8782557eb\") " pod="openshift-multus/network-metrics-daemon-sb4nt" Nov 02 06:15:39.815661 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.815151 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a2ef8269-c442-42a8-8041-6c1b6aaf0746-host-cni-netd\") pod \"ovnkube-node-7wb4x\" (UID: \"a2ef8269-c442-42a8-8041-6c1b6aaf0746\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wb4x" Nov 02 06:15:39.815661 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.815174 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a2ef8269-c442-42a8-8041-6c1b6aaf0746-ovnkube-script-lib\") pod \"ovnkube-node-7wb4x\" (UID: \"a2ef8269-c442-42a8-8041-6c1b6aaf0746\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wb4x" Nov 02 06:15:39.815661 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.815203 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3500e682-2f15-45a2-92d6-bbc35dae3231-socket-dir\") pod \"aws-ebs-csi-driver-node-cbzvd\" (UID: \"3500e682-2f15-45a2-92d6-bbc35dae3231\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cbzvd" Nov 02 06:15:39.815661 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.815241 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/81b393f2-42de-4a99-8114-1445de4d6dc8-tmp\") pod \"tuned-6nk8l\" (UID: \"81b393f2-42de-4a99-8114-1445de4d6dc8\") " pod="openshift-cluster-node-tuning-operator/tuned-6nk8l" Nov 02 06:15:39.815661 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.815265 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/87e797a0-1da6-41ca-af8d-7382afaf3987-system-cni-dir\") pod \"multus-rf7qp\" (UID: \"87e797a0-1da6-41ca-af8d-7382afaf3987\") " pod="openshift-multus/multus-rf7qp" Nov 02 06:15:39.815661 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.815287 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/87e797a0-1da6-41ca-af8d-7382afaf3987-host-run-netns\") pod \"multus-rf7qp\" (UID: \"87e797a0-1da6-41ca-af8d-7382afaf3987\") " pod="openshift-multus/multus-rf7qp" Nov 02 06:15:39.815661 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.815310 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3500e682-2f15-45a2-92d6-bbc35dae3231-sys-fs\") pod \"aws-ebs-csi-driver-node-cbzvd\" (UID: \"3500e682-2f15-45a2-92d6-bbc35dae3231\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cbzvd" Nov 02 06:15:39.815661 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.815334 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/87e797a0-1da6-41ca-af8d-7382afaf3987-multus-conf-dir\") pod \"multus-rf7qp\" (UID: \"87e797a0-1da6-41ca-af8d-7382afaf3987\") " pod="openshift-multus/multus-rf7qp" Nov 02 06:15:39.815661 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.815356 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d4615a9e-da82-4f8b-8225-d60836b1ba80-konnectivity-ca\") pod \"konnectivity-agent-hwc99\" (UID: \"d4615a9e-da82-4f8b-8225-d60836b1ba80\") " pod="kube-system/konnectivity-agent-hwc99" Nov 02 06:15:39.916068 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.916032 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a2ef8269-c442-42a8-8041-6c1b6aaf0746-env-overrides\") pod \"ovnkube-node-7wb4x\" (UID: \"a2ef8269-c442-42a8-8041-6c1b6aaf0746\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wb4x" Nov 02 06:15:39.916068 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.916067 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d8d87317-3f41-46d4-a58e-3c7601e0d445-hosts-file\") pod \"node-resolver-2p95f\" (UID: \"d8d87317-3f41-46d4-a58e-3c7601e0d445\") " pod="openshift-dns/node-resolver-2p95f" Nov 02 06:15:39.916270 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.916082 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/81b393f2-42de-4a99-8114-1445de4d6dc8-etc-sysctl-d\") pod \"tuned-6nk8l\" (UID: \"81b393f2-42de-4a99-8114-1445de4d6dc8\") " pod="openshift-cluster-node-tuning-operator/tuned-6nk8l" Nov 02 06:15:39.916270 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.916099 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/81b393f2-42de-4a99-8114-1445de4d6dc8-lib-modules\") pod \"tuned-6nk8l\" (UID: \"81b393f2-42de-4a99-8114-1445de4d6dc8\") " pod="openshift-cluster-node-tuning-operator/tuned-6nk8l" Nov 02 06:15:39.916270 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.916159 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d8d87317-3f41-46d4-a58e-3c7601e0d445-hosts-file\") pod \"node-resolver-2p95f\" (UID: \"d8d87317-3f41-46d4-a58e-3c7601e0d445\") " pod="openshift-dns/node-resolver-2p95f" Nov 02 06:15:39.916270 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.916214 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/87e797a0-1da6-41ca-af8d-7382afaf3987-host-var-lib-cni-multus\") pod \"multus-rf7qp\" (UID: \"87e797a0-1da6-41ca-af8d-7382afaf3987\") " pod="openshift-multus/multus-rf7qp" Nov 02 06:15:39.916270 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.916226 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/81b393f2-42de-4a99-8114-1445de4d6dc8-etc-sysctl-d\") pod \"tuned-6nk8l\" (UID: \"81b393f2-42de-4a99-8114-1445de4d6dc8\") " pod="openshift-cluster-node-tuning-operator/tuned-6nk8l" Nov 02 06:15:39.916270 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.916243 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/81b393f2-42de-4a99-8114-1445de4d6dc8-lib-modules\") pod \"tuned-6nk8l\" (UID: \"81b393f2-42de-4a99-8114-1445de4d6dc8\") " pod="openshift-cluster-node-tuning-operator/tuned-6nk8l" Nov 02 06:15:39.916270 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.916245 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/87e797a0-1da6-41ca-af8d-7382afaf3987-etc-kubernetes\") pod \"multus-rf7qp\" (UID: \"87e797a0-1da6-41ca-af8d-7382afaf3987\") " pod="openshift-multus/multus-rf7qp" Nov 02 06:15:39.916270 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.916264 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/87e797a0-1da6-41ca-af8d-7382afaf3987-host-var-lib-cni-multus\") pod \"multus-rf7qp\" (UID: \"87e797a0-1da6-41ca-af8d-7382afaf3987\") " pod="openshift-multus/multus-rf7qp" Nov 02 06:15:39.916270 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.916269 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a2ef8269-c442-42a8-8041-6c1b6aaf0746-run-systemd\") pod \"ovnkube-node-7wb4x\" (UID: \"a2ef8269-c442-42a8-8041-6c1b6aaf0746\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wb4x" Nov 02 06:15:39.916555 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.916294 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/87e797a0-1da6-41ca-af8d-7382afaf3987-etc-kubernetes\") pod \"multus-rf7qp\" (UID: \"87e797a0-1da6-41ca-af8d-7382afaf3987\") " pod="openshift-multus/multus-rf7qp" Nov 02 06:15:39.916555 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.916295 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a2ef8269-c442-42a8-8041-6c1b6aaf0746-ovn-node-metrics-cert\") pod \"ovnkube-node-7wb4x\" (UID: \"a2ef8269-c442-42a8-8041-6c1b6aaf0746\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wb4x" Nov 02 06:15:39.916555 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.916330 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a2ef8269-c442-42a8-8041-6c1b6aaf0746-run-systemd\") pod \"ovnkube-node-7wb4x\" (UID: \"a2ef8269-c442-42a8-8041-6c1b6aaf0746\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wb4x" Nov 02 06:15:39.916555 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.916347 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/81b393f2-42de-4a99-8114-1445de4d6dc8-etc-systemd\") pod \"tuned-6nk8l\" (UID: \"81b393f2-42de-4a99-8114-1445de4d6dc8\") " pod="openshift-cluster-node-tuning-operator/tuned-6nk8l" Nov 02 06:15:39.916555 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.916361 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/81b393f2-42de-4a99-8114-1445de4d6dc8-sys\") pod \"tuned-6nk8l\" (UID: \"81b393f2-42de-4a99-8114-1445de4d6dc8\") " pod="openshift-cluster-node-tuning-operator/tuned-6nk8l" Nov 02 06:15:39.916555 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.916375 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/81b393f2-42de-4a99-8114-1445de4d6dc8-etc-tuned\") pod \"tuned-6nk8l\" (UID: \"81b393f2-42de-4a99-8114-1445de4d6dc8\") " pod="openshift-cluster-node-tuning-operator/tuned-6nk8l" Nov 02 06:15:39.916555 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.916405 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/81b393f2-42de-4a99-8114-1445de4d6dc8-etc-systemd\") pod \"tuned-6nk8l\" (UID: \"81b393f2-42de-4a99-8114-1445de4d6dc8\") " pod="openshift-cluster-node-tuning-operator/tuned-6nk8l" Nov 02 06:15:39.916555 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.916432 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8vwr\" (UniqueName: \"kubernetes.io/projected/3eec1082-df0b-420b-b4ff-5a1c0fab1d1d-kube-api-access-k8vwr\") pod \"node-ca-qm6wn\" (UID: \"3eec1082-df0b-420b-b4ff-5a1c0fab1d1d\") " pod="openshift-image-registry/node-ca-qm6wn" Nov 02 06:15:39.916555 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.916437 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/81b393f2-42de-4a99-8114-1445de4d6dc8-sys\") pod \"tuned-6nk8l\" (UID: \"81b393f2-42de-4a99-8114-1445de4d6dc8\") " pod="openshift-cluster-node-tuning-operator/tuned-6nk8l" Nov 02 06:15:39.916555 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.916459 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a2ef8269-c442-42a8-8041-6c1b6aaf0746-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7wb4x\" (UID: \"a2ef8269-c442-42a8-8041-6c1b6aaf0746\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wb4x" Nov 02 06:15:39.916555 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.916476 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b6432a8-b0ce-4954-8e3d-8eb8782557eb-metrics-certs\") pod \"network-metrics-daemon-sb4nt\" (UID: \"3b6432a8-b0ce-4954-8e3d-8eb8782557eb\") " pod="openshift-multus/network-metrics-daemon-sb4nt" Nov 02 06:15:39.916555 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.916491 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a2ef8269-c442-42a8-8041-6c1b6aaf0746-host-cni-netd\") pod \"ovnkube-node-7wb4x\" (UID: \"a2ef8269-c442-42a8-8041-6c1b6aaf0746\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wb4x" Nov 02 06:15:39.916555 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.916507 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a2ef8269-c442-42a8-8041-6c1b6aaf0746-ovnkube-script-lib\") pod \"ovnkube-node-7wb4x\" (UID: \"a2ef8269-c442-42a8-8041-6c1b6aaf0746\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wb4x" Nov 02 06:15:39.916555 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.916531 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3500e682-2f15-45a2-92d6-bbc35dae3231-socket-dir\") pod \"aws-ebs-csi-driver-node-cbzvd\" (UID: \"3500e682-2f15-45a2-92d6-bbc35dae3231\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cbzvd" Nov 02 06:15:39.916555 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.916540 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a2ef8269-c442-42a8-8041-6c1b6aaf0746-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7wb4x\" (UID: \"a2ef8269-c442-42a8-8041-6c1b6aaf0746\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wb4x" Nov 02 06:15:39.916555 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.916553 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/81b393f2-42de-4a99-8114-1445de4d6dc8-tmp\") pod \"tuned-6nk8l\" (UID: \"81b393f2-42de-4a99-8114-1445de4d6dc8\") " pod="openshift-cluster-node-tuning-operator/tuned-6nk8l" Nov 02 06:15:39.917273 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.916576 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/87e797a0-1da6-41ca-af8d-7382afaf3987-system-cni-dir\") pod \"multus-rf7qp\" (UID: \"87e797a0-1da6-41ca-af8d-7382afaf3987\") " pod="openshift-multus/multus-rf7qp" Nov 02 06:15:39.917273 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.916578 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a2ef8269-c442-42a8-8041-6c1b6aaf0746-host-cni-netd\") pod \"ovnkube-node-7wb4x\" (UID: \"a2ef8269-c442-42a8-8041-6c1b6aaf0746\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wb4x" Nov 02 06:15:39.917273 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.916589 2579 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 02 06:15:39.917273 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.916601 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/87e797a0-1da6-41ca-af8d-7382afaf3987-host-run-netns\") pod \"multus-rf7qp\" (UID: \"87e797a0-1da6-41ca-af8d-7382afaf3987\") " pod="openshift-multus/multus-rf7qp" Nov 02 06:15:39.917273 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:15:39.916623 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 06:15:39.917273 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.916627 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3500e682-2f15-45a2-92d6-bbc35dae3231-sys-fs\") pod \"aws-ebs-csi-driver-node-cbzvd\" (UID: \"3500e682-2f15-45a2-92d6-bbc35dae3231\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cbzvd" Nov 02 06:15:39.917273 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.916648 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/87e797a0-1da6-41ca-af8d-7382afaf3987-multus-conf-dir\") pod \"multus-rf7qp\" (UID: \"87e797a0-1da6-41ca-af8d-7382afaf3987\") " pod="openshift-multus/multus-rf7qp" Nov 02 06:15:39.917273 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.916653 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/87e797a0-1da6-41ca-af8d-7382afaf3987-system-cni-dir\") pod \"multus-rf7qp\" (UID: \"87e797a0-1da6-41ca-af8d-7382afaf3987\") " pod="openshift-multus/multus-rf7qp" Nov 02 06:15:39.917273 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.916594 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a2ef8269-c442-42a8-8041-6c1b6aaf0746-env-overrides\") pod \"ovnkube-node-7wb4x\" (UID: \"a2ef8269-c442-42a8-8041-6c1b6aaf0746\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wb4x" Nov 02 06:15:39.917273 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.916684 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d4615a9e-da82-4f8b-8225-d60836b1ba80-konnectivity-ca\") pod \"konnectivity-agent-hwc99\" (UID: \"d4615a9e-da82-4f8b-8225-d60836b1ba80\") " pod="kube-system/konnectivity-agent-hwc99" Nov 02 06:15:39.917273 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.916706 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/87e797a0-1da6-41ca-af8d-7382afaf3987-host-run-netns\") pod \"multus-rf7qp\" (UID: \"87e797a0-1da6-41ca-af8d-7382afaf3987\") " pod="openshift-multus/multus-rf7qp" Nov 02 06:15:39.917273 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.916717 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/87e797a0-1da6-41ca-af8d-7382afaf3987-multus-conf-dir\") pod \"multus-rf7qp\" (UID: \"87e797a0-1da6-41ca-af8d-7382afaf3987\") " pod="openshift-multus/multus-rf7qp" Nov 02 06:15:39.917273 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.916709 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f14e8781-bf82-4810-9909-c126ccdc1ce1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ckddw\" (UID: \"f14e8781-bf82-4810-9909-c126ccdc1ce1\") " pod="openshift-multus/multus-additional-cni-plugins-ckddw" Nov 02 06:15:39.917273 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.916762 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3500e682-2f15-45a2-92d6-bbc35dae3231-socket-dir\") pod \"aws-ebs-csi-driver-node-cbzvd\" (UID: \"3500e682-2f15-45a2-92d6-bbc35dae3231\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cbzvd" Nov 02 06:15:39.917273 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:15:39.916776 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b6432a8-b0ce-4954-8e3d-8eb8782557eb-metrics-certs podName:3b6432a8-b0ce-4954-8e3d-8eb8782557eb nodeName:}" failed. No retries permitted until 2026-11-02 06:15:40.416750825 +0000 UTC m=+2.308569071 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3b6432a8-b0ce-4954-8e3d-8eb8782557eb-metrics-certs") pod "network-metrics-daemon-sb4nt" (UID: "3b6432a8-b0ce-4954-8e3d-8eb8782557eb") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 06:15:39.917273 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.916799 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a2ef8269-c442-42a8-8041-6c1b6aaf0746-log-socket\") pod \"ovnkube-node-7wb4x\" (UID: \"a2ef8269-c442-42a8-8041-6c1b6aaf0746\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wb4x" Nov 02 06:15:39.917273 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.916827 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8lqt\" (UniqueName: \"kubernetes.io/projected/3500e682-2f15-45a2-92d6-bbc35dae3231-kube-api-access-v8lqt\") pod \"aws-ebs-csi-driver-node-cbzvd\" (UID: \"3500e682-2f15-45a2-92d6-bbc35dae3231\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cbzvd" Nov 02 06:15:39.917273 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.916852 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b65bb247-83f7-4749-9aa1-9b2677dd1b3d-dbus\") pod \"global-pull-secret-syncer-ggpsr\" (UID: \"b65bb247-83f7-4749-9aa1-9b2677dd1b3d\") " pod="kube-system/global-pull-secret-syncer-ggpsr" Nov 02 06:15:39.918049 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.916857 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3500e682-2f15-45a2-92d6-bbc35dae3231-sys-fs\") pod \"aws-ebs-csi-driver-node-cbzvd\" (UID: \"3500e682-2f15-45a2-92d6-bbc35dae3231\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cbzvd" Nov 02 06:15:39.918049 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.916873 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/87e797a0-1da6-41ca-af8d-7382afaf3987-cni-binary-copy\") pod \"multus-rf7qp\" (UID: \"87e797a0-1da6-41ca-af8d-7382afaf3987\") " pod="openshift-multus/multus-rf7qp" Nov 02 06:15:39.918049 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.917045 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a2ef8269-c442-42a8-8041-6c1b6aaf0746-ovnkube-script-lib\") pod \"ovnkube-node-7wb4x\" (UID: \"a2ef8269-c442-42a8-8041-6c1b6aaf0746\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wb4x" Nov 02 06:15:39.918049 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.917102 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a2ef8269-c442-42a8-8041-6c1b6aaf0746-log-socket\") pod \"ovnkube-node-7wb4x\" (UID: \"a2ef8269-c442-42a8-8041-6c1b6aaf0746\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wb4x" Nov 02 06:15:39.918049 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.917135 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3419bd74-22aa-4841-87d2-c4340e01dae9-host-slash\") pod \"iptables-alerter-6fnx8\" (UID: \"3419bd74-22aa-4841-87d2-c4340e01dae9\") " pod="openshift-network-operator/iptables-alerter-6fnx8" Nov 02 06:15:39.918049 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.917162 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a2ef8269-c442-42a8-8041-6c1b6aaf0746-host-slash\") pod \"ovnkube-node-7wb4x\" (UID: \"a2ef8269-c442-42a8-8041-6c1b6aaf0746\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wb4x" Nov 02 06:15:39.918049 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.917185 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a2ef8269-c442-42a8-8041-6c1b6aaf0746-host-run-netns\") pod \"ovnkube-node-7wb4x\" (UID: \"a2ef8269-c442-42a8-8041-6c1b6aaf0746\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wb4x" Nov 02 06:15:39.918049 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.917208 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a2ef8269-c442-42a8-8041-6c1b6aaf0746-var-lib-openvswitch\") pod \"ovnkube-node-7wb4x\" (UID: \"a2ef8269-c442-42a8-8041-6c1b6aaf0746\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wb4x" Nov 02 06:15:39.918049 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.917228 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a2ef8269-c442-42a8-8041-6c1b6aaf0746-host-slash\") pod \"ovnkube-node-7wb4x\" (UID: \"a2ef8269-c442-42a8-8041-6c1b6aaf0746\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wb4x" Nov 02 06:15:39.918049 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.917231 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a2ef8269-c442-42a8-8041-6c1b6aaf0746-node-log\") pod \"ovnkube-node-7wb4x\" (UID: \"a2ef8269-c442-42a8-8041-6c1b6aaf0746\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wb4x" Nov 02 06:15:39.918049 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.917268 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a2ef8269-c442-42a8-8041-6c1b6aaf0746-node-log\") pod \"ovnkube-node-7wb4x\" (UID: \"a2ef8269-c442-42a8-8041-6c1b6aaf0746\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wb4x" Nov 02 06:15:39.918049 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.917297 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a2ef8269-c442-42a8-8041-6c1b6aaf0746-var-lib-openvswitch\") pod \"ovnkube-node-7wb4x\" (UID: \"a2ef8269-c442-42a8-8041-6c1b6aaf0746\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wb4x" Nov 02 06:15:39.918049 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.917298 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f14e8781-bf82-4810-9909-c126ccdc1ce1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ckddw\" (UID: \"f14e8781-bf82-4810-9909-c126ccdc1ce1\") " pod="openshift-multus/multus-additional-cni-plugins-ckddw" Nov 02 06:15:39.918049 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.917299 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3419bd74-22aa-4841-87d2-c4340e01dae9-host-slash\") pod \"iptables-alerter-6fnx8\" (UID: \"3419bd74-22aa-4841-87d2-c4340e01dae9\") " pod="openshift-network-operator/iptables-alerter-6fnx8" Nov 02 06:15:39.918049 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.917269 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzxsj\" (UniqueName: \"kubernetes.io/projected/a2ef8269-c442-42a8-8041-6c1b6aaf0746-kube-api-access-qzxsj\") pod \"ovnkube-node-7wb4x\" (UID: \"a2ef8269-c442-42a8-8041-6c1b6aaf0746\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wb4x" Nov 02 06:15:39.918049 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.917208 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b65bb247-83f7-4749-9aa1-9b2677dd1b3d-dbus\") pod \"global-pull-secret-syncer-ggpsr\" (UID: \"b65bb247-83f7-4749-9aa1-9b2677dd1b3d\") " pod="kube-system/global-pull-secret-syncer-ggpsr" Nov 02 06:15:39.918049 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.917326 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a2ef8269-c442-42a8-8041-6c1b6aaf0746-host-run-netns\") pod \"ovnkube-node-7wb4x\" (UID: \"a2ef8269-c442-42a8-8041-6c1b6aaf0746\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wb4x" Nov 02 06:15:39.918049 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.917298 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d4615a9e-da82-4f8b-8225-d60836b1ba80-konnectivity-ca\") pod \"konnectivity-agent-hwc99\" (UID: \"d4615a9e-da82-4f8b-8225-d60836b1ba80\") " pod="kube-system/konnectivity-agent-hwc99" Nov 02 06:15:39.918743 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.917340 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8d87317-3f41-46d4-a58e-3c7601e0d445-tmp-dir\") pod \"node-resolver-2p95f\" (UID: \"d8d87317-3f41-46d4-a58e-3c7601e0d445\") " pod="openshift-dns/node-resolver-2p95f" Nov 02 06:15:39.918743 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.917370 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3500e682-2f15-45a2-92d6-bbc35dae3231-kubelet-dir\") pod \"aws-ebs-csi-driver-node-cbzvd\" (UID: \"3500e682-2f15-45a2-92d6-bbc35dae3231\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cbzvd" Nov 02 06:15:39.918743 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.917395 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f14e8781-bf82-4810-9909-c126ccdc1ce1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ckddw\" (UID: \"f14e8781-bf82-4810-9909-c126ccdc1ce1\") " pod="openshift-multus/multus-additional-cni-plugins-ckddw" Nov 02 06:15:39.918743 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.917398 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/87e797a0-1da6-41ca-af8d-7382afaf3987-cni-binary-copy\") pod \"multus-rf7qp\" (UID: \"87e797a0-1da6-41ca-af8d-7382afaf3987\") " pod="openshift-multus/multus-rf7qp" Nov 02 06:15:39.918743 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.917418 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3419bd74-22aa-4841-87d2-c4340e01dae9-iptables-alerter-script\") pod \"iptables-alerter-6fnx8\" (UID: \"3419bd74-22aa-4841-87d2-c4340e01dae9\") " pod="openshift-network-operator/iptables-alerter-6fnx8" Nov 02 06:15:39.918743 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.917444 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3500e682-2f15-45a2-92d6-bbc35dae3231-kubelet-dir\") pod \"aws-ebs-csi-driver-node-cbzvd\" (UID: \"3500e682-2f15-45a2-92d6-bbc35dae3231\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cbzvd" Nov 02 06:15:39.918743 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.917475 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3500e682-2f15-45a2-92d6-bbc35dae3231-device-dir\") pod \"aws-ebs-csi-driver-node-cbzvd\" (UID: \"3500e682-2f15-45a2-92d6-bbc35dae3231\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cbzvd" Nov 02 06:15:39.918743 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.917499 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/81b393f2-42de-4a99-8114-1445de4d6dc8-etc-sysctl-conf\") pod \"tuned-6nk8l\" (UID: \"81b393f2-42de-4a99-8114-1445de4d6dc8\") " pod="openshift-cluster-node-tuning-operator/tuned-6nk8l" Nov 02 06:15:39.918743 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.917523 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/87e797a0-1da6-41ca-af8d-7382afaf3987-multus-socket-dir-parent\") pod \"multus-rf7qp\" (UID: \"87e797a0-1da6-41ca-af8d-7382afaf3987\") " pod="openshift-multus/multus-rf7qp" Nov 02 06:15:39.918743 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.917529 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3500e682-2f15-45a2-92d6-bbc35dae3231-device-dir\") pod \"aws-ebs-csi-driver-node-cbzvd\" (UID: \"3500e682-2f15-45a2-92d6-bbc35dae3231\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cbzvd" Nov 02 06:15:39.918743 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.917546 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/87e797a0-1da6-41ca-af8d-7382afaf3987-hostroot\") pod \"multus-rf7qp\" (UID: \"87e797a0-1da6-41ca-af8d-7382afaf3987\") " pod="openshift-multus/multus-rf7qp" Nov 02 06:15:39.918743 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.917567 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/87e797a0-1da6-41ca-af8d-7382afaf3987-multus-daemon-config\") pod \"multus-rf7qp\" (UID: \"87e797a0-1da6-41ca-af8d-7382afaf3987\") " pod="openshift-multus/multus-rf7qp" Nov 02 06:15:39.918743 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.917581 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8d87317-3f41-46d4-a58e-3c7601e0d445-tmp-dir\") pod \"node-resolver-2p95f\" (UID: \"d8d87317-3f41-46d4-a58e-3c7601e0d445\") " pod="openshift-dns/node-resolver-2p95f" Nov 02 06:15:39.918743 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.917591 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d4615a9e-da82-4f8b-8225-d60836b1ba80-agent-certs\") pod \"konnectivity-agent-hwc99\" (UID: \"d4615a9e-da82-4f8b-8225-d60836b1ba80\") " pod="kube-system/konnectivity-agent-hwc99" Nov 02 06:15:39.918743 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.917615 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a2ef8269-c442-42a8-8041-6c1b6aaf0746-host-kubelet\") pod \"ovnkube-node-7wb4x\" (UID: \"a2ef8269-c442-42a8-8041-6c1b6aaf0746\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wb4x" Nov 02 06:15:39.918743 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.917634 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/81b393f2-42de-4a99-8114-1445de4d6dc8-etc-sysctl-conf\") pod \"tuned-6nk8l\" (UID: \"81b393f2-42de-4a99-8114-1445de4d6dc8\") " pod="openshift-cluster-node-tuning-operator/tuned-6nk8l" Nov 02 06:15:39.918743 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.917645 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kcrm\" (UniqueName: \"kubernetes.io/projected/d8d87317-3f41-46d4-a58e-3c7601e0d445-kube-api-access-5kcrm\") pod \"node-resolver-2p95f\" (UID: \"d8d87317-3f41-46d4-a58e-3c7601e0d445\") " pod="openshift-dns/node-resolver-2p95f" Nov 02 06:15:39.919550 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.917649 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/87e797a0-1da6-41ca-af8d-7382afaf3987-multus-socket-dir-parent\") pod \"multus-rf7qp\" (UID: \"87e797a0-1da6-41ca-af8d-7382afaf3987\") " pod="openshift-multus/multus-rf7qp" Nov 02 06:15:39.919550 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.917668 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a2ef8269-c442-42a8-8041-6c1b6aaf0746-host-kubelet\") pod \"ovnkube-node-7wb4x\" (UID: \"a2ef8269-c442-42a8-8041-6c1b6aaf0746\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wb4x" Nov 02 06:15:39.919550 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.917670 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrq9j\" (UniqueName: \"kubernetes.io/projected/81b393f2-42de-4a99-8114-1445de4d6dc8-kube-api-access-hrq9j\") pod \"tuned-6nk8l\" (UID: \"81b393f2-42de-4a99-8114-1445de4d6dc8\") " pod="openshift-cluster-node-tuning-operator/tuned-6nk8l" Nov 02 06:15:39.919550 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.917696 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b65bb247-83f7-4749-9aa1-9b2677dd1b3d-kubelet-config\") pod \"global-pull-secret-syncer-ggpsr\" (UID: \"b65bb247-83f7-4749-9aa1-9b2677dd1b3d\") " pod="kube-system/global-pull-secret-syncer-ggpsr" Nov 02 06:15:39.919550 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.917720 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvdp9\" (UniqueName: \"kubernetes.io/projected/3b6432a8-b0ce-4954-8e3d-8eb8782557eb-kube-api-access-pvdp9\") pod \"network-metrics-daemon-sb4nt\" (UID: \"3b6432a8-b0ce-4954-8e3d-8eb8782557eb\") " pod="openshift-multus/network-metrics-daemon-sb4nt" Nov 02 06:15:39.919550 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.917763 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vz9mf\" (UniqueName: \"kubernetes.io/projected/3419bd74-22aa-4841-87d2-c4340e01dae9-kube-api-access-vz9mf\") pod \"iptables-alerter-6fnx8\" (UID: \"3419bd74-22aa-4841-87d2-c4340e01dae9\") " pod="openshift-network-operator/iptables-alerter-6fnx8" Nov 02 06:15:39.919550 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.917803 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3500e682-2f15-45a2-92d6-bbc35dae3231-registration-dir\") pod \"aws-ebs-csi-driver-node-cbzvd\" (UID: \"3500e682-2f15-45a2-92d6-bbc35dae3231\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cbzvd" Nov 02 06:15:39.919550 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.917828 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3eec1082-df0b-420b-b4ff-5a1c0fab1d1d-serviceca\") pod \"node-ca-qm6wn\" (UID: \"3eec1082-df0b-420b-b4ff-5a1c0fab1d1d\") " pod="openshift-image-registry/node-ca-qm6wn" Nov 02 06:15:39.919550 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.917588 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f14e8781-bf82-4810-9909-c126ccdc1ce1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ckddw\" (UID: \"f14e8781-bf82-4810-9909-c126ccdc1ce1\") " pod="openshift-multus/multus-additional-cni-plugins-ckddw" Nov 02 06:15:39.919550 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.917867 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/87e797a0-1da6-41ca-af8d-7382afaf3987-host-var-lib-kubelet\") pod \"multus-rf7qp\" (UID: \"87e797a0-1da6-41ca-af8d-7382afaf3987\") " pod="openshift-multus/multus-rf7qp" Nov 02 06:15:39.919550 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.917917 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnq6k\" (UniqueName: \"kubernetes.io/projected/3ced47b0-9cd0-4710-8f75-2a333437ed01-kube-api-access-gnq6k\") pod \"network-check-target-ft7gs\" (UID: \"3ced47b0-9cd0-4710-8f75-2a333437ed01\") " pod="openshift-network-diagnostics/network-check-target-ft7gs" Nov 02 06:15:39.919550 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.917944 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a2ef8269-c442-42a8-8041-6c1b6aaf0746-host-run-ovn-kubernetes\") pod \"ovnkube-node-7wb4x\" (UID: \"a2ef8269-c442-42a8-8041-6c1b6aaf0746\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wb4x" Nov 02 06:15:39.919550 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.917970 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f14e8781-bf82-4810-9909-c126ccdc1ce1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ckddw\" (UID: \"f14e8781-bf82-4810-9909-c126ccdc1ce1\") " pod="openshift-multus/multus-additional-cni-plugins-ckddw" Nov 02 06:15:39.919550 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.917998 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7xzh\" (UniqueName: \"kubernetes.io/projected/f14e8781-bf82-4810-9909-c126ccdc1ce1-kube-api-access-v7xzh\") pod \"multus-additional-cni-plugins-ckddw\" (UID: \"f14e8781-bf82-4810-9909-c126ccdc1ce1\") " pod="openshift-multus/multus-additional-cni-plugins-ckddw" Nov 02 06:15:39.919550 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.918021 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3500e682-2f15-45a2-92d6-bbc35dae3231-etc-selinux\") pod \"aws-ebs-csi-driver-node-cbzvd\" (UID: \"3500e682-2f15-45a2-92d6-bbc35dae3231\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cbzvd" Nov 02 06:15:39.919550 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.918043 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/81b393f2-42de-4a99-8114-1445de4d6dc8-host\") pod \"tuned-6nk8l\" (UID: \"81b393f2-42de-4a99-8114-1445de4d6dc8\") " pod="openshift-cluster-node-tuning-operator/tuned-6nk8l" Nov 02 06:15:39.919550 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.918045 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3419bd74-22aa-4841-87d2-c4340e01dae9-iptables-alerter-script\") pod \"iptables-alerter-6fnx8\" (UID: \"3419bd74-22aa-4841-87d2-c4340e01dae9\") " pod="openshift-network-operator/iptables-alerter-6fnx8" Nov 02 06:15:39.920323 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.918065 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/87e797a0-1da6-41ca-af8d-7382afaf3987-multus-cni-dir\") pod \"multus-rf7qp\" (UID: \"87e797a0-1da6-41ca-af8d-7382afaf3987\") " pod="openshift-multus/multus-rf7qp" Nov 02 06:15:39.920323 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.918091 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a2ef8269-c442-42a8-8041-6c1b6aaf0746-systemd-units\") pod \"ovnkube-node-7wb4x\" (UID: \"a2ef8269-c442-42a8-8041-6c1b6aaf0746\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wb4x" Nov 02 06:15:39.920323 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.918094 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/87e797a0-1da6-41ca-af8d-7382afaf3987-multus-daemon-config\") pod \"multus-rf7qp\" (UID: \"87e797a0-1da6-41ca-af8d-7382afaf3987\") " pod="openshift-multus/multus-rf7qp" Nov 02 06:15:39.920323 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.918117 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a2ef8269-c442-42a8-8041-6c1b6aaf0746-run-ovn\") pod \"ovnkube-node-7wb4x\" (UID: \"a2ef8269-c442-42a8-8041-6c1b6aaf0746\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wb4x" Nov 02 06:15:39.920323 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.918143 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a2ef8269-c442-42a8-8041-6c1b6aaf0746-ovnkube-config\") pod \"ovnkube-node-7wb4x\" (UID: \"a2ef8269-c442-42a8-8041-6c1b6aaf0746\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wb4x" Nov 02 06:15:39.920323 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.918165 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3eec1082-df0b-420b-b4ff-5a1c0fab1d1d-host\") pod \"node-ca-qm6wn\" (UID: \"3eec1082-df0b-420b-b4ff-5a1c0fab1d1d\") " pod="openshift-image-registry/node-ca-qm6wn" Nov 02 06:15:39.920323 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.918189 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/87e797a0-1da6-41ca-af8d-7382afaf3987-cnibin\") pod \"multus-rf7qp\" (UID: \"87e797a0-1da6-41ca-af8d-7382afaf3987\") " pod="openshift-multus/multus-rf7qp" Nov 02 06:15:39.920323 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.918194 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3500e682-2f15-45a2-92d6-bbc35dae3231-etc-selinux\") pod \"aws-ebs-csi-driver-node-cbzvd\" (UID: \"3500e682-2f15-45a2-92d6-bbc35dae3231\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cbzvd" Nov 02 06:15:39.920323 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.918661 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/81b393f2-42de-4a99-8114-1445de4d6dc8-etc-modprobe-d\") pod \"tuned-6nk8l\" (UID: \"81b393f2-42de-4a99-8114-1445de4d6dc8\") " pod="openshift-cluster-node-tuning-operator/tuned-6nk8l" Nov 02 06:15:39.920323 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.918720 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b65bb247-83f7-4749-9aa1-9b2677dd1b3d-original-pull-secret\") pod \"global-pull-secret-syncer-ggpsr\" (UID: \"b65bb247-83f7-4749-9aa1-9b2677dd1b3d\") " pod="kube-system/global-pull-secret-syncer-ggpsr" Nov 02 06:15:39.920323 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.918749 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/87e797a0-1da6-41ca-af8d-7382afaf3987-cnibin\") pod \"multus-rf7qp\" (UID: \"87e797a0-1da6-41ca-af8d-7382afaf3987\") " pod="openshift-multus/multus-rf7qp" Nov 02 06:15:39.920323 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.918802 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/87e797a0-1da6-41ca-af8d-7382afaf3987-host-var-lib-cni-bin\") pod \"multus-rf7qp\" (UID: \"87e797a0-1da6-41ca-af8d-7382afaf3987\") " pod="openshift-multus/multus-rf7qp" Nov 02 06:15:39.920323 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.918922 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/81b393f2-42de-4a99-8114-1445de4d6dc8-etc-modprobe-d\") pod \"tuned-6nk8l\" (UID: \"81b393f2-42de-4a99-8114-1445de4d6dc8\") " pod="openshift-cluster-node-tuning-operator/tuned-6nk8l" Nov 02 06:15:39.920323 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:15:39.919020 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 06:15:39.920323 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.919025 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3eec1082-df0b-420b-b4ff-5a1c0fab1d1d-serviceca\") pod \"node-ca-qm6wn\" (UID: \"3eec1082-df0b-420b-b4ff-5a1c0fab1d1d\") " pod="openshift-image-registry/node-ca-qm6wn" Nov 02 06:15:39.920323 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:15:39.919080 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b65bb247-83f7-4749-9aa1-9b2677dd1b3d-original-pull-secret podName:b65bb247-83f7-4749-9aa1-9b2677dd1b3d nodeName:}" failed. No retries permitted until 2026-11-02 06:15:40.419057479 +0000 UTC m=+2.310875732 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b65bb247-83f7-4749-9aa1-9b2677dd1b3d-original-pull-secret") pod "global-pull-secret-syncer-ggpsr" (UID: "b65bb247-83f7-4749-9aa1-9b2677dd1b3d") : object "kube-system"/"original-pull-secret" not registered Nov 02 06:15:39.920323 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.919145 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/81b393f2-42de-4a99-8114-1445de4d6dc8-host\") pod \"tuned-6nk8l\" (UID: \"81b393f2-42de-4a99-8114-1445de4d6dc8\") " pod="openshift-cluster-node-tuning-operator/tuned-6nk8l" Nov 02 06:15:39.920323 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.919212 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/87e797a0-1da6-41ca-af8d-7382afaf3987-multus-cni-dir\") pod \"multus-rf7qp\" (UID: \"87e797a0-1da6-41ca-af8d-7382afaf3987\") " pod="openshift-multus/multus-rf7qp" Nov 02 06:15:39.921184 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.919263 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a2ef8269-c442-42a8-8041-6c1b6aaf0746-systemd-units\") pod \"ovnkube-node-7wb4x\" (UID: \"a2ef8269-c442-42a8-8041-6c1b6aaf0746\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wb4x" Nov 02 06:15:39.921184 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.919340 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a2ef8269-c442-42a8-8041-6c1b6aaf0746-run-ovn\") pod \"ovnkube-node-7wb4x\" (UID: \"a2ef8269-c442-42a8-8041-6c1b6aaf0746\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wb4x" Nov 02 06:15:39.921184 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.919635 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3500e682-2f15-45a2-92d6-bbc35dae3231-registration-dir\") pod \"aws-ebs-csi-driver-node-cbzvd\" (UID: \"3500e682-2f15-45a2-92d6-bbc35dae3231\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cbzvd" Nov 02 06:15:39.921184 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.920361 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a2ef8269-c442-42a8-8041-6c1b6aaf0746-host-run-ovn-kubernetes\") pod \"ovnkube-node-7wb4x\" (UID: \"a2ef8269-c442-42a8-8041-6c1b6aaf0746\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wb4x" Nov 02 06:15:39.921184 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.920421 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a2ef8269-c442-42a8-8041-6c1b6aaf0746-ovnkube-config\") pod \"ovnkube-node-7wb4x\" (UID: \"a2ef8269-c442-42a8-8041-6c1b6aaf0746\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wb4x" Nov 02 06:15:39.921184 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.920455 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/81b393f2-42de-4a99-8114-1445de4d6dc8-etc-tuned\") pod \"tuned-6nk8l\" (UID: \"81b393f2-42de-4a99-8114-1445de4d6dc8\") " pod="openshift-cluster-node-tuning-operator/tuned-6nk8l" Nov 02 06:15:39.921184 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.920496 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3eec1082-df0b-420b-b4ff-5a1c0fab1d1d-host\") pod \"node-ca-qm6wn\" (UID: \"3eec1082-df0b-420b-b4ff-5a1c0fab1d1d\") " pod="openshift-image-registry/node-ca-qm6wn" Nov 02 06:15:39.921184 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.920495 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/81b393f2-42de-4a99-8114-1445de4d6dc8-tmp\") pod \"tuned-6nk8l\" (UID: \"81b393f2-42de-4a99-8114-1445de4d6dc8\") " pod="openshift-cluster-node-tuning-operator/tuned-6nk8l" Nov 02 06:15:39.921184 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.918755 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/87e797a0-1da6-41ca-af8d-7382afaf3987-host-var-lib-cni-bin\") pod \"multus-rf7qp\" (UID: \"87e797a0-1da6-41ca-af8d-7382afaf3987\") " pod="openshift-multus/multus-rf7qp" Nov 02 06:15:39.921184 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.917617 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/87e797a0-1da6-41ca-af8d-7382afaf3987-hostroot\") pod \"multus-rf7qp\" (UID: \"87e797a0-1da6-41ca-af8d-7382afaf3987\") " pod="openshift-multus/multus-rf7qp" Nov 02 06:15:39.921184 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.920566 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/87e797a0-1da6-41ca-af8d-7382afaf3987-host-run-multus-certs\") pod \"multus-rf7qp\" (UID: \"87e797a0-1da6-41ca-af8d-7382afaf3987\") " pod="openshift-multus/multus-rf7qp" Nov 02 06:15:39.921184 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.920601 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f14e8781-bf82-4810-9909-c126ccdc1ce1-cnibin\") pod \"multus-additional-cni-plugins-ckddw\" (UID: \"f14e8781-bf82-4810-9909-c126ccdc1ce1\") " pod="openshift-multus/multus-additional-cni-plugins-ckddw" Nov 02 06:15:39.921184 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.920636 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a2ef8269-c442-42a8-8041-6c1b6aaf0746-run-openvswitch\") pod \"ovnkube-node-7wb4x\" (UID: \"a2ef8269-c442-42a8-8041-6c1b6aaf0746\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wb4x" Nov 02 06:15:39.921184 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.920664 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/81b393f2-42de-4a99-8114-1445de4d6dc8-etc-kubernetes\") pod \"tuned-6nk8l\" (UID: \"81b393f2-42de-4a99-8114-1445de4d6dc8\") " pod="openshift-cluster-node-tuning-operator/tuned-6nk8l" Nov 02 06:15:39.921184 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.920690 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/81b393f2-42de-4a99-8114-1445de4d6dc8-run\") pod \"tuned-6nk8l\" (UID: \"81b393f2-42de-4a99-8114-1445de4d6dc8\") " pod="openshift-cluster-node-tuning-operator/tuned-6nk8l" Nov 02 06:15:39.921184 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.920721 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/81b393f2-42de-4a99-8114-1445de4d6dc8-var-lib-kubelet\") pod \"tuned-6nk8l\" (UID: \"81b393f2-42de-4a99-8114-1445de4d6dc8\") " pod="openshift-cluster-node-tuning-operator/tuned-6nk8l" Nov 02 06:15:39.921184 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.920727 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b65bb247-83f7-4749-9aa1-9b2677dd1b3d-kubelet-config\") pod \"global-pull-secret-syncer-ggpsr\" (UID: \"b65bb247-83f7-4749-9aa1-9b2677dd1b3d\") " pod="kube-system/global-pull-secret-syncer-ggpsr" Nov 02 06:15:39.921184 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.920740 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f14e8781-bf82-4810-9909-c126ccdc1ce1-cnibin\") pod \"multus-additional-cni-plugins-ckddw\" (UID: \"f14e8781-bf82-4810-9909-c126ccdc1ce1\") " pod="openshift-multus/multus-additional-cni-plugins-ckddw" Nov 02 06:15:39.921926 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.920853 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/81b393f2-42de-4a99-8114-1445de4d6dc8-run\") pod \"tuned-6nk8l\" (UID: \"81b393f2-42de-4a99-8114-1445de4d6dc8\") " pod="openshift-cluster-node-tuning-operator/tuned-6nk8l" Nov 02 06:15:39.921926 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.920862 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/87e797a0-1da6-41ca-af8d-7382afaf3987-host-run-multus-certs\") pod \"multus-rf7qp\" (UID: \"87e797a0-1da6-41ca-af8d-7382afaf3987\") " pod="openshift-multus/multus-rf7qp" Nov 02 06:15:39.921926 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.920867 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/87e797a0-1da6-41ca-af8d-7382afaf3987-host-var-lib-kubelet\") pod \"multus-rf7qp\" (UID: \"87e797a0-1da6-41ca-af8d-7382afaf3987\") " pod="openshift-multus/multus-rf7qp" Nov 02 06:15:39.921926 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.920936 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a2ef8269-c442-42a8-8041-6c1b6aaf0746-run-openvswitch\") pod \"ovnkube-node-7wb4x\" (UID: \"a2ef8269-c442-42a8-8041-6c1b6aaf0746\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wb4x" Nov 02 06:15:39.921926 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.920755 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f14e8781-bf82-4810-9909-c126ccdc1ce1-system-cni-dir\") pod \"multus-additional-cni-plugins-ckddw\" (UID: \"f14e8781-bf82-4810-9909-c126ccdc1ce1\") " pod="openshift-multus/multus-additional-cni-plugins-ckddw" Nov 02 06:15:39.921926 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.920973 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/81b393f2-42de-4a99-8114-1445de4d6dc8-var-lib-kubelet\") pod \"tuned-6nk8l\" (UID: \"81b393f2-42de-4a99-8114-1445de4d6dc8\") " pod="openshift-cluster-node-tuning-operator/tuned-6nk8l" Nov 02 06:15:39.921926 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.921007 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f14e8781-bf82-4810-9909-c126ccdc1ce1-cni-binary-copy\") pod \"multus-additional-cni-plugins-ckddw\" (UID: \"f14e8781-bf82-4810-9909-c126ccdc1ce1\") " pod="openshift-multus/multus-additional-cni-plugins-ckddw" Nov 02 06:15:39.921926 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.921036 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a2ef8269-c442-42a8-8041-6c1b6aaf0746-etc-openvswitch\") pod \"ovnkube-node-7wb4x\" (UID: \"a2ef8269-c442-42a8-8041-6c1b6aaf0746\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wb4x" Nov 02 06:15:39.921926 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.921041 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f14e8781-bf82-4810-9909-c126ccdc1ce1-system-cni-dir\") pod \"multus-additional-cni-plugins-ckddw\" (UID: \"f14e8781-bf82-4810-9909-c126ccdc1ce1\") " pod="openshift-multus/multus-additional-cni-plugins-ckddw" Nov 02 06:15:39.921926 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.921043 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/81b393f2-42de-4a99-8114-1445de4d6dc8-etc-kubernetes\") pod \"tuned-6nk8l\" (UID: \"81b393f2-42de-4a99-8114-1445de4d6dc8\") " pod="openshift-cluster-node-tuning-operator/tuned-6nk8l" Nov 02 06:15:39.921926 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.921068 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a2ef8269-c442-42a8-8041-6c1b6aaf0746-host-cni-bin\") pod \"ovnkube-node-7wb4x\" (UID: \"a2ef8269-c442-42a8-8041-6c1b6aaf0746\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wb4x" Nov 02 06:15:39.921926 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.921100 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a2ef8269-c442-42a8-8041-6c1b6aaf0746-etc-openvswitch\") pod \"ovnkube-node-7wb4x\" (UID: \"a2ef8269-c442-42a8-8041-6c1b6aaf0746\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wb4x" Nov 02 06:15:39.921926 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.921108 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/81b393f2-42de-4a99-8114-1445de4d6dc8-etc-sysconfig\") pod \"tuned-6nk8l\" (UID: \"81b393f2-42de-4a99-8114-1445de4d6dc8\") " pod="openshift-cluster-node-tuning-operator/tuned-6nk8l" Nov 02 06:15:39.921926 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.921116 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a2ef8269-c442-42a8-8041-6c1b6aaf0746-host-cni-bin\") pod \"ovnkube-node-7wb4x\" (UID: \"a2ef8269-c442-42a8-8041-6c1b6aaf0746\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wb4x" Nov 02 06:15:39.921926 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.921144 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/87e797a0-1da6-41ca-af8d-7382afaf3987-host-run-k8s-cni-cncf-io\") pod \"multus-rf7qp\" (UID: \"87e797a0-1da6-41ca-af8d-7382afaf3987\") " pod="openshift-multus/multus-rf7qp" Nov 02 06:15:39.921926 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.921172 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/81b393f2-42de-4a99-8114-1445de4d6dc8-etc-sysconfig\") pod \"tuned-6nk8l\" (UID: \"81b393f2-42de-4a99-8114-1445de4d6dc8\") " pod="openshift-cluster-node-tuning-operator/tuned-6nk8l" Nov 02 06:15:39.921926 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.921176 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6vx8\" (UniqueName: \"kubernetes.io/projected/87e797a0-1da6-41ca-af8d-7382afaf3987-kube-api-access-l6vx8\") pod \"multus-rf7qp\" (UID: \"87e797a0-1da6-41ca-af8d-7382afaf3987\") " pod="openshift-multus/multus-rf7qp" Nov 02 06:15:39.922596 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.921244 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/87e797a0-1da6-41ca-af8d-7382afaf3987-host-run-k8s-cni-cncf-io\") pod \"multus-rf7qp\" (UID: \"87e797a0-1da6-41ca-af8d-7382afaf3987\") " pod="openshift-multus/multus-rf7qp" Nov 02 06:15:39.922596 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.921329 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a2ef8269-c442-42a8-8041-6c1b6aaf0746-ovn-node-metrics-cert\") pod \"ovnkube-node-7wb4x\" (UID: \"a2ef8269-c442-42a8-8041-6c1b6aaf0746\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wb4x" Nov 02 06:15:39.922596 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.921624 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f14e8781-bf82-4810-9909-c126ccdc1ce1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ckddw\" (UID: \"f14e8781-bf82-4810-9909-c126ccdc1ce1\") " pod="openshift-multus/multus-additional-cni-plugins-ckddw" Nov 02 06:15:39.923486 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.923462 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d4615a9e-da82-4f8b-8225-d60836b1ba80-agent-certs\") pod \"konnectivity-agent-hwc99\" (UID: \"d4615a9e-da82-4f8b-8225-d60836b1ba80\") " pod="kube-system/konnectivity-agent-hwc99" Nov 02 06:15:39.923695 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.923677 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f14e8781-bf82-4810-9909-c126ccdc1ce1-cni-binary-copy\") pod \"multus-additional-cni-plugins-ckddw\" (UID: \"f14e8781-bf82-4810-9909-c126ccdc1ce1\") " pod="openshift-multus/multus-additional-cni-plugins-ckddw" Nov 02 06:15:39.969678 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:15:39.969597 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 06:15:39.969678 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:15:39.969632 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 06:15:39.969678 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:15:39.969650 2579 projected.go:196] Error preparing data for projected volume kube-api-access-gnq6k for pod openshift-network-diagnostics/network-check-target-ft7gs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 06:15:39.969961 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:15:39.969737 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3ced47b0-9cd0-4710-8f75-2a333437ed01-kube-api-access-gnq6k podName:3ced47b0-9cd0-4710-8f75-2a333437ed01 nodeName:}" failed. No retries permitted until 2026-11-02 06:15:40.469716674 +0000 UTC m=+2.361534936 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-gnq6k" (UniqueName: "kubernetes.io/projected/3ced47b0-9cd0-4710-8f75-2a333437ed01-kube-api-access-gnq6k") pod "network-check-target-ft7gs" (UID: "3ced47b0-9cd0-4710-8f75-2a333437ed01") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 06:15:39.970379 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.970350 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzxsj\" (UniqueName: \"kubernetes.io/projected/a2ef8269-c442-42a8-8041-6c1b6aaf0746-kube-api-access-qzxsj\") pod \"ovnkube-node-7wb4x\" (UID: \"a2ef8269-c442-42a8-8041-6c1b6aaf0746\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wb4x" Nov 02 06:15:39.970537 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.970492 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8vwr\" (UniqueName: \"kubernetes.io/projected/3eec1082-df0b-420b-b4ff-5a1c0fab1d1d-kube-api-access-k8vwr\") pod \"node-ca-qm6wn\" (UID: \"3eec1082-df0b-420b-b4ff-5a1c0fab1d1d\") " pod="openshift-image-registry/node-ca-qm6wn" Nov 02 06:15:39.970979 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.970923 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8lqt\" (UniqueName: \"kubernetes.io/projected/3500e682-2f15-45a2-92d6-bbc35dae3231-kube-api-access-v8lqt\") pod \"aws-ebs-csi-driver-node-cbzvd\" (UID: \"3500e682-2f15-45a2-92d6-bbc35dae3231\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cbzvd" Nov 02 06:15:39.971402 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.971375 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kcrm\" (UniqueName: \"kubernetes.io/projected/d8d87317-3f41-46d4-a58e-3c7601e0d445-kube-api-access-5kcrm\") pod \"node-resolver-2p95f\" (UID: \"d8d87317-3f41-46d4-a58e-3c7601e0d445\") " pod="openshift-dns/node-resolver-2p95f" Nov 02 06:15:39.971616 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.971594 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvdp9\" (UniqueName: \"kubernetes.io/projected/3b6432a8-b0ce-4954-8e3d-8eb8782557eb-kube-api-access-pvdp9\") pod \"network-metrics-daemon-sb4nt\" (UID: \"3b6432a8-b0ce-4954-8e3d-8eb8782557eb\") " pod="openshift-multus/network-metrics-daemon-sb4nt" Nov 02 06:15:39.971752 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.971731 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vz9mf\" (UniqueName: \"kubernetes.io/projected/3419bd74-22aa-4841-87d2-c4340e01dae9-kube-api-access-vz9mf\") pod \"iptables-alerter-6fnx8\" (UID: \"3419bd74-22aa-4841-87d2-c4340e01dae9\") " pod="openshift-network-operator/iptables-alerter-6fnx8" Nov 02 06:15:39.972695 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.972672 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7xzh\" (UniqueName: \"kubernetes.io/projected/f14e8781-bf82-4810-9909-c126ccdc1ce1-kube-api-access-v7xzh\") pod \"multus-additional-cni-plugins-ckddw\" (UID: \"f14e8781-bf82-4810-9909-c126ccdc1ce1\") " pod="openshift-multus/multus-additional-cni-plugins-ckddw" Nov 02 06:15:39.973202 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.973185 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrq9j\" (UniqueName: \"kubernetes.io/projected/81b393f2-42de-4a99-8114-1445de4d6dc8-kube-api-access-hrq9j\") pod \"tuned-6nk8l\" (UID: \"81b393f2-42de-4a99-8114-1445de4d6dc8\") " pod="openshift-cluster-node-tuning-operator/tuned-6nk8l" Nov 02 06:15:39.973294 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:39.973280 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6vx8\" (UniqueName: \"kubernetes.io/projected/87e797a0-1da6-41ca-af8d-7382afaf3987-kube-api-access-l6vx8\") pod \"multus-rf7qp\" (UID: \"87e797a0-1da6-41ca-af8d-7382afaf3987\") " pod="openshift-multus/multus-rf7qp" Nov 02 06:15:40.051422 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:40.051387 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-rf7qp" Nov 02 06:15:40.081440 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:40.081414 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-hwc99" Nov 02 06:15:40.085700 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:40.085678 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-7wb4x" Nov 02 06:15:40.089385 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:40.089368 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-2p95f" Nov 02 06:15:40.103064 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:40.103045 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-ckddw" Nov 02 06:15:40.122855 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:40.122831 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cbzvd" Nov 02 06:15:40.124930 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:40.124913 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-6nk8l" Nov 02 06:15:40.140059 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:40.140039 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-6fnx8" Nov 02 06:15:40.143597 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:40.143577 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-qm6wn" Nov 02 06:15:40.223437 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:40.223408 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf7856ce915c3ef5bc71a7a412d369e5.slice/crio-b5d368d976619eaa837c56d4e22406db6bf7495a6d9575c227809c8db3f208ac WatchSource:0}: Error finding container b5d368d976619eaa837c56d4e22406db6bf7495a6d9575c227809c8db3f208ac: Status 404 returned error can't find the container with id b5d368d976619eaa837c56d4e22406db6bf7495a6d9575c227809c8db3f208ac Nov 02 06:15:40.225443 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:40.225417 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71bd052da0df40177024b26f7f0347a4.slice/crio-c1ac0d913b943b018616c75971989a95da11db3c09c0128e4db983e8b0d0eaa3 WatchSource:0}: Error finding container c1ac0d913b943b018616c75971989a95da11db3c09c0128e4db983e8b0d0eaa3: Status 404 returned error can't find the container with id c1ac0d913b943b018616c75971989a95da11db3c09c0128e4db983e8b0d0eaa3 Nov 02 06:15:40.230028 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:40.230012 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 06:15:40.425302 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:40.425268 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b6432a8-b0ce-4954-8e3d-8eb8782557eb-metrics-certs\") pod \"network-metrics-daemon-sb4nt\" (UID: \"3b6432a8-b0ce-4954-8e3d-8eb8782557eb\") " pod="openshift-multus/network-metrics-daemon-sb4nt" Nov 02 06:15:40.425463 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:40.425337 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b65bb247-83f7-4749-9aa1-9b2677dd1b3d-original-pull-secret\") pod \"global-pull-secret-syncer-ggpsr\" (UID: \"b65bb247-83f7-4749-9aa1-9b2677dd1b3d\") " pod="kube-system/global-pull-secret-syncer-ggpsr" Nov 02 06:15:40.425463 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:15:40.425418 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 06:15:40.425529 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:15:40.425475 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b6432a8-b0ce-4954-8e3d-8eb8782557eb-metrics-certs podName:3b6432a8-b0ce-4954-8e3d-8eb8782557eb nodeName:}" failed. No retries permitted until 2026-11-02 06:15:41.425461971 +0000 UTC m=+3.317280225 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3b6432a8-b0ce-4954-8e3d-8eb8782557eb-metrics-certs") pod "network-metrics-daemon-sb4nt" (UID: "3b6432a8-b0ce-4954-8e3d-8eb8782557eb") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 06:15:40.425529 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:15:40.425418 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 06:15:40.425529 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:15:40.425521 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b65bb247-83f7-4749-9aa1-9b2677dd1b3d-original-pull-secret podName:b65bb247-83f7-4749-9aa1-9b2677dd1b3d nodeName:}" failed. No retries permitted until 2026-11-02 06:15:41.425511643 +0000 UTC m=+3.317329883 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b65bb247-83f7-4749-9aa1-9b2677dd1b3d-original-pull-secret") pod "global-pull-secret-syncer-ggpsr" (UID: "b65bb247-83f7-4749-9aa1-9b2677dd1b3d") : object "kube-system"/"original-pull-secret" not registered Nov 02 06:15:40.474929 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:40.474834 2579 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 06:15:40.525820 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:40.525766 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnq6k\" (UniqueName: \"kubernetes.io/projected/3ced47b0-9cd0-4710-8f75-2a333437ed01-kube-api-access-gnq6k\") pod \"network-check-target-ft7gs\" (UID: \"3ced47b0-9cd0-4710-8f75-2a333437ed01\") " pod="openshift-network-diagnostics/network-check-target-ft7gs" Nov 02 06:15:40.526003 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:15:40.525950 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 06:15:40.526003 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:15:40.525970 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 06:15:40.526003 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:15:40.525979 2579 projected.go:196] Error preparing data for projected volume kube-api-access-gnq6k for pod openshift-network-diagnostics/network-check-target-ft7gs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 06:15:40.526118 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:15:40.526033 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3ced47b0-9cd0-4710-8f75-2a333437ed01-kube-api-access-gnq6k podName:3ced47b0-9cd0-4710-8f75-2a333437ed01 nodeName:}" failed. No retries permitted until 2026-11-02 06:15:41.526019678 +0000 UTC m=+3.417837922 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-gnq6k" (UniqueName: "kubernetes.io/projected/3ced47b0-9cd0-4710-8f75-2a333437ed01-kube-api-access-gnq6k") pod "network-check-target-ft7gs" (UID: "3ced47b0-9cd0-4710-8f75-2a333437ed01") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 06:15:40.651940 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:40.651876 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-93.us-west-2.compute.internal" event={"ID":"71bd052da0df40177024b26f7f0347a4","Type":"ContainerStarted","Data":"c1ac0d913b943b018616c75971989a95da11db3c09c0128e4db983e8b0d0eaa3"} Nov 02 06:15:40.652784 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:40.652757 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-93.us-west-2.compute.internal" event={"ID":"df7856ce915c3ef5bc71a7a412d369e5","Type":"ContainerStarted","Data":"b5d368d976619eaa837c56d4e22406db6bf7495a6d9575c227809c8db3f208ac"} Nov 02 06:15:40.813420 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:40.813325 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 06:10:39 +0000 UTC" deadline="2028-07-29 09:07:48.562312344 +0000 UTC" Nov 02 06:15:40.813420 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:40.813365 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15242h52m7.748949514s" Nov 02 06:15:40.940437 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:40.940410 2579 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 06:15:40.953860 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:40.953836 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3500e682_2f15_45a2_92d6_bbc35dae3231.slice/crio-5edecaa33e325dbc20c6c14bc24482a1098e38251bf6e78f228cc2b199f6fc9b WatchSource:0}: Error finding container 5edecaa33e325dbc20c6c14bc24482a1098e38251bf6e78f228cc2b199f6fc9b: Status 404 returned error can't find the container with id 5edecaa33e325dbc20c6c14bc24482a1098e38251bf6e78f228cc2b199f6fc9b Nov 02 06:15:40.985742 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:40.985718 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4615a9e_da82_4f8b_8225_d60836b1ba80.slice/crio-3fe2b451a165e20914129200c0e1ed9201cfbb89b1eb8d68742f2ca731715d93 WatchSource:0}: Error finding container 3fe2b451a165e20914129200c0e1ed9201cfbb89b1eb8d68742f2ca731715d93: Status 404 returned error can't find the container with id 3fe2b451a165e20914129200c0e1ed9201cfbb89b1eb8d68742f2ca731715d93 Nov 02 06:15:41.047488 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:41.047464 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8d87317_3f41_46d4_a58e_3c7601e0d445.slice/crio-84ef672c27542c8ee2cd9aa8315ffdac87f5adea3ca97472f1d9343ae269bc89 WatchSource:0}: Error finding container 84ef672c27542c8ee2cd9aa8315ffdac87f5adea3ca97472f1d9343ae269bc89: Status 404 returned error can't find the container with id 84ef672c27542c8ee2cd9aa8315ffdac87f5adea3ca97472f1d9343ae269bc89 Nov 02 06:15:41.048156 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:41.048131 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81b393f2_42de_4a99_8114_1445de4d6dc8.slice/crio-1c21decbe91ba838f2fc3399ac2f680d567adf5f33582e01cb3d56a57c57b6b8 WatchSource:0}: Error finding container 1c21decbe91ba838f2fc3399ac2f680d567adf5f33582e01cb3d56a57c57b6b8: Status 404 returned error can't find the container with id 1c21decbe91ba838f2fc3399ac2f680d567adf5f33582e01cb3d56a57c57b6b8 Nov 02 06:15:41.066426 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:41.066404 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3eec1082_df0b_420b_b4ff_5a1c0fab1d1d.slice/crio-1e1cdd6d7235fef42913f68e6790d9df4741bd50deb1e6f2c87c1978b3d17f19 WatchSource:0}: Error finding container 1e1cdd6d7235fef42913f68e6790d9df4741bd50deb1e6f2c87c1978b3d17f19: Status 404 returned error can't find the container with id 1e1cdd6d7235fef42913f68e6790d9df4741bd50deb1e6f2c87c1978b3d17f19 Nov 02 06:15:41.085296 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:41.085273 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87e797a0_1da6_41ca_af8d_7382afaf3987.slice/crio-23685e5c401a72636d0a3f2994a72afdb729f870e680b5dbd7d42a469c46b89e WatchSource:0}: Error finding container 23685e5c401a72636d0a3f2994a72afdb729f870e680b5dbd7d42a469c46b89e: Status 404 returned error can't find the container with id 23685e5c401a72636d0a3f2994a72afdb729f870e680b5dbd7d42a469c46b89e Nov 02 06:15:41.120103 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:15:41.120077 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3419bd74_22aa_4841_87d2_c4340e01dae9.slice/crio-52cfcf77d0a8a8269f6a1d6364aa5d81ce424af636a65e86e9902c6b1b9ee167 WatchSource:0}: Error finding container 52cfcf77d0a8a8269f6a1d6364aa5d81ce424af636a65e86e9902c6b1b9ee167: Status 404 returned error can't find the container with id 52cfcf77d0a8a8269f6a1d6364aa5d81ce424af636a65e86e9902c6b1b9ee167 Nov 02 06:15:41.434064 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:41.433966 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b6432a8-b0ce-4954-8e3d-8eb8782557eb-metrics-certs\") pod \"network-metrics-daemon-sb4nt\" (UID: \"3b6432a8-b0ce-4954-8e3d-8eb8782557eb\") " pod="openshift-multus/network-metrics-daemon-sb4nt" Nov 02 06:15:41.434064 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:41.434071 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b65bb247-83f7-4749-9aa1-9b2677dd1b3d-original-pull-secret\") pod \"global-pull-secret-syncer-ggpsr\" (UID: \"b65bb247-83f7-4749-9aa1-9b2677dd1b3d\") " pod="kube-system/global-pull-secret-syncer-ggpsr" Nov 02 06:15:41.434257 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:15:41.434184 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 06:15:41.434257 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:15:41.434246 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b65bb247-83f7-4749-9aa1-9b2677dd1b3d-original-pull-secret podName:b65bb247-83f7-4749-9aa1-9b2677dd1b3d nodeName:}" failed. No retries permitted until 2026-11-02 06:15:43.434227531 +0000 UTC m=+5.326045788 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b65bb247-83f7-4749-9aa1-9b2677dd1b3d-original-pull-secret") pod "global-pull-secret-syncer-ggpsr" (UID: "b65bb247-83f7-4749-9aa1-9b2677dd1b3d") : object "kube-system"/"original-pull-secret" not registered Nov 02 06:15:41.434377 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:15:41.434320 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 06:15:41.434377 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:15:41.434355 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b6432a8-b0ce-4954-8e3d-8eb8782557eb-metrics-certs podName:3b6432a8-b0ce-4954-8e3d-8eb8782557eb nodeName:}" failed. No retries permitted until 2026-11-02 06:15:43.4343438 +0000 UTC m=+5.326162054 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3b6432a8-b0ce-4954-8e3d-8eb8782557eb-metrics-certs") pod "network-metrics-daemon-sb4nt" (UID: "3b6432a8-b0ce-4954-8e3d-8eb8782557eb") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 06:15:41.534699 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:41.534667 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnq6k\" (UniqueName: \"kubernetes.io/projected/3ced47b0-9cd0-4710-8f75-2a333437ed01-kube-api-access-gnq6k\") pod \"network-check-target-ft7gs\" (UID: \"3ced47b0-9cd0-4710-8f75-2a333437ed01\") " pod="openshift-network-diagnostics/network-check-target-ft7gs" Nov 02 06:15:41.535295 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:15:41.534854 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 06:15:41.535295 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:15:41.534876 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 06:15:41.535295 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:15:41.534890 2579 projected.go:196] Error preparing data for projected volume kube-api-access-gnq6k for pod openshift-network-diagnostics/network-check-target-ft7gs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 06:15:41.535295 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:15:41.534969 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3ced47b0-9cd0-4710-8f75-2a333437ed01-kube-api-access-gnq6k podName:3ced47b0-9cd0-4710-8f75-2a333437ed01 nodeName:}" failed. No retries permitted until 2026-11-02 06:15:43.534947801 +0000 UTC m=+5.426766058 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-gnq6k" (UniqueName: "kubernetes.io/projected/3ced47b0-9cd0-4710-8f75-2a333437ed01-kube-api-access-gnq6k") pod "network-check-target-ft7gs" (UID: "3ced47b0-9cd0-4710-8f75-2a333437ed01") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 06:15:41.647918 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:41.647870 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sb4nt" Nov 02 06:15:41.647918 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:41.647901 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ggpsr" Nov 02 06:15:41.648220 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:15:41.648092 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sb4nt" podUID="3b6432a8-b0ce-4954-8e3d-8eb8782557eb" Nov 02 06:15:41.648220 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:41.648116 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ft7gs" Nov 02 06:15:41.648297 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:15:41.648217 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ggpsr" podUID="b65bb247-83f7-4749-9aa1-9b2677dd1b3d" Nov 02 06:15:41.648346 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:15:41.648295 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ft7gs" podUID="3ced47b0-9cd0-4710-8f75-2a333437ed01" Nov 02 06:15:41.656434 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:41.656403 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-6fnx8" event={"ID":"3419bd74-22aa-4841-87d2-c4340e01dae9","Type":"ContainerStarted","Data":"52cfcf77d0a8a8269f6a1d6364aa5d81ce424af636a65e86e9902c6b1b9ee167"} Nov 02 06:15:41.657622 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:41.657598 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-rf7qp" event={"ID":"87e797a0-1da6-41ca-af8d-7382afaf3987","Type":"ContainerStarted","Data":"23685e5c401a72636d0a3f2994a72afdb729f870e680b5dbd7d42a469c46b89e"} Nov 02 06:15:41.658756 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:41.658729 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-qm6wn" event={"ID":"3eec1082-df0b-420b-b4ff-5a1c0fab1d1d","Type":"ContainerStarted","Data":"1e1cdd6d7235fef42913f68e6790d9df4741bd50deb1e6f2c87c1978b3d17f19"} Nov 02 06:15:41.660717 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:41.660648 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-6nk8l" event={"ID":"81b393f2-42de-4a99-8114-1445de4d6dc8","Type":"ContainerStarted","Data":"1c21decbe91ba838f2fc3399ac2f680d567adf5f33582e01cb3d56a57c57b6b8"} Nov 02 06:15:41.661927 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:41.661889 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-hwc99" event={"ID":"d4615a9e-da82-4f8b-8225-d60836b1ba80","Type":"ContainerStarted","Data":"3fe2b451a165e20914129200c0e1ed9201cfbb89b1eb8d68742f2ca731715d93"} Nov 02 06:15:41.663363 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:41.663336 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7wb4x" event={"ID":"a2ef8269-c442-42a8-8041-6c1b6aaf0746","Type":"ContainerStarted","Data":"7e9463781594333c81318e1426f180685a00493bdbfb6deb6f58e1e836dffb13"} Nov 02 06:15:41.664948 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:41.664880 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-2p95f" event={"ID":"d8d87317-3f41-46d4-a58e-3c7601e0d445","Type":"ContainerStarted","Data":"84ef672c27542c8ee2cd9aa8315ffdac87f5adea3ca97472f1d9343ae269bc89"} Nov 02 06:15:41.666450 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:41.666426 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ckddw" event={"ID":"f14e8781-bf82-4810-9909-c126ccdc1ce1","Type":"ContainerStarted","Data":"119ed3531b52ab511c43385dc22917968f984e5c99e56eb48bdcb85ee43db915"} Nov 02 06:15:41.667695 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:41.667675 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cbzvd" event={"ID":"3500e682-2f15-45a2-92d6-bbc35dae3231","Type":"ContainerStarted","Data":"5edecaa33e325dbc20c6c14bc24482a1098e38251bf6e78f228cc2b199f6fc9b"} Nov 02 06:15:41.814305 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:41.814200 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 06:10:39 +0000 UTC" deadline="2028-06-03 23:21:16.5939117 +0000 UTC" Nov 02 06:15:41.814305 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:41.814257 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13913h5m34.779657498s" Nov 02 06:15:42.665050 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:42.665015 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sb4nt" Nov 02 06:15:42.665498 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:15:42.665156 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sb4nt" podUID="3b6432a8-b0ce-4954-8e3d-8eb8782557eb" Nov 02 06:15:43.451136 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:43.450853 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b65bb247-83f7-4749-9aa1-9b2677dd1b3d-original-pull-secret\") pod \"global-pull-secret-syncer-ggpsr\" (UID: \"b65bb247-83f7-4749-9aa1-9b2677dd1b3d\") " pod="kube-system/global-pull-secret-syncer-ggpsr" Nov 02 06:15:43.451316 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:43.451186 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b6432a8-b0ce-4954-8e3d-8eb8782557eb-metrics-certs\") pod \"network-metrics-daemon-sb4nt\" (UID: \"3b6432a8-b0ce-4954-8e3d-8eb8782557eb\") " pod="openshift-multus/network-metrics-daemon-sb4nt" Nov 02 06:15:43.451388 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:15:43.451331 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 06:15:43.451442 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:15:43.451398 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b6432a8-b0ce-4954-8e3d-8eb8782557eb-metrics-certs podName:3b6432a8-b0ce-4954-8e3d-8eb8782557eb nodeName:}" failed. No retries permitted until 2026-11-02 06:15:47.451377622 +0000 UTC m=+9.343195885 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3b6432a8-b0ce-4954-8e3d-8eb8782557eb-metrics-certs") pod "network-metrics-daemon-sb4nt" (UID: "3b6432a8-b0ce-4954-8e3d-8eb8782557eb") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 06:15:43.451509 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:15:43.451474 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 06:15:43.451562 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:15:43.451516 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b65bb247-83f7-4749-9aa1-9b2677dd1b3d-original-pull-secret podName:b65bb247-83f7-4749-9aa1-9b2677dd1b3d nodeName:}" failed. No retries permitted until 2026-11-02 06:15:47.45150432 +0000 UTC m=+9.343322563 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b65bb247-83f7-4749-9aa1-9b2677dd1b3d-original-pull-secret") pod "global-pull-secret-syncer-ggpsr" (UID: "b65bb247-83f7-4749-9aa1-9b2677dd1b3d") : object "kube-system"/"original-pull-secret" not registered Nov 02 06:15:43.551958 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:43.551917 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnq6k\" (UniqueName: \"kubernetes.io/projected/3ced47b0-9cd0-4710-8f75-2a333437ed01-kube-api-access-gnq6k\") pod \"network-check-target-ft7gs\" (UID: \"3ced47b0-9cd0-4710-8f75-2a333437ed01\") " pod="openshift-network-diagnostics/network-check-target-ft7gs" Nov 02 06:15:43.552191 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:15:43.552068 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 06:15:43.552191 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:15:43.552090 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 06:15:43.552191 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:15:43.552101 2579 projected.go:196] Error preparing data for projected volume kube-api-access-gnq6k for pod openshift-network-diagnostics/network-check-target-ft7gs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 06:15:43.552191 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:15:43.552156 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3ced47b0-9cd0-4710-8f75-2a333437ed01-kube-api-access-gnq6k podName:3ced47b0-9cd0-4710-8f75-2a333437ed01 nodeName:}" failed. No retries permitted until 2026-11-02 06:15:47.552142598 +0000 UTC m=+9.443960843 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-gnq6k" (UniqueName: "kubernetes.io/projected/3ced47b0-9cd0-4710-8f75-2a333437ed01-kube-api-access-gnq6k") pod "network-check-target-ft7gs" (UID: "3ced47b0-9cd0-4710-8f75-2a333437ed01") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 06:15:43.648724 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:43.648606 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ft7gs" Nov 02 06:15:43.648881 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:15:43.648752 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ft7gs" podUID="3ced47b0-9cd0-4710-8f75-2a333437ed01" Nov 02 06:15:43.648881 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:43.648834 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ggpsr" Nov 02 06:15:43.649061 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:15:43.648922 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ggpsr" podUID="b65bb247-83f7-4749-9aa1-9b2677dd1b3d" Nov 02 06:15:43.687436 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:43.681744 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-93.us-west-2.compute.internal" event={"ID":"df7856ce915c3ef5bc71a7a412d369e5","Type":"ContainerStarted","Data":"7cf04f4ab88fe8af710fd703df392232e8353cd4be1bbe5ab7df145c65ad7082"} Nov 02 06:15:44.649182 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:44.649098 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sb4nt" Nov 02 06:15:44.649308 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:15:44.649271 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sb4nt" podUID="3b6432a8-b0ce-4954-8e3d-8eb8782557eb" Nov 02 06:15:45.648828 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:45.648789 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ggpsr" Nov 02 06:15:45.649295 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:45.648830 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ft7gs" Nov 02 06:15:45.649295 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:15:45.648958 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ggpsr" podUID="b65bb247-83f7-4749-9aa1-9b2677dd1b3d" Nov 02 06:15:45.649295 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:15:45.649057 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ft7gs" podUID="3ced47b0-9cd0-4710-8f75-2a333437ed01" Nov 02 06:15:45.695080 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:45.694366 2579 generic.go:356] "Generic (PLEG): container finished" podID="f14e8781-bf82-4810-9909-c126ccdc1ce1" containerID="76edac44fe3d033b7481267e1b7ca70bc402fc94925f72e65777944577000b23" exitCode=0 Nov 02 06:15:45.695080 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:45.694437 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ckddw" event={"ID":"f14e8781-bf82-4810-9909-c126ccdc1ce1","Type":"ContainerDied","Data":"76edac44fe3d033b7481267e1b7ca70bc402fc94925f72e65777944577000b23"} Nov 02 06:15:45.699860 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:45.699268 2579 generic.go:356] "Generic (PLEG): container finished" podID="71bd052da0df40177024b26f7f0347a4" containerID="06999e0d9cadcd372d30037d1ddca8b04b836b2fc7c351f5fd08606d319a06e3" exitCode=0 Nov 02 06:15:45.699860 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:45.699298 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-93.us-west-2.compute.internal" event={"ID":"71bd052da0df40177024b26f7f0347a4","Type":"ContainerDied","Data":"06999e0d9cadcd372d30037d1ddca8b04b836b2fc7c351f5fd08606d319a06e3"} Nov 02 06:15:45.905525 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:45.904801 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-93.us-west-2.compute.internal" podStartSLOduration=7.904782642 podStartE2EDuration="7.904782642s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 06:15:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:15:43.794669555 +0000 UTC m=+5.686487819" watchObservedRunningTime="2026-11-02 06:15:45.904782642 +0000 UTC m=+7.796600907" Nov 02 06:15:46.652746 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:46.652714 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sb4nt" Nov 02 06:15:46.653222 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:15:46.652858 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sb4nt" podUID="3b6432a8-b0ce-4954-8e3d-8eb8782557eb" Nov 02 06:15:46.705703 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:46.705667 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-93.us-west-2.compute.internal" event={"ID":"71bd052da0df40177024b26f7f0347a4","Type":"ContainerStarted","Data":"2d7ef57bbc7970aa1520b98dd62ad041d6041a58224f00535e5e79b211c56a00"} Nov 02 06:15:46.708444 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:46.708411 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-hwc99" event={"ID":"d4615a9e-da82-4f8b-8225-d60836b1ba80","Type":"ContainerStarted","Data":"3f8a1ff258d50c396bd72de9699d70fd600bd4824440391f52f1f2b38479fb36"} Nov 02 06:15:46.818361 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:46.818311 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-93.us-west-2.compute.internal" podStartSLOduration=8.818293897 podStartE2EDuration="8.818293897s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 06:15:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:15:46.818108575 +0000 UTC m=+8.709926859" watchObservedRunningTime="2026-11-02 06:15:46.818293897 +0000 UTC m=+8.710112160" Nov 02 06:15:46.924773 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:46.924709 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-hwc99" podStartSLOduration=4.216126082 podStartE2EDuration="8.924691847s" totalImagesPullingTime="4.708565765s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:15:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:15:46.924144672 +0000 UTC m=+8.815962954" watchObservedRunningTime="2026-11-02 06:15:46.924691847 +0000 UTC m=+8.816510113" Nov 02 06:15:47.484620 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:47.484038 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b65bb247-83f7-4749-9aa1-9b2677dd1b3d-original-pull-secret\") pod \"global-pull-secret-syncer-ggpsr\" (UID: \"b65bb247-83f7-4749-9aa1-9b2677dd1b3d\") " pod="kube-system/global-pull-secret-syncer-ggpsr" Nov 02 06:15:47.484620 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:47.484110 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b6432a8-b0ce-4954-8e3d-8eb8782557eb-metrics-certs\") pod \"network-metrics-daemon-sb4nt\" (UID: \"3b6432a8-b0ce-4954-8e3d-8eb8782557eb\") " pod="openshift-multus/network-metrics-daemon-sb4nt" Nov 02 06:15:47.484620 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:15:47.484247 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 06:15:47.484620 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:15:47.484364 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b6432a8-b0ce-4954-8e3d-8eb8782557eb-metrics-certs podName:3b6432a8-b0ce-4954-8e3d-8eb8782557eb nodeName:}" failed. No retries permitted until 2026-11-02 06:15:55.484344526 +0000 UTC m=+17.376162786 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3b6432a8-b0ce-4954-8e3d-8eb8782557eb-metrics-certs") pod "network-metrics-daemon-sb4nt" (UID: "3b6432a8-b0ce-4954-8e3d-8eb8782557eb") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 06:15:47.484620 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:15:47.484442 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 06:15:47.484620 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:15:47.484477 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b65bb247-83f7-4749-9aa1-9b2677dd1b3d-original-pull-secret podName:b65bb247-83f7-4749-9aa1-9b2677dd1b3d nodeName:}" failed. No retries permitted until 2026-11-02 06:15:55.484465674 +0000 UTC m=+17.376283932 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b65bb247-83f7-4749-9aa1-9b2677dd1b3d-original-pull-secret") pod "global-pull-secret-syncer-ggpsr" (UID: "b65bb247-83f7-4749-9aa1-9b2677dd1b3d") : object "kube-system"/"original-pull-secret" not registered Nov 02 06:15:47.585566 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:47.585231 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnq6k\" (UniqueName: \"kubernetes.io/projected/3ced47b0-9cd0-4710-8f75-2a333437ed01-kube-api-access-gnq6k\") pod \"network-check-target-ft7gs\" (UID: \"3ced47b0-9cd0-4710-8f75-2a333437ed01\") " pod="openshift-network-diagnostics/network-check-target-ft7gs" Nov 02 06:15:47.585566 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:15:47.585410 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 06:15:47.585566 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:15:47.585451 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 06:15:47.585566 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:15:47.585464 2579 projected.go:196] Error preparing data for projected volume kube-api-access-gnq6k for pod openshift-network-diagnostics/network-check-target-ft7gs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 06:15:47.585566 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:15:47.585524 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3ced47b0-9cd0-4710-8f75-2a333437ed01-kube-api-access-gnq6k podName:3ced47b0-9cd0-4710-8f75-2a333437ed01 nodeName:}" failed. No retries permitted until 2026-11-02 06:15:55.58550764 +0000 UTC m=+17.477325895 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-gnq6k" (UniqueName: "kubernetes.io/projected/3ced47b0-9cd0-4710-8f75-2a333437ed01-kube-api-access-gnq6k") pod "network-check-target-ft7gs" (UID: "3ced47b0-9cd0-4710-8f75-2a333437ed01") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 06:15:47.648452 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:47.648131 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ft7gs" Nov 02 06:15:47.648452 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:15:47.648262 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ft7gs" podUID="3ced47b0-9cd0-4710-8f75-2a333437ed01" Nov 02 06:15:47.648452 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:47.648337 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ggpsr" Nov 02 06:15:47.648452 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:15:47.648401 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ggpsr" podUID="b65bb247-83f7-4749-9aa1-9b2677dd1b3d" Nov 02 06:15:47.715511 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:47.715415 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cbzvd" event={"ID":"3500e682-2f15-45a2-92d6-bbc35dae3231","Type":"ContainerStarted","Data":"059a30c8d2c4f4dc18e0c48f26a4ae70ba46d25afa0b9ca4541fccbfd4ae550e"} Nov 02 06:15:47.719881 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:47.719439 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-qm6wn" event={"ID":"3eec1082-df0b-420b-b4ff-5a1c0fab1d1d","Type":"ContainerStarted","Data":"b7ecfc617547db75f56815502676378e815457521d50ae36a3b0258a1b913042"} Nov 02 06:15:48.238011 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:48.237710 2579 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-hwc99" Nov 02 06:15:48.238657 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:48.238424 2579 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-hwc99" Nov 02 06:15:48.346285 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:48.345660 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-qm6wn" podStartSLOduration=3.80532191 podStartE2EDuration="9.345642249s" totalImagesPullingTime="5.540320339s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 06:15:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:15:47.828772429 +0000 UTC m=+9.720590692" watchObservedRunningTime="2026-11-02 06:15:48.345642249 +0000 UTC m=+10.237460517" Nov 02 06:15:48.650627 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:48.650548 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sb4nt" Nov 02 06:15:48.650788 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:15:48.650677 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sb4nt" podUID="3b6432a8-b0ce-4954-8e3d-8eb8782557eb" Nov 02 06:15:49.418987 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:49.418952 2579 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 02 06:15:49.579459 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:49.579354 2579 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-02T06:15:49.418972018Z","UUID":"b59fca5a-7d60-4695-be7a-ec6ac49e8323","Handler":null,"Name":"","Endpoint":""} Nov 02 06:15:49.581419 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:49.581200 2579 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 02 06:15:49.581530 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:49.581428 2579 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 02 06:15:49.648379 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:49.648347 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ft7gs" Nov 02 06:15:49.648548 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:49.648394 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ggpsr" Nov 02 06:15:49.648548 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:15:49.648475 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ft7gs" podUID="3ced47b0-9cd0-4710-8f75-2a333437ed01" Nov 02 06:15:49.648651 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:15:49.648552 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ggpsr" podUID="b65bb247-83f7-4749-9aa1-9b2677dd1b3d" Nov 02 06:15:49.731112 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:49.730508 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-rf7qp" event={"ID":"87e797a0-1da6-41ca-af8d-7382afaf3987","Type":"ContainerStarted","Data":"b05ea457fac0fe722aa5ed78bab61fbf9322bdb150a16bb6929a8cde01e7f591"} Nov 02 06:15:49.732846 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:49.732462 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-2p95f" event={"ID":"d8d87317-3f41-46d4-a58e-3c7601e0d445","Type":"ContainerStarted","Data":"6087cb9ef525e104164ef2b15ab46bbced4b2b8dbaa2ec6d46aa6600bd1227a7"} Nov 02 06:15:49.735420 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:49.735393 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cbzvd" event={"ID":"3500e682-2f15-45a2-92d6-bbc35dae3231","Type":"ContainerStarted","Data":"493625c2dba1395ae3f8067c13ba78430ab9b530a18974ed054756d2b71808ec"} Nov 02 06:15:49.735633 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:49.735618 2579 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 06:15:49.838871 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:49.838783 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-rf7qp" podStartSLOduration=3.8133763050000002 podStartE2EDuration="11.838770635s" totalImagesPullingTime="8.02539433s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 06:15:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:15:49.838741995 +0000 UTC m=+11.730560275" watchObservedRunningTime="2026-11-02 06:15:49.838770635 +0000 UTC m=+11.730588897" Nov 02 06:15:49.945376 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:49.945327 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-2p95f" podStartSLOduration=4.048869609 podStartE2EDuration="11.945306502s" totalImagesPullingTime="7.896436893s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 06:15:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:15:49.944553742 +0000 UTC m=+11.836372008" watchObservedRunningTime="2026-11-02 06:15:49.945306502 +0000 UTC m=+11.837124781" Nov 02 06:15:50.648764 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:50.648726 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sb4nt" Nov 02 06:15:50.649280 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:15:50.648863 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sb4nt" podUID="3b6432a8-b0ce-4954-8e3d-8eb8782557eb" Nov 02 06:15:50.740362 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:50.740327 2579 generic.go:356] "Generic (PLEG): container finished" podID="f14e8781-bf82-4810-9909-c126ccdc1ce1" containerID="94631bd8da05ef22b70e792603345d4a8c23600a4c60a85272ac7327b8c9e6de" exitCode=0 Nov 02 06:15:50.740532 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:50.740392 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ckddw" event={"ID":"f14e8781-bf82-4810-9909-c126ccdc1ce1","Type":"ContainerDied","Data":"94631bd8da05ef22b70e792603345d4a8c23600a4c60a85272ac7327b8c9e6de"} Nov 02 06:15:51.648453 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:51.648422 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ft7gs" Nov 02 06:15:51.648453 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:51.648450 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ggpsr" Nov 02 06:15:51.648667 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:15:51.648546 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ft7gs" podUID="3ced47b0-9cd0-4710-8f75-2a333437ed01" Nov 02 06:15:51.648667 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:15:51.648651 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ggpsr" podUID="b65bb247-83f7-4749-9aa1-9b2677dd1b3d" Nov 02 06:15:51.743555 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:51.743520 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-6fnx8" event={"ID":"3419bd74-22aa-4841-87d2-c4340e01dae9","Type":"ContainerStarted","Data":"cb5eaa35ad9152ef2c861065b524f413b9a966f7b9ed91bf606362a6a9732386"} Nov 02 06:15:51.849686 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:51.849634 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-6fnx8" podStartSLOduration=4.406144897 podStartE2EDuration="12.849616265s" totalImagesPullingTime="8.443471368s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 06:15:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:15:51.849027313 +0000 UTC m=+13.740845577" watchObservedRunningTime="2026-11-02 06:15:51.849616265 +0000 UTC m=+13.741434530" Nov 02 06:15:52.648634 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:52.648610 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sb4nt" Nov 02 06:15:52.648747 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:15:52.648716 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sb4nt" podUID="3b6432a8-b0ce-4954-8e3d-8eb8782557eb" Nov 02 06:15:52.747463 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:52.747430 2579 generic.go:356] "Generic (PLEG): container finished" podID="f14e8781-bf82-4810-9909-c126ccdc1ce1" containerID="6319d210ebfcd8b05a29408bdb389e63c5195f58158050e90c9ce7b60421c445" exitCode=0 Nov 02 06:15:52.748099 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:52.747508 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ckddw" event={"ID":"f14e8781-bf82-4810-9909-c126ccdc1ce1","Type":"ContainerDied","Data":"6319d210ebfcd8b05a29408bdb389e63c5195f58158050e90c9ce7b60421c445"} Nov 02 06:15:52.749687 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:52.749658 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cbzvd" event={"ID":"3500e682-2f15-45a2-92d6-bbc35dae3231","Type":"ContainerStarted","Data":"54761ea19110d589faa5e1b0a48abf501e83020b6fa7be233c1f9bbc6c98830a"} Nov 02 06:15:52.753163 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:52.753142 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-6nk8l" event={"ID":"81b393f2-42de-4a99-8114-1445de4d6dc8","Type":"ContainerStarted","Data":"85913b55a457824c9c2ac7c223bdfb9d0b10d58634084d2ad40e0e0c3fbd173e"} Nov 02 06:15:52.962130 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:52.962031 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cbzvd" podStartSLOduration=4.590661233 podStartE2EDuration="14.96201384s" totalImagesPullingTime="10.371352607s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 06:15:38 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:15:52.961797733 +0000 UTC m=+14.853615996" watchObservedRunningTime="2026-11-02 06:15:52.96201384 +0000 UTC m=+14.853832106" Nov 02 06:15:53.069023 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:53.068971 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-6nk8l" podStartSLOduration=3.34345218 podStartE2EDuration="14.068959102s" totalImagesPullingTime="10.725506922s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 06:15:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:15:53.068226089 +0000 UTC m=+14.960044353" watchObservedRunningTime="2026-11-02 06:15:53.068959102 +0000 UTC m=+14.960777363" Nov 02 06:15:53.648754 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:53.648724 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ft7gs" Nov 02 06:15:53.648975 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:53.648724 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ggpsr" Nov 02 06:15:53.648975 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:15:53.648835 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ft7gs" podUID="3ced47b0-9cd0-4710-8f75-2a333437ed01" Nov 02 06:15:53.648975 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:15:53.648891 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ggpsr" podUID="b65bb247-83f7-4749-9aa1-9b2677dd1b3d" Nov 02 06:15:54.648314 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:54.647992 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sb4nt" Nov 02 06:15:54.648731 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:15:54.648365 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sb4nt" podUID="3b6432a8-b0ce-4954-8e3d-8eb8782557eb" Nov 02 06:15:54.759253 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:54.759219 2579 generic.go:356] "Generic (PLEG): container finished" podID="f14e8781-bf82-4810-9909-c126ccdc1ce1" containerID="461ad93213230147cb160bb730bef28178a765173b74c29daa8aa46e000b25c2" exitCode=0 Nov 02 06:15:54.759425 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:54.759263 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ckddw" event={"ID":"f14e8781-bf82-4810-9909-c126ccdc1ce1","Type":"ContainerDied","Data":"461ad93213230147cb160bb730bef28178a765173b74c29daa8aa46e000b25c2"} Nov 02 06:15:55.543156 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:55.543114 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b65bb247-83f7-4749-9aa1-9b2677dd1b3d-original-pull-secret\") pod \"global-pull-secret-syncer-ggpsr\" (UID: \"b65bb247-83f7-4749-9aa1-9b2677dd1b3d\") " pod="kube-system/global-pull-secret-syncer-ggpsr" Nov 02 06:15:55.543345 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:55.543183 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b6432a8-b0ce-4954-8e3d-8eb8782557eb-metrics-certs\") pod \"network-metrics-daemon-sb4nt\" (UID: \"3b6432a8-b0ce-4954-8e3d-8eb8782557eb\") " pod="openshift-multus/network-metrics-daemon-sb4nt" Nov 02 06:15:55.543345 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:15:55.543277 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 06:15:55.543345 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:15:55.543338 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 06:15:55.543454 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:15:55.543354 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b65bb247-83f7-4749-9aa1-9b2677dd1b3d-original-pull-secret podName:b65bb247-83f7-4749-9aa1-9b2677dd1b3d nodeName:}" failed. No retries permitted until 2026-11-02 06:16:11.543339263 +0000 UTC m=+33.435157503 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b65bb247-83f7-4749-9aa1-9b2677dd1b3d-original-pull-secret") pod "global-pull-secret-syncer-ggpsr" (UID: "b65bb247-83f7-4749-9aa1-9b2677dd1b3d") : object "kube-system"/"original-pull-secret" not registered Nov 02 06:15:55.543454 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:15:55.543389 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b6432a8-b0ce-4954-8e3d-8eb8782557eb-metrics-certs podName:3b6432a8-b0ce-4954-8e3d-8eb8782557eb nodeName:}" failed. No retries permitted until 2026-11-02 06:16:11.54337706 +0000 UTC m=+33.435195303 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3b6432a8-b0ce-4954-8e3d-8eb8782557eb-metrics-certs") pod "network-metrics-daemon-sb4nt" (UID: "3b6432a8-b0ce-4954-8e3d-8eb8782557eb") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 06:15:55.643754 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:55.643720 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnq6k\" (UniqueName: \"kubernetes.io/projected/3ced47b0-9cd0-4710-8f75-2a333437ed01-kube-api-access-gnq6k\") pod \"network-check-target-ft7gs\" (UID: \"3ced47b0-9cd0-4710-8f75-2a333437ed01\") " pod="openshift-network-diagnostics/network-check-target-ft7gs" Nov 02 06:15:55.643964 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:15:55.643894 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 06:15:55.643964 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:15:55.643929 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 06:15:55.643964 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:15:55.643941 2579 projected.go:196] Error preparing data for projected volume kube-api-access-gnq6k for pod openshift-network-diagnostics/network-check-target-ft7gs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 06:15:55.644125 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:15:55.643998 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3ced47b0-9cd0-4710-8f75-2a333437ed01-kube-api-access-gnq6k podName:3ced47b0-9cd0-4710-8f75-2a333437ed01 nodeName:}" failed. No retries permitted until 2026-11-02 06:16:11.643978338 +0000 UTC m=+33.535796588 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-gnq6k" (UniqueName: "kubernetes.io/projected/3ced47b0-9cd0-4710-8f75-2a333437ed01-kube-api-access-gnq6k") pod "network-check-target-ft7gs" (UID: "3ced47b0-9cd0-4710-8f75-2a333437ed01") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 06:15:55.648321 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:55.648299 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ggpsr" Nov 02 06:15:55.648749 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:55.648331 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ft7gs" Nov 02 06:15:55.648749 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:15:55.648410 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ggpsr" podUID="b65bb247-83f7-4749-9aa1-9b2677dd1b3d" Nov 02 06:15:55.648749 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:15:55.648470 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ft7gs" podUID="3ced47b0-9cd0-4710-8f75-2a333437ed01" Nov 02 06:15:56.650504 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:56.650471 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sb4nt" Nov 02 06:15:56.651007 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:15:56.650625 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sb4nt" podUID="3b6432a8-b0ce-4954-8e3d-8eb8782557eb" Nov 02 06:15:57.648624 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:57.648584 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ggpsr" Nov 02 06:15:57.648817 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:15:57.648712 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ggpsr" podUID="b65bb247-83f7-4749-9aa1-9b2677dd1b3d" Nov 02 06:15:57.648817 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:57.648749 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ft7gs" Nov 02 06:15:57.648960 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:15:57.648822 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ft7gs" podUID="3ced47b0-9cd0-4710-8f75-2a333437ed01" Nov 02 06:15:57.768629 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:57.768561 2579 generic.go:356] "Generic (PLEG): container finished" podID="f14e8781-bf82-4810-9909-c126ccdc1ce1" containerID="2590abd77229ac30d7d9c7b2730e33a159aae8da0294b29c2cb2179fce248d00" exitCode=0 Nov 02 06:15:57.768629 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:57.768604 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ckddw" event={"ID":"f14e8781-bf82-4810-9909-c126ccdc1ce1","Type":"ContainerDied","Data":"2590abd77229ac30d7d9c7b2730e33a159aae8da0294b29c2cb2179fce248d00"} Nov 02 06:15:58.648785 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:58.648749 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sb4nt" Nov 02 06:15:58.648973 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:15:58.648857 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sb4nt" podUID="3b6432a8-b0ce-4954-8e3d-8eb8782557eb" Nov 02 06:15:58.773116 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:58.773084 2579 generic.go:356] "Generic (PLEG): container finished" podID="f14e8781-bf82-4810-9909-c126ccdc1ce1" containerID="6126d2d3da241baf441593f8786423fe729eaafe1d673f2edba5283c8315c686" exitCode=0 Nov 02 06:15:58.773525 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:58.773138 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ckddw" event={"ID":"f14e8781-bf82-4810-9909-c126ccdc1ce1","Type":"ContainerDied","Data":"6126d2d3da241baf441593f8786423fe729eaafe1d673f2edba5283c8315c686"} Nov 02 06:15:59.647782 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:59.647748 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ggpsr" Nov 02 06:15:59.647782 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:59.647764 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ft7gs" Nov 02 06:15:59.648026 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:15:59.647844 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ggpsr" podUID="b65bb247-83f7-4749-9aa1-9b2677dd1b3d" Nov 02 06:15:59.648026 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:15:59.647915 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ft7gs" podUID="3ced47b0-9cd0-4710-8f75-2a333437ed01" Nov 02 06:15:59.778019 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:59.777982 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7wb4x" event={"ID":"a2ef8269-c442-42a8-8041-6c1b6aaf0746","Type":"ContainerStarted","Data":"b1993145b4c3058f202fc49a5391867fa514adbc4e4e4cfa1e929be3a1286208"} Nov 02 06:15:59.778559 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:59.778069 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7wb4x" event={"ID":"a2ef8269-c442-42a8-8041-6c1b6aaf0746","Type":"ContainerStarted","Data":"4fb3354a589c1fb0703946adad0c0ef51d5cdd77a05bee06a768769c3974e494"} Nov 02 06:15:59.778559 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:59.778121 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7wb4x" event={"ID":"a2ef8269-c442-42a8-8041-6c1b6aaf0746","Type":"ContainerStarted","Data":"06e2597484b695673f022e244f57b3659171cfe75b1c0073906e70f7d93de4de"} Nov 02 06:15:59.778559 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:59.778169 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7wb4x" event={"ID":"a2ef8269-c442-42a8-8041-6c1b6aaf0746","Type":"ContainerStarted","Data":"0a9178c17375555341a868f574498cfb16e9ec99274950fb9c34479adbdc821f"} Nov 02 06:15:59.778559 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:59.778212 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7wb4x" event={"ID":"a2ef8269-c442-42a8-8041-6c1b6aaf0746","Type":"ContainerStarted","Data":"b6b0eeb266e67ab77f566b445b00f96790aea6ba3cba8d34c7e146b8f76e10f0"} Nov 02 06:15:59.778559 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:59.778252 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7wb4x" event={"ID":"a2ef8269-c442-42a8-8041-6c1b6aaf0746","Type":"ContainerStarted","Data":"47cbb43f3d07a7e97c1e949fca1c104ae88d89459f89c2842f198b7e707e85ef"} Nov 02 06:15:59.781212 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:59.781188 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ckddw" event={"ID":"f14e8781-bf82-4810-9909-c126ccdc1ce1","Type":"ContainerStarted","Data":"1d1abeb04c293a90804ab8b6aaa354f3306f1fc56edf2a68e260812d780c8306"} Nov 02 06:15:59.895563 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:15:59.895519 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-ckddw" podStartSLOduration=8.447410632 podStartE2EDuration="21.89550797s" totalImagesPullingTime="13.448097338s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 06:15:38 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:15:59.895288889 +0000 UTC m=+21.787107158" watchObservedRunningTime="2026-11-02 06:15:59.89550797 +0000 UTC m=+21.787326232" Nov 02 06:16:00.650845 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:00.650815 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sb4nt" Nov 02 06:16:00.651025 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:16:00.650946 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sb4nt" podUID="3b6432a8-b0ce-4954-8e3d-8eb8782557eb" Nov 02 06:16:01.648096 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:01.648018 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ft7gs" Nov 02 06:16:01.648096 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:01.648063 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ggpsr" Nov 02 06:16:01.648578 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:16:01.648130 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ft7gs" podUID="3ced47b0-9cd0-4710-8f75-2a333437ed01" Nov 02 06:16:01.648578 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:16:01.648237 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ggpsr" podUID="b65bb247-83f7-4749-9aa1-9b2677dd1b3d" Nov 02 06:16:01.787696 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:01.787665 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7wb4x" event={"ID":"a2ef8269-c442-42a8-8041-6c1b6aaf0746","Type":"ContainerStarted","Data":"121238c794896c72851c85f49e5332639574d02ce85c776d05ac119a1682199d"} Nov 02 06:16:02.648419 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:02.648387 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sb4nt" Nov 02 06:16:02.648977 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:16:02.648558 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sb4nt" podUID="3b6432a8-b0ce-4954-8e3d-8eb8782557eb" Nov 02 06:16:03.648062 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:03.647987 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ft7gs" Nov 02 06:16:03.648214 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:03.647986 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ggpsr" Nov 02 06:16:03.648214 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:16:03.648085 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ft7gs" podUID="3ced47b0-9cd0-4710-8f75-2a333437ed01" Nov 02 06:16:03.648214 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:16:03.648158 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ggpsr" podUID="b65bb247-83f7-4749-9aa1-9b2677dd1b3d" Nov 02 06:16:03.796447 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:03.796411 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7wb4x" event={"ID":"a2ef8269-c442-42a8-8041-6c1b6aaf0746","Type":"ContainerStarted","Data":"b8437fa4f1b2a83f86997fc2d91aeb0d36885ea6365a97414a6ef3fd58985d07"} Nov 02 06:16:03.797178 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:03.796765 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7wb4x" Nov 02 06:16:03.812013 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:03.811987 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7wb4x" Nov 02 06:16:04.067761 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:04.067660 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-7wb4x" podStartSLOduration=8.193167728 podStartE2EDuration="26.067645497s" totalImagesPullingTime="17.874477769s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 06:15:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:16:04.019806481 +0000 UTC m=+25.911624754" watchObservedRunningTime="2026-11-02 06:16:04.067645497 +0000 UTC m=+25.959463758" Nov 02 06:16:04.648608 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:04.648562 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sb4nt" Nov 02 06:16:04.648786 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:16:04.648744 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sb4nt" podUID="3b6432a8-b0ce-4954-8e3d-8eb8782557eb" Nov 02 06:16:04.799395 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:04.799357 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7wb4x" Nov 02 06:16:04.799787 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:04.799452 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7wb4x" Nov 02 06:16:04.813784 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:04.813760 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7wb4x" Nov 02 06:16:05.647876 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:05.647841 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ggpsr" Nov 02 06:16:05.648072 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:05.647841 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ft7gs" Nov 02 06:16:05.648072 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:16:05.647952 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ggpsr" podUID="b65bb247-83f7-4749-9aa1-9b2677dd1b3d" Nov 02 06:16:05.648072 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:16:05.648031 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ft7gs" podUID="3ced47b0-9cd0-4710-8f75-2a333437ed01" Nov 02 06:16:05.879364 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:05.879176 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-ggpsr"] Nov 02 06:16:05.879719 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:05.879434 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ggpsr" Nov 02 06:16:05.879719 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:16:05.879518 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ggpsr" podUID="b65bb247-83f7-4749-9aa1-9b2677dd1b3d" Nov 02 06:16:05.882853 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:05.882827 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-sb4nt"] Nov 02 06:16:05.883013 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:05.882966 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sb4nt" Nov 02 06:16:05.883103 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:16:05.883080 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sb4nt" podUID="3b6432a8-b0ce-4954-8e3d-8eb8782557eb" Nov 02 06:16:05.883424 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:05.883399 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-ft7gs"] Nov 02 06:16:05.883527 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:05.883481 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ft7gs" Nov 02 06:16:05.883572 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:16:05.883553 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ft7gs" podUID="3ced47b0-9cd0-4710-8f75-2a333437ed01" Nov 02 06:16:07.648626 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:07.648591 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ggpsr" Nov 02 06:16:07.649147 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:07.648732 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sb4nt" Nov 02 06:16:07.649147 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:07.648762 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ft7gs" Nov 02 06:16:07.649147 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:16:07.648755 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ggpsr" podUID="b65bb247-83f7-4749-9aa1-9b2677dd1b3d" Nov 02 06:16:07.649147 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:16:07.648828 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sb4nt" podUID="3b6432a8-b0ce-4954-8e3d-8eb8782557eb" Nov 02 06:16:07.649147 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:16:07.648885 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ft7gs" podUID="3ced47b0-9cd0-4710-8f75-2a333437ed01" Nov 02 06:16:08.842680 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:08.842644 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-hwc99" Nov 02 06:16:08.843286 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:08.842782 2579 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 06:16:08.843286 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:08.843239 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-hwc99" Nov 02 06:16:09.648161 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:09.648127 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ft7gs" Nov 02 06:16:09.648161 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:09.648154 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sb4nt" Nov 02 06:16:09.648483 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:16:09.648222 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ft7gs" podUID="3ced47b0-9cd0-4710-8f75-2a333437ed01" Nov 02 06:16:09.648483 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:09.648276 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ggpsr" Nov 02 06:16:09.648483 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:16:09.648341 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ggpsr" podUID="b65bb247-83f7-4749-9aa1-9b2677dd1b3d" Nov 02 06:16:09.648483 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:16:09.648399 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sb4nt" podUID="3b6432a8-b0ce-4954-8e3d-8eb8782557eb" Nov 02 06:16:11.564536 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:11.564504 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b6432a8-b0ce-4954-8e3d-8eb8782557eb-metrics-certs\") pod \"network-metrics-daemon-sb4nt\" (UID: \"3b6432a8-b0ce-4954-8e3d-8eb8782557eb\") " pod="openshift-multus/network-metrics-daemon-sb4nt" Nov 02 06:16:11.564924 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:11.564573 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b65bb247-83f7-4749-9aa1-9b2677dd1b3d-original-pull-secret\") pod \"global-pull-secret-syncer-ggpsr\" (UID: \"b65bb247-83f7-4749-9aa1-9b2677dd1b3d\") " pod="kube-system/global-pull-secret-syncer-ggpsr" Nov 02 06:16:11.564924 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:16:11.564652 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 06:16:11.564924 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:16:11.564661 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 06:16:11.564924 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:16:11.564711 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b6432a8-b0ce-4954-8e3d-8eb8782557eb-metrics-certs podName:3b6432a8-b0ce-4954-8e3d-8eb8782557eb nodeName:}" failed. No retries permitted until 2026-11-02 06:16:43.564694561 +0000 UTC m=+65.456512805 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3b6432a8-b0ce-4954-8e3d-8eb8782557eb-metrics-certs") pod "network-metrics-daemon-sb4nt" (UID: "3b6432a8-b0ce-4954-8e3d-8eb8782557eb") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 06:16:11.564924 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:16:11.564726 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b65bb247-83f7-4749-9aa1-9b2677dd1b3d-original-pull-secret podName:b65bb247-83f7-4749-9aa1-9b2677dd1b3d nodeName:}" failed. No retries permitted until 2026-11-02 06:16:43.564719957 +0000 UTC m=+65.456538202 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b65bb247-83f7-4749-9aa1-9b2677dd1b3d-original-pull-secret") pod "global-pull-secret-syncer-ggpsr" (UID: "b65bb247-83f7-4749-9aa1-9b2677dd1b3d") : object "kube-system"/"original-pull-secret" not registered Nov 02 06:16:11.648342 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:11.648310 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sb4nt" Nov 02 06:16:11.648342 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:11.648336 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ggpsr" Nov 02 06:16:11.648532 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:11.648310 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ft7gs" Nov 02 06:16:11.648532 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:16:11.648412 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sb4nt" podUID="3b6432a8-b0ce-4954-8e3d-8eb8782557eb" Nov 02 06:16:11.648532 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:16:11.648486 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ft7gs" podUID="3ced47b0-9cd0-4710-8f75-2a333437ed01" Nov 02 06:16:11.648668 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:16:11.648545 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ggpsr" podUID="b65bb247-83f7-4749-9aa1-9b2677dd1b3d" Nov 02 06:16:11.665567 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:11.665532 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnq6k\" (UniqueName: \"kubernetes.io/projected/3ced47b0-9cd0-4710-8f75-2a333437ed01-kube-api-access-gnq6k\") pod \"network-check-target-ft7gs\" (UID: \"3ced47b0-9cd0-4710-8f75-2a333437ed01\") " pod="openshift-network-diagnostics/network-check-target-ft7gs" Nov 02 06:16:11.665725 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:16:11.665685 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 06:16:11.665725 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:16:11.665703 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 06:16:11.665725 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:16:11.665714 2579 projected.go:196] Error preparing data for projected volume kube-api-access-gnq6k for pod openshift-network-diagnostics/network-check-target-ft7gs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 06:16:11.665853 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:16:11.665774 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3ced47b0-9cd0-4710-8f75-2a333437ed01-kube-api-access-gnq6k podName:3ced47b0-9cd0-4710-8f75-2a333437ed01 nodeName:}" failed. No retries permitted until 2026-11-02 06:16:43.66575426 +0000 UTC m=+65.557572522 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-gnq6k" (UniqueName: "kubernetes.io/projected/3ced47b0-9cd0-4710-8f75-2a333437ed01-kube-api-access-gnq6k") pod "network-check-target-ft7gs" (UID: "3ced47b0-9cd0-4710-8f75-2a333437ed01") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 06:16:11.947068 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:11.946992 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-93.us-west-2.compute.internal" event="NodeReady" Nov 02 06:16:11.947226 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:11.947111 2579 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 02 06:16:12.095767 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.095736 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-kd68m"] Nov 02 06:16:12.120138 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.120098 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-kb9wk"] Nov 02 06:16:12.120314 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.120238 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-kd68m" Nov 02 06:16:12.129938 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.129889 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-jrtnt"] Nov 02 06:16:12.130372 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.130350 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-kb9wk" Nov 02 06:16:12.146207 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.146183 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6689c6d46b-vw558"] Nov 02 06:16:12.146317 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.146253 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-jrtnt" Nov 02 06:16:12.161638 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.161610 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-746899d75-dtjhs"] Nov 02 06:16:12.161823 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.161798 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6689c6d46b-vw558" Nov 02 06:16:12.171763 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.171742 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-5f5c58667c-29wnm"] Nov 02 06:16:12.171901 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.171851 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-746899d75-dtjhs" Nov 02 06:16:12.186518 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.186495 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-2rs6c"] Nov 02 06:16:12.186661 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.186607 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5f5c58667c-29wnm" Nov 02 06:16:12.192404 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.192385 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Nov 02 06:16:12.196948 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.196930 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Nov 02 06:16:12.204021 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.203972 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-sktt2"] Nov 02 06:16:12.204117 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.204080 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2rs6c" Nov 02 06:16:12.216322 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.216300 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-csfhg"] Nov 02 06:16:12.216433 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.216403 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-sktt2" Nov 02 06:16:12.222742 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.222721 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-d6bc4"] Nov 02 06:16:12.223179 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.222859 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-csfhg" Nov 02 06:16:12.229560 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.229540 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-wcrtl"] Nov 02 06:16:12.229670 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.229654 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-d6bc4" Nov 02 06:16:12.237894 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.237877 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-8fbcn"] Nov 02 06:16:12.238057 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.238041 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wcrtl" Nov 02 06:16:12.241166 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.241146 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Nov 02 06:16:12.241258 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.241199 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Nov 02 06:16:12.241311 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.241301 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Nov 02 06:16:12.241360 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.241306 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Nov 02 06:16:12.241360 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.241306 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Nov 02 06:16:12.241360 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.241329 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Nov 02 06:16:12.241472 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.241377 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-rpvjg\"" Nov 02 06:16:12.248477 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.248462 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-fh4sz"] Nov 02 06:16:12.248577 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.248564 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-8fbcn" Nov 02 06:16:12.256803 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.256786 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-c8dcq"] Nov 02 06:16:12.256914 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.256857 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-fh4sz" Nov 02 06:16:12.264544 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.264528 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-k9hcs"] Nov 02 06:16:12.264651 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.264639 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-c8dcq" Nov 02 06:16:12.268968 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.268951 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f0531b4d-c954-4830-85f6-fe876c76437a-ca-trust-extracted\") pod \"image-registry-6689c6d46b-vw558\" (UID: \"f0531b4d-c954-4830-85f6-fe876c76437a\") " pod="openshift-image-registry/image-registry-6689c6d46b-vw558" Nov 02 06:16:12.269056 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.268982 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f0531b4d-c954-4830-85f6-fe876c76437a-installation-pull-secrets\") pod \"image-registry-6689c6d46b-vw558\" (UID: \"f0531b4d-c954-4830-85f6-fe876c76437a\") " pod="openshift-image-registry/image-registry-6689c6d46b-vw558" Nov 02 06:16:12.269056 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.269024 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fa9c5a91-d480-4d7b-9044-be6437a76a02-installation-pull-secrets\") pod \"image-registry-746899d75-dtjhs\" (UID: \"fa9c5a91-d480-4d7b-9044-be6437a76a02\") " pod="openshift-image-registry/image-registry-746899d75-dtjhs" Nov 02 06:16:12.269141 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.269126 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cctkp\" (UniqueName: \"kubernetes.io/projected/fa9c5a91-d480-4d7b-9044-be6437a76a02-kube-api-access-cctkp\") pod \"image-registry-746899d75-dtjhs\" (UID: \"fa9c5a91-d480-4d7b-9044-be6437a76a02\") " pod="openshift-image-registry/image-registry-746899d75-dtjhs" Nov 02 06:16:12.269185 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.269155 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f0531b4d-c954-4830-85f6-fe876c76437a-registry-certificates\") pod \"image-registry-6689c6d46b-vw558\" (UID: \"f0531b4d-c954-4830-85f6-fe876c76437a\") " pod="openshift-image-registry/image-registry-6689c6d46b-vw558" Nov 02 06:16:12.269222 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.269199 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f0531b4d-c954-4830-85f6-fe876c76437a-bound-sa-token\") pod \"image-registry-6689c6d46b-vw558\" (UID: \"f0531b4d-c954-4830-85f6-fe876c76437a\") " pod="openshift-image-registry/image-registry-6689c6d46b-vw558" Nov 02 06:16:12.269265 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.269236 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45e5ce56-5cab-4546-8b31-02fc06cfa0cc-service-ca-bundle\") pod \"router-default-5f5c58667c-29wnm\" (UID: \"45e5ce56-5cab-4546-8b31-02fc06cfa0cc\") " pod="openshift-ingress/router-default-5f5c58667c-29wnm" Nov 02 06:16:12.269306 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.269263 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fa9c5a91-d480-4d7b-9044-be6437a76a02-image-registry-private-configuration\") pod \"image-registry-746899d75-dtjhs\" (UID: \"fa9c5a91-d480-4d7b-9044-be6437a76a02\") " pod="openshift-image-registry/image-registry-746899d75-dtjhs" Nov 02 06:16:12.269306 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.269302 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f0531b4d-c954-4830-85f6-fe876c76437a-trusted-ca\") pod \"image-registry-6689c6d46b-vw558\" (UID: \"f0531b4d-c954-4830-85f6-fe876c76437a\") " pod="openshift-image-registry/image-registry-6689c6d46b-vw558" Nov 02 06:16:12.269370 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.269325 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3fcb7621-86c8-448f-b4fa-b5bd76be7356-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-2rs6c\" (UID: \"3fcb7621-86c8-448f-b4fa-b5bd76be7356\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2rs6c" Nov 02 06:16:12.269370 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.269360 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fa9c5a91-d480-4d7b-9044-be6437a76a02-trusted-ca\") pod \"image-registry-746899d75-dtjhs\" (UID: \"fa9c5a91-d480-4d7b-9044-be6437a76a02\") " pod="openshift-image-registry/image-registry-746899d75-dtjhs" Nov 02 06:16:12.269439 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.269405 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrh7x\" (UniqueName: \"kubernetes.io/projected/8e8f2e60-de1b-4c8b-9285-56568c150d92-kube-api-access-zrh7x\") pod \"kas-connection-checker-685895c7c9-kd68m\" (UID: \"8e8f2e60-de1b-4c8b-9285-56568c150d92\") " pod="kube-system/kas-connection-checker-685895c7c9-kd68m" Nov 02 06:16:12.269439 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.269424 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f0531b4d-c954-4830-85f6-fe876c76437a-registry-tls\") pod \"image-registry-6689c6d46b-vw558\" (UID: \"f0531b4d-c954-4830-85f6-fe876c76437a\") " pod="openshift-image-registry/image-registry-6689c6d46b-vw558" Nov 02 06:16:12.269543 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.269442 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m89g8\" (UniqueName: \"kubernetes.io/projected/f0531b4d-c954-4830-85f6-fe876c76437a-kube-api-access-m89g8\") pod \"image-registry-6689c6d46b-vw558\" (UID: \"f0531b4d-c954-4830-85f6-fe876c76437a\") " pod="openshift-image-registry/image-registry-6689c6d46b-vw558" Nov 02 06:16:12.269543 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.269462 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/45e5ce56-5cab-4546-8b31-02fc06cfa0cc-stats-auth\") pod \"router-default-5f5c58667c-29wnm\" (UID: \"45e5ce56-5cab-4546-8b31-02fc06cfa0cc\") " pod="openshift-ingress/router-default-5f5c58667c-29wnm" Nov 02 06:16:12.269543 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.269477 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f0531b4d-c954-4830-85f6-fe876c76437a-image-registry-private-configuration\") pod \"image-registry-6689c6d46b-vw558\" (UID: \"f0531b4d-c954-4830-85f6-fe876c76437a\") " pod="openshift-image-registry/image-registry-6689c6d46b-vw558" Nov 02 06:16:12.269543 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.269521 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwh5f\" (UniqueName: \"kubernetes.io/projected/3fcb7621-86c8-448f-b4fa-b5bd76be7356-kube-api-access-mwh5f\") pod \"cluster-monitoring-operator-884576c57-2rs6c\" (UID: \"3fcb7621-86c8-448f-b4fa-b5bd76be7356\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2rs6c" Nov 02 06:16:12.269751 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.269587 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fa9c5a91-d480-4d7b-9044-be6437a76a02-ca-trust-extracted\") pod \"image-registry-746899d75-dtjhs\" (UID: \"fa9c5a91-d480-4d7b-9044-be6437a76a02\") " pod="openshift-image-registry/image-registry-746899d75-dtjhs" Nov 02 06:16:12.269751 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.269628 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fa9c5a91-d480-4d7b-9044-be6437a76a02-bound-sa-token\") pod \"image-registry-746899d75-dtjhs\" (UID: \"fa9c5a91-d480-4d7b-9044-be6437a76a02\") " pod="openshift-image-registry/image-registry-746899d75-dtjhs" Nov 02 06:16:12.269751 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.269658 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txffk\" (UniqueName: \"kubernetes.io/projected/dee3b502-2562-4b21-bc46-2031580c4b0a-kube-api-access-txffk\") pod \"kas-connection-checker-685895c7c9-jrtnt\" (UID: \"dee3b502-2562-4b21-bc46-2031580c4b0a\") " pod="kube-system/kas-connection-checker-685895c7c9-jrtnt" Nov 02 06:16:12.269751 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.269678 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/45e5ce56-5cab-4546-8b31-02fc06cfa0cc-default-certificate\") pod \"router-default-5f5c58667c-29wnm\" (UID: \"45e5ce56-5cab-4546-8b31-02fc06cfa0cc\") " pod="openshift-ingress/router-default-5f5c58667c-29wnm" Nov 02 06:16:12.269751 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.269712 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/3fcb7621-86c8-448f-b4fa-b5bd76be7356-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-2rs6c\" (UID: \"3fcb7621-86c8-448f-b4fa-b5bd76be7356\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2rs6c" Nov 02 06:16:12.270090 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.269775 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45e5ce56-5cab-4546-8b31-02fc06cfa0cc-metrics-certs\") pod \"router-default-5f5c58667c-29wnm\" (UID: \"45e5ce56-5cab-4546-8b31-02fc06cfa0cc\") " pod="openshift-ingress/router-default-5f5c58667c-29wnm" Nov 02 06:16:12.270090 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.269792 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvpvr\" (UniqueName: \"kubernetes.io/projected/45e5ce56-5cab-4546-8b31-02fc06cfa0cc-kube-api-access-xvpvr\") pod \"router-default-5f5c58667c-29wnm\" (UID: \"45e5ce56-5cab-4546-8b31-02fc06cfa0cc\") " pod="openshift-ingress/router-default-5f5c58667c-29wnm" Nov 02 06:16:12.270090 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.269830 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fa9c5a91-d480-4d7b-9044-be6437a76a02-registry-tls\") pod \"image-registry-746899d75-dtjhs\" (UID: \"fa9c5a91-d480-4d7b-9044-be6437a76a02\") " pod="openshift-image-registry/image-registry-746899d75-dtjhs" Nov 02 06:16:12.270090 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.269877 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fa9c5a91-d480-4d7b-9044-be6437a76a02-registry-certificates\") pod \"image-registry-746899d75-dtjhs\" (UID: \"fa9c5a91-d480-4d7b-9044-be6437a76a02\") " pod="openshift-image-registry/image-registry-746899d75-dtjhs" Nov 02 06:16:12.270090 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.269934 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5q7t\" (UniqueName: \"kubernetes.io/projected/7434256f-45c3-47ff-9111-352114b0f6d8-kube-api-access-l5q7t\") pod \"kas-connection-checker-685895c7c9-kb9wk\" (UID: \"7434256f-45c3-47ff-9111-352114b0f6d8\") " pod="kube-system/kas-connection-checker-685895c7c9-kb9wk" Nov 02 06:16:12.274975 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.274961 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-gf726"] Nov 02 06:16:12.275091 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.275076 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-k9hcs" Nov 02 06:16:12.285360 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.285344 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-6tt27"] Nov 02 06:16:12.285472 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.285458 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-gf726" Nov 02 06:16:12.293026 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.293008 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 02 06:16:12.293114 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.293069 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Nov 02 06:16:12.293176 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.293143 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Nov 02 06:16:12.293236 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.293205 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Nov 02 06:16:12.293272 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.293247 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Nov 02 06:16:12.293328 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.293316 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Nov 02 06:16:12.293545 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.293530 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Nov 02 06:16:12.293606 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.293554 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Nov 02 06:16:12.293606 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.293567 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Nov 02 06:16:12.293685 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.293605 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Nov 02 06:16:12.293685 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.293650 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Nov 02 06:16:12.293778 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.293764 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 02 06:16:12.293778 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.293775 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 02 06:16:12.293873 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.293786 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Nov 02 06:16:12.293873 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.293824 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Nov 02 06:16:12.293873 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.293842 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 02 06:16:12.293999 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.293983 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Nov 02 06:16:12.294106 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.294095 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Nov 02 06:16:12.297619 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.297604 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-kd68m"] Nov 02 06:16:12.297690 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.297629 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-kb9wk"] Nov 02 06:16:12.297690 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.297644 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-jrtnt"] Nov 02 06:16:12.297690 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.297659 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-8fbcn"] Nov 02 06:16:12.297690 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.297674 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-k9hcs"] Nov 02 06:16:12.297812 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.297693 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-wcrtl"] Nov 02 06:16:12.297812 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.297710 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-c8dcq"] Nov 02 06:16:12.297812 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.297708 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-6tt27" Nov 02 06:16:12.297943 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.297722 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-2rs6c"] Nov 02 06:16:12.297943 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.297882 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-csfhg"] Nov 02 06:16:12.298040 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.298010 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-d6bc4"] Nov 02 06:16:12.298040 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.298037 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-gf726"] Nov 02 06:16:12.298111 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.298054 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5f5c58667c-29wnm"] Nov 02 06:16:12.298111 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.298069 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-746899d75-dtjhs"] Nov 02 06:16:12.298111 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.298109 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6689c6d46b-vw558"] Nov 02 06:16:12.298216 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.298125 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-6tt27"] Nov 02 06:16:12.298216 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.298137 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-fh4sz"] Nov 02 06:16:12.298216 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.298153 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-sktt2"] Nov 02 06:16:12.342274 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.342241 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Nov 02 06:16:12.342447 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.342284 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Nov 02 06:16:12.342447 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.342309 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 02 06:16:12.342447 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.342309 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Nov 02 06:16:12.342447 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.342330 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Nov 02 06:16:12.342447 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.342344 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Nov 02 06:16:12.342447 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.342329 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Nov 02 06:16:12.342447 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.342412 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-nxd68\"" Nov 02 06:16:12.342447 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.342427 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Nov 02 06:16:12.343268 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.343237 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Nov 02 06:16:12.343375 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.343362 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Nov 02 06:16:12.370771 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.370745 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnw94\" (UniqueName: \"kubernetes.io/projected/121e38c0-6fa3-46aa-9191-5c6f91c8764c-kube-api-access-fnw94\") pod \"volume-data-source-validator-58c9f5f46d-8fbcn\" (UID: \"121e38c0-6fa3-46aa-9191-5c6f91c8764c\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-8fbcn" Nov 02 06:16:12.370839 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.370777 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/37982aa4-bdbd-40e8-a2dd-5094209f3c96-serving-cert\") pod \"insights-operator-7c4986d449-c8dcq\" (UID: \"37982aa4-bdbd-40e8-a2dd-5094209f3c96\") " pod="openshift-insights/insights-operator-7c4986d449-c8dcq" Nov 02 06:16:12.370839 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.370807 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwqbt\" (UniqueName: \"kubernetes.io/projected/cf8e8e65-8525-484b-adfe-f56517ccdc1b-kube-api-access-vwqbt\") pod \"cluster-samples-operator-6ddb6c8b94-d6bc4\" (UID: \"cf8e8e65-8525-484b-adfe-f56517ccdc1b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-d6bc4" Nov 02 06:16:12.370969 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.370863 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fa9c5a91-d480-4d7b-9044-be6437a76a02-bound-sa-token\") pod \"image-registry-746899d75-dtjhs\" (UID: \"fa9c5a91-d480-4d7b-9044-be6437a76a02\") " pod="openshift-image-registry/image-registry-746899d75-dtjhs" Nov 02 06:16:12.370969 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.370921 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/37982aa4-bdbd-40e8-a2dd-5094209f3c96-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-c8dcq\" (UID: \"37982aa4-bdbd-40e8-a2dd-5094209f3c96\") " pod="openshift-insights/insights-operator-7c4986d449-c8dcq" Nov 02 06:16:12.370969 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.370955 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4ba9b2b6-4be9-4303-8eab-cd464c5a32c7-config-volume\") pod \"dns-default-gf726\" (UID: \"4ba9b2b6-4be9-4303-8eab-cd464c5a32c7\") " pod="openshift-dns/dns-default-gf726" Nov 02 06:16:12.371091 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.370987 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txffk\" (UniqueName: \"kubernetes.io/projected/dee3b502-2562-4b21-bc46-2031580c4b0a-kube-api-access-txffk\") pod \"kas-connection-checker-685895c7c9-jrtnt\" (UID: \"dee3b502-2562-4b21-bc46-2031580c4b0a\") " pod="kube-system/kas-connection-checker-685895c7c9-jrtnt" Nov 02 06:16:12.371091 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.371028 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/17dee72b-2f47-4016-8ffe-3a89934e66ee-config\") pod \"console-operator-7cd56b8dbc-csfhg\" (UID: \"17dee72b-2f47-4016-8ffe-3a89934e66ee\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-csfhg" Nov 02 06:16:12.371187 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.371123 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0d2b53c4-9d39-47fc-9367-c9a8e7f5da81-cert\") pod \"ingress-canary-6tt27\" (UID: \"0d2b53c4-9d39-47fc-9367-c9a8e7f5da81\") " pod="openshift-ingress-canary/ingress-canary-6tt27" Nov 02 06:16:12.371187 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.371160 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cc33996a-0d4c-49c4-ba67-283bd6a2de14-serving-cert\") pod \"service-ca-operator-544d749c8c-wcrtl\" (UID: \"cc33996a-0d4c-49c4-ba67-283bd6a2de14\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wcrtl" Nov 02 06:16:12.371286 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.371198 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fa9c5a91-d480-4d7b-9044-be6437a76a02-registry-tls\") pod \"image-registry-746899d75-dtjhs\" (UID: \"fa9c5a91-d480-4d7b-9044-be6437a76a02\") " pod="openshift-image-registry/image-registry-746899d75-dtjhs" Nov 02 06:16:12.371286 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.371226 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f0531b4d-c954-4830-85f6-fe876c76437a-registry-certificates\") pod \"image-registry-6689c6d46b-vw558\" (UID: \"f0531b4d-c954-4830-85f6-fe876c76437a\") " pod="openshift-image-registry/image-registry-6689c6d46b-vw558" Nov 02 06:16:12.371286 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.371257 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f0531b4d-c954-4830-85f6-fe876c76437a-bound-sa-token\") pod \"image-registry-6689c6d46b-vw558\" (UID: \"f0531b4d-c954-4830-85f6-fe876c76437a\") " pod="openshift-image-registry/image-registry-6689c6d46b-vw558" Nov 02 06:16:12.371426 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.371283 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/3fcb7621-86c8-448f-b4fa-b5bd76be7356-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-2rs6c\" (UID: \"3fcb7621-86c8-448f-b4fa-b5bd76be7356\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2rs6c" Nov 02 06:16:12.371426 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:16:12.371287 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 06:16:12.371426 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:16:12.371318 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-746899d75-dtjhs: secret "image-registry-tls" not found Nov 02 06:16:12.371426 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:16:12.371371 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fa9c5a91-d480-4d7b-9044-be6437a76a02-registry-tls podName:fa9c5a91-d480-4d7b-9044-be6437a76a02 nodeName:}" failed. No retries permitted until 2026-11-02 06:16:12.871352231 +0000 UTC m=+34.763170491 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fa9c5a91-d480-4d7b-9044-be6437a76a02-registry-tls") pod "image-registry-746899d75-dtjhs" (UID: "fa9c5a91-d480-4d7b-9044-be6437a76a02") : secret "image-registry-tls" not found Nov 02 06:16:12.371609 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.371451 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7l5t\" (UniqueName: \"kubernetes.io/projected/37982aa4-bdbd-40e8-a2dd-5094209f3c96-kube-api-access-h7l5t\") pod \"insights-operator-7c4986d449-c8dcq\" (UID: \"37982aa4-bdbd-40e8-a2dd-5094209f3c96\") " pod="openshift-insights/insights-operator-7c4986d449-c8dcq" Nov 02 06:16:12.371609 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.371480 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/12b6cc1c-537d-4125-92ff-fe5f6927c53e-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-sktt2\" (UID: \"12b6cc1c-537d-4125-92ff-fe5f6927c53e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-sktt2" Nov 02 06:16:12.371609 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.371497 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmsjq\" (UniqueName: \"kubernetes.io/projected/12b6cc1c-537d-4125-92ff-fe5f6927c53e-kube-api-access-bmsjq\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-sktt2\" (UID: \"12b6cc1c-537d-4125-92ff-fe5f6927c53e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-sktt2" Nov 02 06:16:12.371609 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.371537 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tz6t\" (UniqueName: \"kubernetes.io/projected/17dee72b-2f47-4016-8ffe-3a89934e66ee-kube-api-access-9tz6t\") pod \"console-operator-7cd56b8dbc-csfhg\" (UID: \"17dee72b-2f47-4016-8ffe-3a89934e66ee\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-csfhg" Nov 02 06:16:12.371609 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.371587 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ba9b2b6-4be9-4303-8eab-cd464c5a32c7-tmp-dir\") pod \"dns-default-gf726\" (UID: \"4ba9b2b6-4be9-4303-8eab-cd464c5a32c7\") " pod="openshift-dns/dns-default-gf726" Nov 02 06:16:12.371843 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.371624 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f0531b4d-c954-4830-85f6-fe876c76437a-installation-pull-secrets\") pod \"image-registry-6689c6d46b-vw558\" (UID: \"f0531b4d-c954-4830-85f6-fe876c76437a\") " pod="openshift-image-registry/image-registry-6689c6d46b-vw558" Nov 02 06:16:12.371843 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.371737 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/53aa1e5a-c35f-4aa6-9f23-cc61ccd331fc-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-fh4sz\" (UID: \"53aa1e5a-c35f-4aa6-9f23-cc61ccd331fc\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-fh4sz" Nov 02 06:16:12.371843 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.371760 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/37982aa4-bdbd-40e8-a2dd-5094209f3c96-snapshots\") pod \"insights-operator-7c4986d449-c8dcq\" (UID: \"37982aa4-bdbd-40e8-a2dd-5094209f3c96\") " pod="openshift-insights/insights-operator-7c4986d449-c8dcq" Nov 02 06:16:12.371843 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.371777 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rd8hq\" (UniqueName: \"kubernetes.io/projected/cc33996a-0d4c-49c4-ba67-283bd6a2de14-kube-api-access-rd8hq\") pod \"service-ca-operator-544d749c8c-wcrtl\" (UID: \"cc33996a-0d4c-49c4-ba67-283bd6a2de14\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wcrtl" Nov 02 06:16:12.371843 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.371794 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dv9xj\" (UniqueName: \"kubernetes.io/projected/4ba9b2b6-4be9-4303-8eab-cd464c5a32c7-kube-api-access-dv9xj\") pod \"dns-default-gf726\" (UID: \"4ba9b2b6-4be9-4303-8eab-cd464c5a32c7\") " pod="openshift-dns/dns-default-gf726" Nov 02 06:16:12.371843 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.371819 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f0531b4d-c954-4830-85f6-fe876c76437a-registry-certificates\") pod \"image-registry-6689c6d46b-vw558\" (UID: \"f0531b4d-c954-4830-85f6-fe876c76437a\") " pod="openshift-image-registry/image-registry-6689c6d46b-vw558" Nov 02 06:16:12.371843 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.371836 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/cf8e8e65-8525-484b-adfe-f56517ccdc1b-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-d6bc4\" (UID: \"cf8e8e65-8525-484b-adfe-f56517ccdc1b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-d6bc4" Nov 02 06:16:12.372178 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.371857 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/3fcb7621-86c8-448f-b4fa-b5bd76be7356-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-2rs6c\" (UID: \"3fcb7621-86c8-448f-b4fa-b5bd76be7356\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2rs6c" Nov 02 06:16:12.372178 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.371939 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cctkp\" (UniqueName: \"kubernetes.io/projected/fa9c5a91-d480-4d7b-9044-be6437a76a02-kube-api-access-cctkp\") pod \"image-registry-746899d75-dtjhs\" (UID: \"fa9c5a91-d480-4d7b-9044-be6437a76a02\") " pod="openshift-image-registry/image-registry-746899d75-dtjhs" Nov 02 06:16:12.372178 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.371973 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwh5f\" (UniqueName: \"kubernetes.io/projected/3fcb7621-86c8-448f-b4fa-b5bd76be7356-kube-api-access-mwh5f\") pod \"cluster-monitoring-operator-884576c57-2rs6c\" (UID: \"3fcb7621-86c8-448f-b4fa-b5bd76be7356\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2rs6c" Nov 02 06:16:12.372178 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.372015 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/17dee72b-2f47-4016-8ffe-3a89934e66ee-serving-cert\") pod \"console-operator-7cd56b8dbc-csfhg\" (UID: \"17dee72b-2f47-4016-8ffe-3a89934e66ee\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-csfhg" Nov 02 06:16:12.372178 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.372081 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45e5ce56-5cab-4546-8b31-02fc06cfa0cc-service-ca-bundle\") pod \"router-default-5f5c58667c-29wnm\" (UID: \"45e5ce56-5cab-4546-8b31-02fc06cfa0cc\") " pod="openshift-ingress/router-default-5f5c58667c-29wnm" Nov 02 06:16:12.372178 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.372104 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3fcb7621-86c8-448f-b4fa-b5bd76be7356-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-2rs6c\" (UID: \"3fcb7621-86c8-448f-b4fa-b5bd76be7356\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2rs6c" Nov 02 06:16:12.372178 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.372135 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fa9c5a91-d480-4d7b-9044-be6437a76a02-trusted-ca\") pod \"image-registry-746899d75-dtjhs\" (UID: \"fa9c5a91-d480-4d7b-9044-be6437a76a02\") " pod="openshift-image-registry/image-registry-746899d75-dtjhs" Nov 02 06:16:12.372178 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.372174 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5989k\" (UniqueName: \"kubernetes.io/projected/20cb7690-c976-471c-b459-a9ab15cc51bf-kube-api-access-5989k\") pod \"network-check-source-6f78d59f99-k9hcs\" (UID: \"20cb7690-c976-471c-b459-a9ab15cc51bf\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-k9hcs" Nov 02 06:16:12.372481 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:16:12.372188 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/45e5ce56-5cab-4546-8b31-02fc06cfa0cc-service-ca-bundle podName:45e5ce56-5cab-4546-8b31-02fc06cfa0cc nodeName:}" failed. No retries permitted until 2026-11-02 06:16:12.872168155 +0000 UTC m=+34.763986395 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/45e5ce56-5cab-4546-8b31-02fc06cfa0cc-service-ca-bundle") pod "router-default-5f5c58667c-29wnm" (UID: "45e5ce56-5cab-4546-8b31-02fc06cfa0cc") : configmap references non-existent config key: service-ca.crt Nov 02 06:16:12.372481 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.372230 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/45e5ce56-5cab-4546-8b31-02fc06cfa0cc-stats-auth\") pod \"router-default-5f5c58667c-29wnm\" (UID: \"45e5ce56-5cab-4546-8b31-02fc06cfa0cc\") " pod="openshift-ingress/router-default-5f5c58667c-29wnm" Nov 02 06:16:12.372481 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:16:12.372238 2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 02 06:16:12.372481 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:16:12.372290 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3fcb7621-86c8-448f-b4fa-b5bd76be7356-cluster-monitoring-operator-tls podName:3fcb7621-86c8-448f-b4fa-b5bd76be7356 nodeName:}" failed. No retries permitted until 2026-11-02 06:16:12.872273109 +0000 UTC m=+34.764091358 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/3fcb7621-86c8-448f-b4fa-b5bd76be7356-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-2rs6c" (UID: "3fcb7621-86c8-448f-b4fa-b5bd76be7356") : secret "cluster-monitoring-operator-tls" not found Nov 02 06:16:12.373248 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.373191 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fa9c5a91-d480-4d7b-9044-be6437a76a02-ca-trust-extracted\") pod \"image-registry-746899d75-dtjhs\" (UID: \"fa9c5a91-d480-4d7b-9044-be6437a76a02\") " pod="openshift-image-registry/image-registry-746899d75-dtjhs" Nov 02 06:16:12.373347 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.373323 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/45e5ce56-5cab-4546-8b31-02fc06cfa0cc-default-certificate\") pod \"router-default-5f5c58667c-29wnm\" (UID: \"45e5ce56-5cab-4546-8b31-02fc06cfa0cc\") " pod="openshift-ingress/router-default-5f5c58667c-29wnm" Nov 02 06:16:12.373404 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.373367 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/cc33996a-0d4c-49c4-ba67-283bd6a2de14-config\") pod \"service-ca-operator-544d749c8c-wcrtl\" (UID: \"cc33996a-0d4c-49c4-ba67-283bd6a2de14\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wcrtl" Nov 02 06:16:12.373460 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.373404 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/cf8e8e65-8525-484b-adfe-f56517ccdc1b-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-d6bc4\" (UID: \"cf8e8e65-8525-484b-adfe-f56517ccdc1b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-d6bc4" Nov 02 06:16:12.373460 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.373409 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fa9c5a91-d480-4d7b-9044-be6437a76a02-ca-trust-extracted\") pod \"image-registry-746899d75-dtjhs\" (UID: \"fa9c5a91-d480-4d7b-9044-be6437a76a02\") " pod="openshift-image-registry/image-registry-746899d75-dtjhs" Nov 02 06:16:12.373668 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.373621 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m89g8\" (UniqueName: \"kubernetes.io/projected/f0531b4d-c954-4830-85f6-fe876c76437a-kube-api-access-m89g8\") pod \"image-registry-6689c6d46b-vw558\" (UID: \"f0531b4d-c954-4830-85f6-fe876c76437a\") " pod="openshift-image-registry/image-registry-6689c6d46b-vw558" Nov 02 06:16:12.373801 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.373778 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45e5ce56-5cab-4546-8b31-02fc06cfa0cc-metrics-certs\") pod \"router-default-5f5c58667c-29wnm\" (UID: \"45e5ce56-5cab-4546-8b31-02fc06cfa0cc\") " pod="openshift-ingress/router-default-5f5c58667c-29wnm" Nov 02 06:16:12.373921 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.373814 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mplc\" (UniqueName: \"kubernetes.io/projected/0d2b53c4-9d39-47fc-9367-c9a8e7f5da81-kube-api-access-8mplc\") pod \"ingress-canary-6tt27\" (UID: \"0d2b53c4-9d39-47fc-9367-c9a8e7f5da81\") " pod="openshift-ingress-canary/ingress-canary-6tt27" Nov 02 06:16:12.373921 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.373885 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvpvr\" (UniqueName: \"kubernetes.io/projected/45e5ce56-5cab-4546-8b31-02fc06cfa0cc-kube-api-access-xvpvr\") pod \"router-default-5f5c58667c-29wnm\" (UID: \"45e5ce56-5cab-4546-8b31-02fc06cfa0cc\") " pod="openshift-ingress/router-default-5f5c58667c-29wnm" Nov 02 06:16:12.374030 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.373944 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fa9c5a91-d480-4d7b-9044-be6437a76a02-registry-certificates\") pod \"image-registry-746899d75-dtjhs\" (UID: \"fa9c5a91-d480-4d7b-9044-be6437a76a02\") " pod="openshift-image-registry/image-registry-746899d75-dtjhs" Nov 02 06:16:12.374030 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.374020 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5q7t\" (UniqueName: \"kubernetes.io/projected/7434256f-45c3-47ff-9111-352114b0f6d8-kube-api-access-l5q7t\") pod \"kas-connection-checker-685895c7c9-kb9wk\" (UID: \"7434256f-45c3-47ff-9111-352114b0f6d8\") " pod="kube-system/kas-connection-checker-685895c7c9-kb9wk" Nov 02 06:16:12.374121 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.374059 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f0531b4d-c954-4830-85f6-fe876c76437a-ca-trust-extracted\") pod \"image-registry-6689c6d46b-vw558\" (UID: \"f0531b4d-c954-4830-85f6-fe876c76437a\") " pod="openshift-image-registry/image-registry-6689c6d46b-vw558" Nov 02 06:16:12.374173 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.374119 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4ba9b2b6-4be9-4303-8eab-cd464c5a32c7-metrics-tls\") pod \"dns-default-gf726\" (UID: \"4ba9b2b6-4be9-4303-8eab-cd464c5a32c7\") " pod="openshift-dns/dns-default-gf726" Nov 02 06:16:12.374221 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.374173 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fa9c5a91-d480-4d7b-9044-be6437a76a02-trusted-ca\") pod \"image-registry-746899d75-dtjhs\" (UID: \"fa9c5a91-d480-4d7b-9044-be6437a76a02\") " pod="openshift-image-registry/image-registry-746899d75-dtjhs" Nov 02 06:16:12.374221 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.374185 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fa9c5a91-d480-4d7b-9044-be6437a76a02-installation-pull-secrets\") pod \"image-registry-746899d75-dtjhs\" (UID: \"fa9c5a91-d480-4d7b-9044-be6437a76a02\") " pod="openshift-image-registry/image-registry-746899d75-dtjhs" Nov 02 06:16:12.374315 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.374248 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/37982aa4-bdbd-40e8-a2dd-5094209f3c96-service-ca-bundle\") pod \"insights-operator-7c4986d449-c8dcq\" (UID: \"37982aa4-bdbd-40e8-a2dd-5094209f3c96\") " pod="openshift-insights/insights-operator-7c4986d449-c8dcq" Nov 02 06:16:12.374362 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.374305 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fa9c5a91-d480-4d7b-9044-be6437a76a02-image-registry-private-configuration\") pod \"image-registry-746899d75-dtjhs\" (UID: \"fa9c5a91-d480-4d7b-9044-be6437a76a02\") " pod="openshift-image-registry/image-registry-746899d75-dtjhs" Nov 02 06:16:12.374410 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.374359 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f0531b4d-c954-4830-85f6-fe876c76437a-trusted-ca\") pod \"image-registry-6689c6d46b-vw558\" (UID: \"f0531b4d-c954-4830-85f6-fe876c76437a\") " pod="openshift-image-registry/image-registry-6689c6d46b-vw558" Nov 02 06:16:12.374696 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.374409 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/17dee72b-2f47-4016-8ffe-3a89934e66ee-trusted-ca\") pod \"console-operator-7cd56b8dbc-csfhg\" (UID: \"17dee72b-2f47-4016-8ffe-3a89934e66ee\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-csfhg" Nov 02 06:16:12.374696 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:16:12.374622 2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 02 06:16:12.374696 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.374661 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/37982aa4-bdbd-40e8-a2dd-5094209f3c96-tmp\") pod \"insights-operator-7c4986d449-c8dcq\" (UID: \"37982aa4-bdbd-40e8-a2dd-5094209f3c96\") " pod="openshift-insights/insights-operator-7c4986d449-c8dcq" Nov 02 06:16:12.374696 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:16:12.374686 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45e5ce56-5cab-4546-8b31-02fc06cfa0cc-metrics-certs podName:45e5ce56-5cab-4546-8b31-02fc06cfa0cc nodeName:}" failed. No retries permitted until 2026-11-02 06:16:12.874671434 +0000 UTC m=+34.766489691 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/45e5ce56-5cab-4546-8b31-02fc06cfa0cc-metrics-certs") pod "router-default-5f5c58667c-29wnm" (UID: "45e5ce56-5cab-4546-8b31-02fc06cfa0cc") : secret "router-metrics-certs-default" not found Nov 02 06:16:12.374920 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.374746 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/12b6cc1c-537d-4125-92ff-fe5f6927c53e-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-sktt2\" (UID: \"12b6cc1c-537d-4125-92ff-fe5f6927c53e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-sktt2" Nov 02 06:16:12.374920 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.374794 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrh7x\" (UniqueName: \"kubernetes.io/projected/8e8f2e60-de1b-4c8b-9285-56568c150d92-kube-api-access-zrh7x\") pod \"kas-connection-checker-685895c7c9-kd68m\" (UID: \"8e8f2e60-de1b-4c8b-9285-56568c150d92\") " pod="kube-system/kas-connection-checker-685895c7c9-kd68m" Nov 02 06:16:12.374920 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.374873 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f0531b4d-c954-4830-85f6-fe876c76437a-registry-tls\") pod \"image-registry-6689c6d46b-vw558\" (UID: \"f0531b4d-c954-4830-85f6-fe876c76437a\") " pod="openshift-image-registry/image-registry-6689c6d46b-vw558" Nov 02 06:16:12.375067 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.374947 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/53aa1e5a-c35f-4aa6-9f23-cc61ccd331fc-nginx-conf\") pod \"networking-console-plugin-86c478bb88-fh4sz\" (UID: \"53aa1e5a-c35f-4aa6-9f23-cc61ccd331fc\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-fh4sz" Nov 02 06:16:12.375067 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:16:12.375034 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 06:16:12.375067 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:16:12.375053 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6689c6d46b-vw558: secret "image-registry-tls" not found Nov 02 06:16:12.375195 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:16:12.375106 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f0531b4d-c954-4830-85f6-fe876c76437a-registry-tls podName:f0531b4d-c954-4830-85f6-fe876c76437a nodeName:}" failed. No retries permitted until 2026-11-02 06:16:12.875083797 +0000 UTC m=+34.766902057 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f0531b4d-c954-4830-85f6-fe876c76437a-registry-tls") pod "image-registry-6689c6d46b-vw558" (UID: "f0531b4d-c954-4830-85f6-fe876c76437a") : secret "image-registry-tls" not found Nov 02 06:16:12.375415 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.375396 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f0531b4d-c954-4830-85f6-fe876c76437a-image-registry-private-configuration\") pod \"image-registry-6689c6d46b-vw558\" (UID: \"f0531b4d-c954-4830-85f6-fe876c76437a\") " pod="openshift-image-registry/image-registry-6689c6d46b-vw558" Nov 02 06:16:12.375662 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.375641 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f0531b4d-c954-4830-85f6-fe876c76437a-ca-trust-extracted\") pod \"image-registry-6689c6d46b-vw558\" (UID: \"f0531b4d-c954-4830-85f6-fe876c76437a\") " pod="openshift-image-registry/image-registry-6689c6d46b-vw558" Nov 02 06:16:12.379691 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.376471 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fa9c5a91-d480-4d7b-9044-be6437a76a02-registry-certificates\") pod \"image-registry-746899d75-dtjhs\" (UID: \"fa9c5a91-d480-4d7b-9044-be6437a76a02\") " pod="openshift-image-registry/image-registry-746899d75-dtjhs" Nov 02 06:16:12.379691 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.376478 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f0531b4d-c954-4830-85f6-fe876c76437a-installation-pull-secrets\") pod \"image-registry-6689c6d46b-vw558\" (UID: \"f0531b4d-c954-4830-85f6-fe876c76437a\") " pod="openshift-image-registry/image-registry-6689c6d46b-vw558" Nov 02 06:16:12.379691 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.376889 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/45e5ce56-5cab-4546-8b31-02fc06cfa0cc-stats-auth\") pod \"router-default-5f5c58667c-29wnm\" (UID: \"45e5ce56-5cab-4546-8b31-02fc06cfa0cc\") " pod="openshift-ingress/router-default-5f5c58667c-29wnm" Nov 02 06:16:12.379691 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.377835 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f0531b4d-c954-4830-85f6-fe876c76437a-trusted-ca\") pod \"image-registry-6689c6d46b-vw558\" (UID: \"f0531b4d-c954-4830-85f6-fe876c76437a\") " pod="openshift-image-registry/image-registry-6689c6d46b-vw558" Nov 02 06:16:12.379691 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.377857 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/45e5ce56-5cab-4546-8b31-02fc06cfa0cc-default-certificate\") pod \"router-default-5f5c58667c-29wnm\" (UID: \"45e5ce56-5cab-4546-8b31-02fc06cfa0cc\") " pod="openshift-ingress/router-default-5f5c58667c-29wnm" Nov 02 06:16:12.379691 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.378053 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fa9c5a91-d480-4d7b-9044-be6437a76a02-installation-pull-secrets\") pod \"image-registry-746899d75-dtjhs\" (UID: \"fa9c5a91-d480-4d7b-9044-be6437a76a02\") " pod="openshift-image-registry/image-registry-746899d75-dtjhs" Nov 02 06:16:12.379691 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.378310 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fa9c5a91-d480-4d7b-9044-be6437a76a02-image-registry-private-configuration\") pod \"image-registry-746899d75-dtjhs\" (UID: \"fa9c5a91-d480-4d7b-9044-be6437a76a02\") " pod="openshift-image-registry/image-registry-746899d75-dtjhs" Nov 02 06:16:12.379691 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.379000 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f0531b4d-c954-4830-85f6-fe876c76437a-image-registry-private-configuration\") pod \"image-registry-6689c6d46b-vw558\" (UID: \"f0531b4d-c954-4830-85f6-fe876c76437a\") " pod="openshift-image-registry/image-registry-6689c6d46b-vw558" Nov 02 06:16:12.393006 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.392987 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Nov 02 06:16:12.393319 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.393232 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Nov 02 06:16:12.393319 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.393264 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Nov 02 06:16:12.393319 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.393295 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 02 06:16:12.393319 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.393316 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 02 06:16:12.393522 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.393315 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 02 06:16:12.393522 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.393336 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 02 06:16:12.393522 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.393315 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Nov 02 06:16:12.393670 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.393651 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 02 06:16:12.393731 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.393721 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 02 06:16:12.393793 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.393761 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 02 06:16:12.393928 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.393899 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Nov 02 06:16:12.430666 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.430642 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fa9c5a91-d480-4d7b-9044-be6437a76a02-bound-sa-token\") pod \"image-registry-746899d75-dtjhs\" (UID: \"fa9c5a91-d480-4d7b-9044-be6437a76a02\") " pod="openshift-image-registry/image-registry-746899d75-dtjhs" Nov 02 06:16:12.430772 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.430721 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-txffk\" (UniqueName: \"kubernetes.io/projected/dee3b502-2562-4b21-bc46-2031580c4b0a-kube-api-access-txffk\") pod \"kas-connection-checker-685895c7c9-jrtnt\" (UID: \"dee3b502-2562-4b21-bc46-2031580c4b0a\") " pod="kube-system/kas-connection-checker-685895c7c9-jrtnt" Nov 02 06:16:12.430842 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.430791 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f0531b4d-c954-4830-85f6-fe876c76437a-bound-sa-token\") pod \"image-registry-6689c6d46b-vw558\" (UID: \"f0531b4d-c954-4830-85f6-fe876c76437a\") " pod="openshift-image-registry/image-registry-6689c6d46b-vw558" Nov 02 06:16:12.433126 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.433099 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvpvr\" (UniqueName: \"kubernetes.io/projected/45e5ce56-5cab-4546-8b31-02fc06cfa0cc-kube-api-access-xvpvr\") pod \"router-default-5f5c58667c-29wnm\" (UID: \"45e5ce56-5cab-4546-8b31-02fc06cfa0cc\") " pod="openshift-ingress/router-default-5f5c58667c-29wnm" Nov 02 06:16:12.433229 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.433169 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5q7t\" (UniqueName: \"kubernetes.io/projected/7434256f-45c3-47ff-9111-352114b0f6d8-kube-api-access-l5q7t\") pod \"kas-connection-checker-685895c7c9-kb9wk\" (UID: \"7434256f-45c3-47ff-9111-352114b0f6d8\") " pod="kube-system/kas-connection-checker-685895c7c9-kb9wk" Nov 02 06:16:12.433805 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.433779 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwh5f\" (UniqueName: \"kubernetes.io/projected/3fcb7621-86c8-448f-b4fa-b5bd76be7356-kube-api-access-mwh5f\") pod \"cluster-monitoring-operator-884576c57-2rs6c\" (UID: \"3fcb7621-86c8-448f-b4fa-b5bd76be7356\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2rs6c" Nov 02 06:16:12.433881 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.433856 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cctkp\" (UniqueName: \"kubernetes.io/projected/fa9c5a91-d480-4d7b-9044-be6437a76a02-kube-api-access-cctkp\") pod \"image-registry-746899d75-dtjhs\" (UID: \"fa9c5a91-d480-4d7b-9044-be6437a76a02\") " pod="openshift-image-registry/image-registry-746899d75-dtjhs" Nov 02 06:16:12.433881 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.433861 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m89g8\" (UniqueName: \"kubernetes.io/projected/f0531b4d-c954-4830-85f6-fe876c76437a-kube-api-access-m89g8\") pod \"image-registry-6689c6d46b-vw558\" (UID: \"f0531b4d-c954-4830-85f6-fe876c76437a\") " pod="openshift-image-registry/image-registry-6689c6d46b-vw558" Nov 02 06:16:12.434417 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.434401 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrh7x\" (UniqueName: \"kubernetes.io/projected/8e8f2e60-de1b-4c8b-9285-56568c150d92-kube-api-access-zrh7x\") pod \"kas-connection-checker-685895c7c9-kd68m\" (UID: \"8e8f2e60-de1b-4c8b-9285-56568c150d92\") " pod="kube-system/kas-connection-checker-685895c7c9-kd68m" Nov 02 06:16:12.440841 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.440823 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-rkkc6\"" Nov 02 06:16:12.440961 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.440939 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 02 06:16:12.443100 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.443079 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-kd68m" Nov 02 06:16:12.458207 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.458160 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-kb9wk" Nov 02 06:16:12.467968 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.467948 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-jrtnt" Nov 02 06:16:12.477291 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.477272 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4ba9b2b6-4be9-4303-8eab-cd464c5a32c7-metrics-tls\") pod \"dns-default-gf726\" (UID: \"4ba9b2b6-4be9-4303-8eab-cd464c5a32c7\") " pod="openshift-dns/dns-default-gf726" Nov 02 06:16:12.477336 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.477304 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/37982aa4-bdbd-40e8-a2dd-5094209f3c96-service-ca-bundle\") pod \"insights-operator-7c4986d449-c8dcq\" (UID: \"37982aa4-bdbd-40e8-a2dd-5094209f3c96\") " pod="openshift-insights/insights-operator-7c4986d449-c8dcq" Nov 02 06:16:12.477336 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.477324 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/17dee72b-2f47-4016-8ffe-3a89934e66ee-trusted-ca\") pod \"console-operator-7cd56b8dbc-csfhg\" (UID: \"17dee72b-2f47-4016-8ffe-3a89934e66ee\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-csfhg" Nov 02 06:16:12.477407 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.477346 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/37982aa4-bdbd-40e8-a2dd-5094209f3c96-tmp\") pod \"insights-operator-7c4986d449-c8dcq\" (UID: \"37982aa4-bdbd-40e8-a2dd-5094209f3c96\") " pod="openshift-insights/insights-operator-7c4986d449-c8dcq" Nov 02 06:16:12.477407 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.477371 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/12b6cc1c-537d-4125-92ff-fe5f6927c53e-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-sktt2\" (UID: \"12b6cc1c-537d-4125-92ff-fe5f6927c53e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-sktt2" Nov 02 06:16:12.477496 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.477413 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/53aa1e5a-c35f-4aa6-9f23-cc61ccd331fc-nginx-conf\") pod \"networking-console-plugin-86c478bb88-fh4sz\" (UID: \"53aa1e5a-c35f-4aa6-9f23-cc61ccd331fc\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-fh4sz" Nov 02 06:16:12.477496 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:16:12.477421 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 06:16:12.477496 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.477444 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnw94\" (UniqueName: \"kubernetes.io/projected/121e38c0-6fa3-46aa-9191-5c6f91c8764c-kube-api-access-fnw94\") pod \"volume-data-source-validator-58c9f5f46d-8fbcn\" (UID: \"121e38c0-6fa3-46aa-9191-5c6f91c8764c\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-8fbcn" Nov 02 06:16:12.477496 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:16:12.477473 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ba9b2b6-4be9-4303-8eab-cd464c5a32c7-metrics-tls podName:4ba9b2b6-4be9-4303-8eab-cd464c5a32c7 nodeName:}" failed. No retries permitted until 2026-11-02 06:16:12.977454713 +0000 UTC m=+34.869272973 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4ba9b2b6-4be9-4303-8eab-cd464c5a32c7-metrics-tls") pod "dns-default-gf726" (UID: "4ba9b2b6-4be9-4303-8eab-cd464c5a32c7") : secret "dns-default-metrics-tls" not found Nov 02 06:16:12.477690 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.477503 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/37982aa4-bdbd-40e8-a2dd-5094209f3c96-serving-cert\") pod \"insights-operator-7c4986d449-c8dcq\" (UID: \"37982aa4-bdbd-40e8-a2dd-5094209f3c96\") " pod="openshift-insights/insights-operator-7c4986d449-c8dcq" Nov 02 06:16:12.477690 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.477545 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwqbt\" (UniqueName: \"kubernetes.io/projected/cf8e8e65-8525-484b-adfe-f56517ccdc1b-kube-api-access-vwqbt\") pod \"cluster-samples-operator-6ddb6c8b94-d6bc4\" (UID: \"cf8e8e65-8525-484b-adfe-f56517ccdc1b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-d6bc4" Nov 02 06:16:12.477690 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.477572 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/37982aa4-bdbd-40e8-a2dd-5094209f3c96-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-c8dcq\" (UID: \"37982aa4-bdbd-40e8-a2dd-5094209f3c96\") " pod="openshift-insights/insights-operator-7c4986d449-c8dcq" Nov 02 06:16:12.477690 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.477595 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4ba9b2b6-4be9-4303-8eab-cd464c5a32c7-config-volume\") pod \"dns-default-gf726\" (UID: \"4ba9b2b6-4be9-4303-8eab-cd464c5a32c7\") " pod="openshift-dns/dns-default-gf726" Nov 02 06:16:12.477690 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.477620 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/17dee72b-2f47-4016-8ffe-3a89934e66ee-config\") pod \"console-operator-7cd56b8dbc-csfhg\" (UID: \"17dee72b-2f47-4016-8ffe-3a89934e66ee\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-csfhg" Nov 02 06:16:12.477690 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.477643 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0d2b53c4-9d39-47fc-9367-c9a8e7f5da81-cert\") pod \"ingress-canary-6tt27\" (UID: \"0d2b53c4-9d39-47fc-9367-c9a8e7f5da81\") " pod="openshift-ingress-canary/ingress-canary-6tt27" Nov 02 06:16:12.477690 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.477665 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cc33996a-0d4c-49c4-ba67-283bd6a2de14-serving-cert\") pod \"service-ca-operator-544d749c8c-wcrtl\" (UID: \"cc33996a-0d4c-49c4-ba67-283bd6a2de14\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wcrtl" Nov 02 06:16:12.478035 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.477714 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7l5t\" (UniqueName: \"kubernetes.io/projected/37982aa4-bdbd-40e8-a2dd-5094209f3c96-kube-api-access-h7l5t\") pod \"insights-operator-7c4986d449-c8dcq\" (UID: \"37982aa4-bdbd-40e8-a2dd-5094209f3c96\") " pod="openshift-insights/insights-operator-7c4986d449-c8dcq" Nov 02 06:16:12.478035 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.477746 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/12b6cc1c-537d-4125-92ff-fe5f6927c53e-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-sktt2\" (UID: \"12b6cc1c-537d-4125-92ff-fe5f6927c53e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-sktt2" Nov 02 06:16:12.478035 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.477793 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmsjq\" (UniqueName: \"kubernetes.io/projected/12b6cc1c-537d-4125-92ff-fe5f6927c53e-kube-api-access-bmsjq\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-sktt2\" (UID: \"12b6cc1c-537d-4125-92ff-fe5f6927c53e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-sktt2" Nov 02 06:16:12.478035 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.477826 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/37982aa4-bdbd-40e8-a2dd-5094209f3c96-tmp\") pod \"insights-operator-7c4986d449-c8dcq\" (UID: \"37982aa4-bdbd-40e8-a2dd-5094209f3c96\") " pod="openshift-insights/insights-operator-7c4986d449-c8dcq" Nov 02 06:16:12.478035 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.477837 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tz6t\" (UniqueName: \"kubernetes.io/projected/17dee72b-2f47-4016-8ffe-3a89934e66ee-kube-api-access-9tz6t\") pod \"console-operator-7cd56b8dbc-csfhg\" (UID: \"17dee72b-2f47-4016-8ffe-3a89934e66ee\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-csfhg" Nov 02 06:16:12.478035 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.477860 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ba9b2b6-4be9-4303-8eab-cd464c5a32c7-tmp-dir\") pod \"dns-default-gf726\" (UID: \"4ba9b2b6-4be9-4303-8eab-cd464c5a32c7\") " pod="openshift-dns/dns-default-gf726" Nov 02 06:16:12.478035 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.477892 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/53aa1e5a-c35f-4aa6-9f23-cc61ccd331fc-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-fh4sz\" (UID: \"53aa1e5a-c35f-4aa6-9f23-cc61ccd331fc\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-fh4sz" Nov 02 06:16:12.478035 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.477945 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/37982aa4-bdbd-40e8-a2dd-5094209f3c96-snapshots\") pod \"insights-operator-7c4986d449-c8dcq\" (UID: \"37982aa4-bdbd-40e8-a2dd-5094209f3c96\") " pod="openshift-insights/insights-operator-7c4986d449-c8dcq" Nov 02 06:16:12.478035 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.477971 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rd8hq\" (UniqueName: \"kubernetes.io/projected/cc33996a-0d4c-49c4-ba67-283bd6a2de14-kube-api-access-rd8hq\") pod \"service-ca-operator-544d749c8c-wcrtl\" (UID: \"cc33996a-0d4c-49c4-ba67-283bd6a2de14\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wcrtl" Nov 02 06:16:12.478035 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.477996 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dv9xj\" (UniqueName: \"kubernetes.io/projected/4ba9b2b6-4be9-4303-8eab-cd464c5a32c7-kube-api-access-dv9xj\") pod \"dns-default-gf726\" (UID: \"4ba9b2b6-4be9-4303-8eab-cd464c5a32c7\") " pod="openshift-dns/dns-default-gf726" Nov 02 06:16:12.478035 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.478018 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/cf8e8e65-8525-484b-adfe-f56517ccdc1b-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-d6bc4\" (UID: \"cf8e8e65-8525-484b-adfe-f56517ccdc1b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-d6bc4" Nov 02 06:16:12.478537 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.478056 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/17dee72b-2f47-4016-8ffe-3a89934e66ee-serving-cert\") pod \"console-operator-7cd56b8dbc-csfhg\" (UID: \"17dee72b-2f47-4016-8ffe-3a89934e66ee\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-csfhg" Nov 02 06:16:12.478537 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.478111 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/37982aa4-bdbd-40e8-a2dd-5094209f3c96-service-ca-bundle\") pod \"insights-operator-7c4986d449-c8dcq\" (UID: \"37982aa4-bdbd-40e8-a2dd-5094209f3c96\") " pod="openshift-insights/insights-operator-7c4986d449-c8dcq" Nov 02 06:16:12.478537 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.478117 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5989k\" (UniqueName: \"kubernetes.io/projected/20cb7690-c976-471c-b459-a9ab15cc51bf-kube-api-access-5989k\") pod \"network-check-source-6f78d59f99-k9hcs\" (UID: \"20cb7690-c976-471c-b459-a9ab15cc51bf\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-k9hcs" Nov 02 06:16:12.478537 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.478111 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/12b6cc1c-537d-4125-92ff-fe5f6927c53e-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-sktt2\" (UID: \"12b6cc1c-537d-4125-92ff-fe5f6927c53e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-sktt2" Nov 02 06:16:12.478537 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.478317 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/53aa1e5a-c35f-4aa6-9f23-cc61ccd331fc-nginx-conf\") pod \"networking-console-plugin-86c478bb88-fh4sz\" (UID: \"53aa1e5a-c35f-4aa6-9f23-cc61ccd331fc\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-fh4sz" Nov 02 06:16:12.478537 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.478460 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ba9b2b6-4be9-4303-8eab-cd464c5a32c7-tmp-dir\") pod \"dns-default-gf726\" (UID: \"4ba9b2b6-4be9-4303-8eab-cd464c5a32c7\") " pod="openshift-dns/dns-default-gf726" Nov 02 06:16:12.478537 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:16:12.478491 2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 02 06:16:12.478537 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:16:12.478538 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53aa1e5a-c35f-4aa6-9f23-cc61ccd331fc-networking-console-plugin-cert podName:53aa1e5a-c35f-4aa6-9f23-cc61ccd331fc nodeName:}" failed. No retries permitted until 2026-11-02 06:16:12.978522699 +0000 UTC m=+34.870340956 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/53aa1e5a-c35f-4aa6-9f23-cc61ccd331fc-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-fh4sz" (UID: "53aa1e5a-c35f-4aa6-9f23-cc61ccd331fc") : secret "networking-console-plugin-cert" not found Nov 02 06:16:12.478923 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.478807 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/37982aa4-bdbd-40e8-a2dd-5094209f3c96-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-c8dcq\" (UID: \"37982aa4-bdbd-40e8-a2dd-5094209f3c96\") " pod="openshift-insights/insights-operator-7c4986d449-c8dcq" Nov 02 06:16:12.478923 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.478886 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/cc33996a-0d4c-49c4-ba67-283bd6a2de14-config\") pod \"service-ca-operator-544d749c8c-wcrtl\" (UID: \"cc33996a-0d4c-49c4-ba67-283bd6a2de14\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wcrtl" Nov 02 06:16:12.479022 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.478942 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/cf8e8e65-8525-484b-adfe-f56517ccdc1b-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-d6bc4\" (UID: \"cf8e8e65-8525-484b-adfe-f56517ccdc1b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-d6bc4" Nov 02 06:16:12.479211 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:16:12.479190 2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 02 06:16:12.479277 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:16:12.479249 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf8e8e65-8525-484b-adfe-f56517ccdc1b-samples-operator-tls podName:cf8e8e65-8525-484b-adfe-f56517ccdc1b nodeName:}" failed. No retries permitted until 2026-11-02 06:16:12.979231483 +0000 UTC m=+34.871049730 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/cf8e8e65-8525-484b-adfe-f56517ccdc1b-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-d6bc4" (UID: "cf8e8e65-8525-484b-adfe-f56517ccdc1b") : secret "samples-operator-tls" not found Nov 02 06:16:12.479477 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.479458 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/37982aa4-bdbd-40e8-a2dd-5094209f3c96-snapshots\") pod \"insights-operator-7c4986d449-c8dcq\" (UID: \"37982aa4-bdbd-40e8-a2dd-5094209f3c96\") " pod="openshift-insights/insights-operator-7c4986d449-c8dcq" Nov 02 06:16:12.479835 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.479817 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/cc33996a-0d4c-49c4-ba67-283bd6a2de14-config\") pod \"service-ca-operator-544d749c8c-wcrtl\" (UID: \"cc33996a-0d4c-49c4-ba67-283bd6a2de14\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wcrtl" Nov 02 06:16:12.479929 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.479872 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/cf8e8e65-8525-484b-adfe-f56517ccdc1b-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-d6bc4\" (UID: \"cf8e8e65-8525-484b-adfe-f56517ccdc1b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-d6bc4" Nov 02 06:16:12.480273 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.480250 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mplc\" (UniqueName: \"kubernetes.io/projected/0d2b53c4-9d39-47fc-9367-c9a8e7f5da81-kube-api-access-8mplc\") pod \"ingress-canary-6tt27\" (UID: \"0d2b53c4-9d39-47fc-9367-c9a8e7f5da81\") " pod="openshift-ingress-canary/ingress-canary-6tt27" Nov 02 06:16:12.480411 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.480385 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4ba9b2b6-4be9-4303-8eab-cd464c5a32c7-config-volume\") pod \"dns-default-gf726\" (UID: \"4ba9b2b6-4be9-4303-8eab-cd464c5a32c7\") " pod="openshift-dns/dns-default-gf726" Nov 02 06:16:12.481294 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.480833 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/17dee72b-2f47-4016-8ffe-3a89934e66ee-config\") pod \"console-operator-7cd56b8dbc-csfhg\" (UID: \"17dee72b-2f47-4016-8ffe-3a89934e66ee\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-csfhg" Nov 02 06:16:12.481294 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.480995 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/37982aa4-bdbd-40e8-a2dd-5094209f3c96-serving-cert\") pod \"insights-operator-7c4986d449-c8dcq\" (UID: \"37982aa4-bdbd-40e8-a2dd-5094209f3c96\") " pod="openshift-insights/insights-operator-7c4986d449-c8dcq" Nov 02 06:16:12.481294 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.481181 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/17dee72b-2f47-4016-8ffe-3a89934e66ee-trusted-ca\") pod \"console-operator-7cd56b8dbc-csfhg\" (UID: \"17dee72b-2f47-4016-8ffe-3a89934e66ee\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-csfhg" Nov 02 06:16:12.481493 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.481466 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/12b6cc1c-537d-4125-92ff-fe5f6927c53e-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-sktt2\" (UID: \"12b6cc1c-537d-4125-92ff-fe5f6927c53e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-sktt2" Nov 02 06:16:12.482280 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.482261 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/17dee72b-2f47-4016-8ffe-3a89934e66ee-serving-cert\") pod \"console-operator-7cd56b8dbc-csfhg\" (UID: \"17dee72b-2f47-4016-8ffe-3a89934e66ee\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-csfhg" Nov 02 06:16:12.482362 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.482343 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cc33996a-0d4c-49c4-ba67-283bd6a2de14-serving-cert\") pod \"service-ca-operator-544d749c8c-wcrtl\" (UID: \"cc33996a-0d4c-49c4-ba67-283bd6a2de14\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wcrtl" Nov 02 06:16:12.482775 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.482757 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0d2b53c4-9d39-47fc-9367-c9a8e7f5da81-cert\") pod \"ingress-canary-6tt27\" (UID: \"0d2b53c4-9d39-47fc-9367-c9a8e7f5da81\") " pod="openshift-ingress-canary/ingress-canary-6tt27" Nov 02 06:16:12.534689 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.534641 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5989k\" (UniqueName: \"kubernetes.io/projected/20cb7690-c976-471c-b459-a9ab15cc51bf-kube-api-access-5989k\") pod \"network-check-source-6f78d59f99-k9hcs\" (UID: \"20cb7690-c976-471c-b459-a9ab15cc51bf\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-k9hcs" Nov 02 06:16:12.534982 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.534957 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnw94\" (UniqueName: \"kubernetes.io/projected/121e38c0-6fa3-46aa-9191-5c6f91c8764c-kube-api-access-fnw94\") pod \"volume-data-source-validator-58c9f5f46d-8fbcn\" (UID: \"121e38c0-6fa3-46aa-9191-5c6f91c8764c\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-8fbcn" Nov 02 06:16:12.535535 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.535510 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rd8hq\" (UniqueName: \"kubernetes.io/projected/cc33996a-0d4c-49c4-ba67-283bd6a2de14-kube-api-access-rd8hq\") pod \"service-ca-operator-544d749c8c-wcrtl\" (UID: \"cc33996a-0d4c-49c4-ba67-283bd6a2de14\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wcrtl" Nov 02 06:16:12.536599 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.536576 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tz6t\" (UniqueName: \"kubernetes.io/projected/17dee72b-2f47-4016-8ffe-3a89934e66ee-kube-api-access-9tz6t\") pod \"console-operator-7cd56b8dbc-csfhg\" (UID: \"17dee72b-2f47-4016-8ffe-3a89934e66ee\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-csfhg" Nov 02 06:16:12.537161 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.537112 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dv9xj\" (UniqueName: \"kubernetes.io/projected/4ba9b2b6-4be9-4303-8eab-cd464c5a32c7-kube-api-access-dv9xj\") pod \"dns-default-gf726\" (UID: \"4ba9b2b6-4be9-4303-8eab-cd464c5a32c7\") " pod="openshift-dns/dns-default-gf726" Nov 02 06:16:12.537339 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.537314 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mplc\" (UniqueName: \"kubernetes.io/projected/0d2b53c4-9d39-47fc-9367-c9a8e7f5da81-kube-api-access-8mplc\") pod \"ingress-canary-6tt27\" (UID: \"0d2b53c4-9d39-47fc-9367-c9a8e7f5da81\") " pod="openshift-ingress-canary/ingress-canary-6tt27" Nov 02 06:16:12.537482 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.537464 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7l5t\" (UniqueName: \"kubernetes.io/projected/37982aa4-bdbd-40e8-a2dd-5094209f3c96-kube-api-access-h7l5t\") pod \"insights-operator-7c4986d449-c8dcq\" (UID: \"37982aa4-bdbd-40e8-a2dd-5094209f3c96\") " pod="openshift-insights/insights-operator-7c4986d449-c8dcq" Nov 02 06:16:12.538055 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.538022 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwqbt\" (UniqueName: \"kubernetes.io/projected/cf8e8e65-8525-484b-adfe-f56517ccdc1b-kube-api-access-vwqbt\") pod \"cluster-samples-operator-6ddb6c8b94-d6bc4\" (UID: \"cf8e8e65-8525-484b-adfe-f56517ccdc1b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-d6bc4" Nov 02 06:16:12.538649 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.538610 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmsjq\" (UniqueName: \"kubernetes.io/projected/12b6cc1c-537d-4125-92ff-fe5f6927c53e-kube-api-access-bmsjq\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-sktt2\" (UID: \"12b6cc1c-537d-4125-92ff-fe5f6927c53e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-sktt2" Nov 02 06:16:12.547475 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.546945 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-csfhg" Nov 02 06:16:12.569329 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.569080 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-8fbcn" Nov 02 06:16:12.577953 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.577929 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wcrtl" Nov 02 06:16:12.590258 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.590238 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-c8dcq" Nov 02 06:16:12.617437 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.617331 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-k9hcs" Nov 02 06:16:12.634078 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.631670 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-6tt27" Nov 02 06:16:12.836137 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.836109 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-sktt2" Nov 02 06:16:12.861797 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.861733 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-kb9wk"] Nov 02 06:16:12.864980 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.864959 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-kd68m"] Nov 02 06:16:12.868719 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:16:12.868687 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7434256f_45c3_47ff_9111_352114b0f6d8.slice/crio-755856f4f662e5878f1383579803c0114371ba096156e09fb4df933877698348 WatchSource:0}: Error finding container 755856f4f662e5878f1383579803c0114371ba096156e09fb4df933877698348: Status 404 returned error can't find the container with id 755856f4f662e5878f1383579803c0114371ba096156e09fb4df933877698348 Nov 02 06:16:12.870453 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:16:12.870429 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e8f2e60_de1b_4c8b_9285_56568c150d92.slice/crio-e0370d945d4e54d879de7ca1d43138dcadfb5673f42e318252a19874123f5c6c WatchSource:0}: Error finding container e0370d945d4e54d879de7ca1d43138dcadfb5673f42e318252a19874123f5c6c: Status 404 returned error can't find the container with id e0370d945d4e54d879de7ca1d43138dcadfb5673f42e318252a19874123f5c6c Nov 02 06:16:12.885646 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.884176 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45e5ce56-5cab-4546-8b31-02fc06cfa0cc-metrics-certs\") pod \"router-default-5f5c58667c-29wnm\" (UID: \"45e5ce56-5cab-4546-8b31-02fc06cfa0cc\") " pod="openshift-ingress/router-default-5f5c58667c-29wnm" Nov 02 06:16:12.885646 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.884234 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f0531b4d-c954-4830-85f6-fe876c76437a-registry-tls\") pod \"image-registry-6689c6d46b-vw558\" (UID: \"f0531b4d-c954-4830-85f6-fe876c76437a\") " pod="openshift-image-registry/image-registry-6689c6d46b-vw558" Nov 02 06:16:12.885646 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.884285 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fa9c5a91-d480-4d7b-9044-be6437a76a02-registry-tls\") pod \"image-registry-746899d75-dtjhs\" (UID: \"fa9c5a91-d480-4d7b-9044-be6437a76a02\") " pod="openshift-image-registry/image-registry-746899d75-dtjhs" Nov 02 06:16:12.885646 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:16:12.884322 2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 02 06:16:12.885646 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.884351 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45e5ce56-5cab-4546-8b31-02fc06cfa0cc-service-ca-bundle\") pod \"router-default-5f5c58667c-29wnm\" (UID: \"45e5ce56-5cab-4546-8b31-02fc06cfa0cc\") " pod="openshift-ingress/router-default-5f5c58667c-29wnm" Nov 02 06:16:12.885646 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:16:12.884376 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45e5ce56-5cab-4546-8b31-02fc06cfa0cc-metrics-certs podName:45e5ce56-5cab-4546-8b31-02fc06cfa0cc nodeName:}" failed. No retries permitted until 2026-11-02 06:16:13.884357388 +0000 UTC m=+35.776175642 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/45e5ce56-5cab-4546-8b31-02fc06cfa0cc-metrics-certs") pod "router-default-5f5c58667c-29wnm" (UID: "45e5ce56-5cab-4546-8b31-02fc06cfa0cc") : secret "router-metrics-certs-default" not found Nov 02 06:16:12.885646 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.884391 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-jrtnt"] Nov 02 06:16:12.885646 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.884405 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3fcb7621-86c8-448f-b4fa-b5bd76be7356-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-2rs6c\" (UID: \"3fcb7621-86c8-448f-b4fa-b5bd76be7356\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2rs6c" Nov 02 06:16:12.885646 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:16:12.884419 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/45e5ce56-5cab-4546-8b31-02fc06cfa0cc-service-ca-bundle podName:45e5ce56-5cab-4546-8b31-02fc06cfa0cc nodeName:}" failed. No retries permitted until 2026-11-02 06:16:13.884408097 +0000 UTC m=+35.776226341 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/45e5ce56-5cab-4546-8b31-02fc06cfa0cc-service-ca-bundle") pod "router-default-5f5c58667c-29wnm" (UID: "45e5ce56-5cab-4546-8b31-02fc06cfa0cc") : configmap references non-existent config key: service-ca.crt Nov 02 06:16:12.885646 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:16:12.884480 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 06:16:12.885646 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:16:12.884492 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6689c6d46b-vw558: secret "image-registry-tls" not found Nov 02 06:16:12.885646 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:16:12.884523 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f0531b4d-c954-4830-85f6-fe876c76437a-registry-tls podName:f0531b4d-c954-4830-85f6-fe876c76437a nodeName:}" failed. No retries permitted until 2026-11-02 06:16:13.884512372 +0000 UTC m=+35.776330629 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f0531b4d-c954-4830-85f6-fe876c76437a-registry-tls") pod "image-registry-6689c6d46b-vw558" (UID: "f0531b4d-c954-4830-85f6-fe876c76437a") : secret "image-registry-tls" not found Nov 02 06:16:12.885646 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:16:12.884529 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 06:16:12.885646 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:16:12.884539 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-746899d75-dtjhs: secret "image-registry-tls" not found Nov 02 06:16:12.885646 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:16:12.884539 2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 02 06:16:12.885646 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:16:12.884571 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fa9c5a91-d480-4d7b-9044-be6437a76a02-registry-tls podName:fa9c5a91-d480-4d7b-9044-be6437a76a02 nodeName:}" failed. No retries permitted until 2026-11-02 06:16:13.884562015 +0000 UTC m=+35.776380261 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fa9c5a91-d480-4d7b-9044-be6437a76a02-registry-tls") pod "image-registry-746899d75-dtjhs" (UID: "fa9c5a91-d480-4d7b-9044-be6437a76a02") : secret "image-registry-tls" not found Nov 02 06:16:12.886553 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:16:12.884592 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3fcb7621-86c8-448f-b4fa-b5bd76be7356-cluster-monitoring-operator-tls podName:3fcb7621-86c8-448f-b4fa-b5bd76be7356 nodeName:}" failed. No retries permitted until 2026-11-02 06:16:13.884577026 +0000 UTC m=+35.776395285 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/3fcb7621-86c8-448f-b4fa-b5bd76be7356-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-2rs6c" (UID: "3fcb7621-86c8-448f-b4fa-b5bd76be7356") : secret "cluster-monitoring-operator-tls" not found Nov 02 06:16:12.890318 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:16:12.890288 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddee3b502_2562_4b21_bc46_2031580c4b0a.slice/crio-860a45d95193ea55c54e30a2f0d13b46a725340587fe8a32c4c6a6be79374f1d WatchSource:0}: Error finding container 860a45d95193ea55c54e30a2f0d13b46a725340587fe8a32c4c6a6be79374f1d: Status 404 returned error can't find the container with id 860a45d95193ea55c54e30a2f0d13b46a725340587fe8a32c4c6a6be79374f1d Nov 02 06:16:12.934147 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.934119 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-csfhg"] Nov 02 06:16:12.968780 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.968716 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-wcrtl"] Nov 02 06:16:12.978846 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:16:12.978793 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc33996a_0d4c_49c4_ba67_283bd6a2de14.slice/crio-c53b73150c37aad5859309630eab56b7a553276fb27542d80ef212490baaeb30 WatchSource:0}: Error finding container c53b73150c37aad5859309630eab56b7a553276fb27542d80ef212490baaeb30: Status 404 returned error can't find the container with id c53b73150c37aad5859309630eab56b7a553276fb27542d80ef212490baaeb30 Nov 02 06:16:12.985055 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.985028 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/cf8e8e65-8525-484b-adfe-f56517ccdc1b-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-d6bc4\" (UID: \"cf8e8e65-8525-484b-adfe-f56517ccdc1b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-d6bc4" Nov 02 06:16:12.985162 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.985106 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4ba9b2b6-4be9-4303-8eab-cd464c5a32c7-metrics-tls\") pod \"dns-default-gf726\" (UID: \"4ba9b2b6-4be9-4303-8eab-cd464c5a32c7\") " pod="openshift-dns/dns-default-gf726" Nov 02 06:16:12.985227 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:12.985176 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/53aa1e5a-c35f-4aa6-9f23-cc61ccd331fc-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-fh4sz\" (UID: \"53aa1e5a-c35f-4aa6-9f23-cc61ccd331fc\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-fh4sz" Nov 02 06:16:12.985485 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:16:12.985327 2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 02 06:16:12.985485 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:16:12.985366 2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 02 06:16:12.985485 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:16:12.985366 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 06:16:12.985485 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:16:12.985376 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53aa1e5a-c35f-4aa6-9f23-cc61ccd331fc-networking-console-plugin-cert podName:53aa1e5a-c35f-4aa6-9f23-cc61ccd331fc nodeName:}" failed. No retries permitted until 2026-11-02 06:16:13.985358882 +0000 UTC m=+35.877177138 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/53aa1e5a-c35f-4aa6-9f23-cc61ccd331fc-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-fh4sz" (UID: "53aa1e5a-c35f-4aa6-9f23-cc61ccd331fc") : secret "networking-console-plugin-cert" not found Nov 02 06:16:12.985485 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:16:12.985457 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf8e8e65-8525-484b-adfe-f56517ccdc1b-samples-operator-tls podName:cf8e8e65-8525-484b-adfe-f56517ccdc1b nodeName:}" failed. No retries permitted until 2026-11-02 06:16:13.985439894 +0000 UTC m=+35.877258138 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/cf8e8e65-8525-484b-adfe-f56517ccdc1b-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-d6bc4" (UID: "cf8e8e65-8525-484b-adfe-f56517ccdc1b") : secret "samples-operator-tls" not found Nov 02 06:16:12.985485 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:16:12.985474 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ba9b2b6-4be9-4303-8eab-cd464c5a32c7-metrics-tls podName:4ba9b2b6-4be9-4303-8eab-cd464c5a32c7 nodeName:}" failed. No retries permitted until 2026-11-02 06:16:13.985463883 +0000 UTC m=+35.877282130 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4ba9b2b6-4be9-4303-8eab-cd464c5a32c7-metrics-tls") pod "dns-default-gf726" (UID: "4ba9b2b6-4be9-4303-8eab-cd464c5a32c7") : secret "dns-default-metrics-tls" not found Nov 02 06:16:13.005166 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:13.005125 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-c8dcq"] Nov 02 06:16:13.011236 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:16:13.011205 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37982aa4_bdbd_40e8_a2dd_5094209f3c96.slice/crio-91abcd1f2335c5276a0294d88a772c710c698e73c3a2b82f1a7042821d779a32 WatchSource:0}: Error finding container 91abcd1f2335c5276a0294d88a772c710c698e73c3a2b82f1a7042821d779a32: Status 404 returned error can't find the container with id 91abcd1f2335c5276a0294d88a772c710c698e73c3a2b82f1a7042821d779a32 Nov 02 06:16:13.031254 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:13.030712 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-k9hcs"] Nov 02 06:16:13.035091 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:13.035072 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-6tt27"] Nov 02 06:16:13.037370 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:16:13.037344 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20cb7690_c976_471c_b459_a9ab15cc51bf.slice/crio-4963902e93a3be86d04c977ac33a031fb4a55dbdfd7d021a6e3debdefa6f663a WatchSource:0}: Error finding container 4963902e93a3be86d04c977ac33a031fb4a55dbdfd7d021a6e3debdefa6f663a: Status 404 returned error can't find the container with id 4963902e93a3be86d04c977ac33a031fb4a55dbdfd7d021a6e3debdefa6f663a Nov 02 06:16:13.039892 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:16:13.039869 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d2b53c4_9d39_47fc_9367_c9a8e7f5da81.slice/crio-216fc5ad7a00312abf4a46ba90063240b9601015205ec36f60ddec22af761f9c WatchSource:0}: Error finding container 216fc5ad7a00312abf4a46ba90063240b9601015205ec36f60ddec22af761f9c: Status 404 returned error can't find the container with id 216fc5ad7a00312abf4a46ba90063240b9601015205ec36f60ddec22af761f9c Nov 02 06:16:13.084305 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:13.084278 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-8fbcn"] Nov 02 06:16:13.089958 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:16:13.089928 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod121e38c0_6fa3_46aa_9191_5c6f91c8764c.slice/crio-ab0ebd2db011ef10e8f93786016b5cc45593e7dc0f17ead3905307dd1d9fcd2d WatchSource:0}: Error finding container ab0ebd2db011ef10e8f93786016b5cc45593e7dc0f17ead3905307dd1d9fcd2d: Status 404 returned error can't find the container with id ab0ebd2db011ef10e8f93786016b5cc45593e7dc0f17ead3905307dd1d9fcd2d Nov 02 06:16:13.230409 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:13.230310 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-sktt2"] Nov 02 06:16:13.235661 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:16:13.235636 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12b6cc1c_537d_4125_92ff_fe5f6927c53e.slice/crio-62d6ac9953134850fb004827c8a962446a4b2f6330e743b941096a9ab30409f5 WatchSource:0}: Error finding container 62d6ac9953134850fb004827c8a962446a4b2f6330e743b941096a9ab30409f5: Status 404 returned error can't find the container with id 62d6ac9953134850fb004827c8a962446a4b2f6330e743b941096a9ab30409f5 Nov 02 06:16:13.648228 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:13.648206 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ggpsr" Nov 02 06:16:13.648828 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:13.648288 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sb4nt" Nov 02 06:16:13.648828 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:13.648371 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ft7gs" Nov 02 06:16:13.700712 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:13.700682 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-2646z\"" Nov 02 06:16:13.700712 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:13.700697 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 02 06:16:13.700712 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:13.700682 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-cdk7k\"" Nov 02 06:16:13.700965 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:13.700701 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 02 06:16:13.700965 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:13.700766 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-9bfjz\"" Nov 02 06:16:13.820037 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:13.820008 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-6tt27" event={"ID":"0d2b53c4-9d39-47fc-9367-c9a8e7f5da81","Type":"ContainerStarted","Data":"216fc5ad7a00312abf4a46ba90063240b9601015205ec36f60ddec22af761f9c"} Nov 02 06:16:13.821023 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:13.820999 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-sktt2" event={"ID":"12b6cc1c-537d-4125-92ff-fe5f6927c53e","Type":"ContainerStarted","Data":"62d6ac9953134850fb004827c8a962446a4b2f6330e743b941096a9ab30409f5"} Nov 02 06:16:13.822060 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:13.822035 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-c8dcq" event={"ID":"37982aa4-bdbd-40e8-a2dd-5094209f3c96","Type":"ContainerStarted","Data":"91abcd1f2335c5276a0294d88a772c710c698e73c3a2b82f1a7042821d779a32"} Nov 02 06:16:13.823322 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:13.823298 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-kd68m" event={"ID":"8e8f2e60-de1b-4c8b-9285-56568c150d92","Type":"ContainerStarted","Data":"d6ead965aaa9242bbf0054feb8ab4e8b9dedca8df1b0e3752924c978d9d80e45"} Nov 02 06:16:13.823413 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:13.823343 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-kd68m" event={"ID":"8e8f2e60-de1b-4c8b-9285-56568c150d92","Type":"ContainerStarted","Data":"e0370d945d4e54d879de7ca1d43138dcadfb5673f42e318252a19874123f5c6c"} Nov 02 06:16:13.824310 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:13.824292 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wcrtl" event={"ID":"cc33996a-0d4c-49c4-ba67-283bd6a2de14","Type":"ContainerStarted","Data":"c53b73150c37aad5859309630eab56b7a553276fb27542d80ef212490baaeb30"} Nov 02 06:16:13.825382 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:13.825361 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-k9hcs" event={"ID":"20cb7690-c976-471c-b459-a9ab15cc51bf","Type":"ContainerStarted","Data":"4963902e93a3be86d04c977ac33a031fb4a55dbdfd7d021a6e3debdefa6f663a"} Nov 02 06:16:13.828600 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:13.827312 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-jrtnt" event={"ID":"dee3b502-2562-4b21-bc46-2031580c4b0a","Type":"ContainerStarted","Data":"f71998697909d51a5c60caa58d65a7996dc634f72c05013fd1f0468cc7c3ac00"} Nov 02 06:16:13.828600 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:13.827353 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-jrtnt" event={"ID":"dee3b502-2562-4b21-bc46-2031580c4b0a","Type":"ContainerStarted","Data":"860a45d95193ea55c54e30a2f0d13b46a725340587fe8a32c4c6a6be79374f1d"} Nov 02 06:16:13.829614 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:13.829591 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-8fbcn" event={"ID":"121e38c0-6fa3-46aa-9191-5c6f91c8764c","Type":"ContainerStarted","Data":"ab0ebd2db011ef10e8f93786016b5cc45593e7dc0f17ead3905307dd1d9fcd2d"} Nov 02 06:16:13.830963 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:13.830937 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-csfhg" event={"ID":"17dee72b-2f47-4016-8ffe-3a89934e66ee","Type":"ContainerStarted","Data":"d1b287d3147a3d6fbc93b84f0c5883cfd49ac31f846e5d01f59df87df61be411"} Nov 02 06:16:13.832645 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:13.832626 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-kb9wk" event={"ID":"7434256f-45c3-47ff-9111-352114b0f6d8","Type":"ContainerStarted","Data":"e9e46cf7dfd1998bebd33d78c1f6f644b7d831057b1e77609622af0c31b37a82"} Nov 02 06:16:13.832729 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:13.832660 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-kb9wk" event={"ID":"7434256f-45c3-47ff-9111-352114b0f6d8","Type":"ContainerStarted","Data":"755856f4f662e5878f1383579803c0114371ba096156e09fb4df933877698348"} Nov 02 06:16:13.891125 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:13.891095 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45e5ce56-5cab-4546-8b31-02fc06cfa0cc-service-ca-bundle\") pod \"router-default-5f5c58667c-29wnm\" (UID: \"45e5ce56-5cab-4546-8b31-02fc06cfa0cc\") " pod="openshift-ingress/router-default-5f5c58667c-29wnm" Nov 02 06:16:13.891125 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:13.891126 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3fcb7621-86c8-448f-b4fa-b5bd76be7356-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-2rs6c\" (UID: \"3fcb7621-86c8-448f-b4fa-b5bd76be7356\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2rs6c" Nov 02 06:16:13.891306 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:13.891187 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45e5ce56-5cab-4546-8b31-02fc06cfa0cc-metrics-certs\") pod \"router-default-5f5c58667c-29wnm\" (UID: \"45e5ce56-5cab-4546-8b31-02fc06cfa0cc\") " pod="openshift-ingress/router-default-5f5c58667c-29wnm" Nov 02 06:16:13.891306 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:16:13.891270 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/45e5ce56-5cab-4546-8b31-02fc06cfa0cc-service-ca-bundle podName:45e5ce56-5cab-4546-8b31-02fc06cfa0cc nodeName:}" failed. No retries permitted until 2026-11-02 06:16:15.891250209 +0000 UTC m=+37.783068470 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/45e5ce56-5cab-4546-8b31-02fc06cfa0cc-service-ca-bundle") pod "router-default-5f5c58667c-29wnm" (UID: "45e5ce56-5cab-4546-8b31-02fc06cfa0cc") : configmap references non-existent config key: service-ca.crt Nov 02 06:16:13.891421 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:16:13.891328 2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 02 06:16:13.891421 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:16:13.891389 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 06:16:13.891421 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:16:13.891393 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3fcb7621-86c8-448f-b4fa-b5bd76be7356-cluster-monitoring-operator-tls podName:3fcb7621-86c8-448f-b4fa-b5bd76be7356 nodeName:}" failed. No retries permitted until 2026-11-02 06:16:15.891376401 +0000 UTC m=+37.783194657 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/3fcb7621-86c8-448f-b4fa-b5bd76be7356-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-2rs6c" (UID: "3fcb7621-86c8-448f-b4fa-b5bd76be7356") : secret "cluster-monitoring-operator-tls" not found Nov 02 06:16:13.891421 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:16:13.891402 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6689c6d46b-vw558: secret "image-registry-tls" not found Nov 02 06:16:13.891421 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:16:13.891336 2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 02 06:16:13.891421 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:13.891297 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f0531b4d-c954-4830-85f6-fe876c76437a-registry-tls\") pod \"image-registry-6689c6d46b-vw558\" (UID: \"f0531b4d-c954-4830-85f6-fe876c76437a\") " pod="openshift-image-registry/image-registry-6689c6d46b-vw558" Nov 02 06:16:13.891684 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:16:13.891443 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f0531b4d-c954-4830-85f6-fe876c76437a-registry-tls podName:f0531b4d-c954-4830-85f6-fe876c76437a nodeName:}" failed. No retries permitted until 2026-11-02 06:16:15.891431533 +0000 UTC m=+37.783249773 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f0531b4d-c954-4830-85f6-fe876c76437a-registry-tls") pod "image-registry-6689c6d46b-vw558" (UID: "f0531b4d-c954-4830-85f6-fe876c76437a") : secret "image-registry-tls" not found Nov 02 06:16:13.891684 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:16:13.891459 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45e5ce56-5cab-4546-8b31-02fc06cfa0cc-metrics-certs podName:45e5ce56-5cab-4546-8b31-02fc06cfa0cc nodeName:}" failed. No retries permitted until 2026-11-02 06:16:15.891450532 +0000 UTC m=+37.783268775 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/45e5ce56-5cab-4546-8b31-02fc06cfa0cc-metrics-certs") pod "router-default-5f5c58667c-29wnm" (UID: "45e5ce56-5cab-4546-8b31-02fc06cfa0cc") : secret "router-metrics-certs-default" not found Nov 02 06:16:13.891684 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:13.891504 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fa9c5a91-d480-4d7b-9044-be6437a76a02-registry-tls\") pod \"image-registry-746899d75-dtjhs\" (UID: \"fa9c5a91-d480-4d7b-9044-be6437a76a02\") " pod="openshift-image-registry/image-registry-746899d75-dtjhs" Nov 02 06:16:13.891684 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:16:13.891578 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 06:16:13.891684 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:16:13.891586 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-746899d75-dtjhs: secret "image-registry-tls" not found Nov 02 06:16:13.891684 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:16:13.891614 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fa9c5a91-d480-4d7b-9044-be6437a76a02-registry-tls podName:fa9c5a91-d480-4d7b-9044-be6437a76a02 nodeName:}" failed. No retries permitted until 2026-11-02 06:16:15.891605033 +0000 UTC m=+37.783423274 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fa9c5a91-d480-4d7b-9044-be6437a76a02-registry-tls") pod "image-registry-746899d75-dtjhs" (UID: "fa9c5a91-d480-4d7b-9044-be6437a76a02") : secret "image-registry-tls" not found Nov 02 06:16:13.992593 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:13.992529 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/cf8e8e65-8525-484b-adfe-f56517ccdc1b-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-d6bc4\" (UID: \"cf8e8e65-8525-484b-adfe-f56517ccdc1b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-d6bc4" Nov 02 06:16:13.992593 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:13.992589 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4ba9b2b6-4be9-4303-8eab-cd464c5a32c7-metrics-tls\") pod \"dns-default-gf726\" (UID: \"4ba9b2b6-4be9-4303-8eab-cd464c5a32c7\") " pod="openshift-dns/dns-default-gf726" Nov 02 06:16:13.992784 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:13.992664 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/53aa1e5a-c35f-4aa6-9f23-cc61ccd331fc-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-fh4sz\" (UID: \"53aa1e5a-c35f-4aa6-9f23-cc61ccd331fc\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-fh4sz" Nov 02 06:16:13.992784 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:16:13.992684 2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 02 06:16:13.992784 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:16:13.992741 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf8e8e65-8525-484b-adfe-f56517ccdc1b-samples-operator-tls podName:cf8e8e65-8525-484b-adfe-f56517ccdc1b nodeName:}" failed. No retries permitted until 2026-11-02 06:16:15.992723843 +0000 UTC m=+37.884542097 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/cf8e8e65-8525-484b-adfe-f56517ccdc1b-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-d6bc4" (UID: "cf8e8e65-8525-484b-adfe-f56517ccdc1b") : secret "samples-operator-tls" not found Nov 02 06:16:13.992784 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:16:13.992758 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 06:16:13.992958 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:16:13.992759 2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 02 06:16:13.992958 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:16:13.992806 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ba9b2b6-4be9-4303-8eab-cd464c5a32c7-metrics-tls podName:4ba9b2b6-4be9-4303-8eab-cd464c5a32c7 nodeName:}" failed. No retries permitted until 2026-11-02 06:16:15.99279191 +0000 UTC m=+37.884610166 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4ba9b2b6-4be9-4303-8eab-cd464c5a32c7-metrics-tls") pod "dns-default-gf726" (UID: "4ba9b2b6-4be9-4303-8eab-cd464c5a32c7") : secret "dns-default-metrics-tls" not found Nov 02 06:16:13.992958 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:16:13.992830 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53aa1e5a-c35f-4aa6-9f23-cc61ccd331fc-networking-console-plugin-cert podName:53aa1e5a-c35f-4aa6-9f23-cc61ccd331fc nodeName:}" failed. No retries permitted until 2026-11-02 06:16:15.992819671 +0000 UTC m=+37.884637912 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/53aa1e5a-c35f-4aa6-9f23-cc61ccd331fc-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-fh4sz" (UID: "53aa1e5a-c35f-4aa6-9f23-cc61ccd331fc") : secret "networking-console-plugin-cert" not found Nov 02 06:16:14.152489 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:14.152441 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-kb9wk" podStartSLOduration=269.152427563 podStartE2EDuration="4m29.152427563s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:11:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:16:14.152084041 +0000 UTC m=+36.043902305" watchObservedRunningTime="2026-11-02 06:16:14.152427563 +0000 UTC m=+36.044245824" Nov 02 06:16:14.259422 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:14.259321 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-jrtnt" podStartSLOduration=269.25930447 podStartE2EDuration="4m29.25930447s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:11:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:16:14.258823061 +0000 UTC m=+36.150641325" watchObservedRunningTime="2026-11-02 06:16:14.25930447 +0000 UTC m=+36.151122735" Nov 02 06:16:15.912577 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:15.912534 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f0531b4d-c954-4830-85f6-fe876c76437a-registry-tls\") pod \"image-registry-6689c6d46b-vw558\" (UID: \"f0531b4d-c954-4830-85f6-fe876c76437a\") " pod="openshift-image-registry/image-registry-6689c6d46b-vw558" Nov 02 06:16:15.913213 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:15.912620 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fa9c5a91-d480-4d7b-9044-be6437a76a02-registry-tls\") pod \"image-registry-746899d75-dtjhs\" (UID: \"fa9c5a91-d480-4d7b-9044-be6437a76a02\") " pod="openshift-image-registry/image-registry-746899d75-dtjhs" Nov 02 06:16:15.913213 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:15.912698 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45e5ce56-5cab-4546-8b31-02fc06cfa0cc-service-ca-bundle\") pod \"router-default-5f5c58667c-29wnm\" (UID: \"45e5ce56-5cab-4546-8b31-02fc06cfa0cc\") " pod="openshift-ingress/router-default-5f5c58667c-29wnm" Nov 02 06:16:15.913213 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:15.912724 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3fcb7621-86c8-448f-b4fa-b5bd76be7356-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-2rs6c\" (UID: \"3fcb7621-86c8-448f-b4fa-b5bd76be7356\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2rs6c" Nov 02 06:16:15.913213 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:15.912782 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45e5ce56-5cab-4546-8b31-02fc06cfa0cc-metrics-certs\") pod \"router-default-5f5c58667c-29wnm\" (UID: \"45e5ce56-5cab-4546-8b31-02fc06cfa0cc\") " pod="openshift-ingress/router-default-5f5c58667c-29wnm" Nov 02 06:16:15.913213 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:16:15.912897 2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 02 06:16:15.913213 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:16:15.912963 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45e5ce56-5cab-4546-8b31-02fc06cfa0cc-metrics-certs podName:45e5ce56-5cab-4546-8b31-02fc06cfa0cc nodeName:}" failed. No retries permitted until 2026-11-02 06:16:19.912949234 +0000 UTC m=+41.804767478 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/45e5ce56-5cab-4546-8b31-02fc06cfa0cc-metrics-certs") pod "router-default-5f5c58667c-29wnm" (UID: "45e5ce56-5cab-4546-8b31-02fc06cfa0cc") : secret "router-metrics-certs-default" not found Nov 02 06:16:15.913478 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:16:15.913367 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 06:16:15.913478 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:16:15.913383 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6689c6d46b-vw558: secret "image-registry-tls" not found Nov 02 06:16:15.913478 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:16:15.913427 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f0531b4d-c954-4830-85f6-fe876c76437a-registry-tls podName:f0531b4d-c954-4830-85f6-fe876c76437a nodeName:}" failed. No retries permitted until 2026-11-02 06:16:19.913412689 +0000 UTC m=+41.805230936 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f0531b4d-c954-4830-85f6-fe876c76437a-registry-tls") pod "image-registry-6689c6d46b-vw558" (UID: "f0531b4d-c954-4830-85f6-fe876c76437a") : secret "image-registry-tls" not found Nov 02 06:16:15.913478 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:16:15.913496 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 06:16:15.913478 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:16:15.913506 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-746899d75-dtjhs: secret "image-registry-tls" not found Nov 02 06:16:15.913789 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:16:15.913601 2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 02 06:16:15.913789 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:16:15.913655 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3fcb7621-86c8-448f-b4fa-b5bd76be7356-cluster-monitoring-operator-tls podName:3fcb7621-86c8-448f-b4fa-b5bd76be7356 nodeName:}" failed. No retries permitted until 2026-11-02 06:16:19.913641942 +0000 UTC m=+41.805460186 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/3fcb7621-86c8-448f-b4fa-b5bd76be7356-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-2rs6c" (UID: "3fcb7621-86c8-448f-b4fa-b5bd76be7356") : secret "cluster-monitoring-operator-tls" not found Nov 02 06:16:15.913789 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:16:15.913694 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/45e5ce56-5cab-4546-8b31-02fc06cfa0cc-service-ca-bundle podName:45e5ce56-5cab-4546-8b31-02fc06cfa0cc nodeName:}" failed. No retries permitted until 2026-11-02 06:16:19.913670299 +0000 UTC m=+41.805488542 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/45e5ce56-5cab-4546-8b31-02fc06cfa0cc-service-ca-bundle") pod "router-default-5f5c58667c-29wnm" (UID: "45e5ce56-5cab-4546-8b31-02fc06cfa0cc") : configmap references non-existent config key: service-ca.crt Nov 02 06:16:15.913789 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:16:15.913727 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fa9c5a91-d480-4d7b-9044-be6437a76a02-registry-tls podName:fa9c5a91-d480-4d7b-9044-be6437a76a02 nodeName:}" failed. No retries permitted until 2026-11-02 06:16:19.913715056 +0000 UTC m=+41.805533297 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fa9c5a91-d480-4d7b-9044-be6437a76a02-registry-tls") pod "image-registry-746899d75-dtjhs" (UID: "fa9c5a91-d480-4d7b-9044-be6437a76a02") : secret "image-registry-tls" not found Nov 02 06:16:16.014381 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:16.013680 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/53aa1e5a-c35f-4aa6-9f23-cc61ccd331fc-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-fh4sz\" (UID: \"53aa1e5a-c35f-4aa6-9f23-cc61ccd331fc\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-fh4sz" Nov 02 06:16:16.014381 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:16.013785 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/cf8e8e65-8525-484b-adfe-f56517ccdc1b-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-d6bc4\" (UID: \"cf8e8e65-8525-484b-adfe-f56517ccdc1b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-d6bc4" Nov 02 06:16:16.014381 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:16.013986 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4ba9b2b6-4be9-4303-8eab-cd464c5a32c7-metrics-tls\") pod \"dns-default-gf726\" (UID: \"4ba9b2b6-4be9-4303-8eab-cd464c5a32c7\") " pod="openshift-dns/dns-default-gf726" Nov 02 06:16:16.014381 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:16:16.014027 2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 02 06:16:16.014381 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:16:16.014090 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53aa1e5a-c35f-4aa6-9f23-cc61ccd331fc-networking-console-plugin-cert podName:53aa1e5a-c35f-4aa6-9f23-cc61ccd331fc nodeName:}" failed. No retries permitted until 2026-11-02 06:16:20.014073133 +0000 UTC m=+41.905891392 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/53aa1e5a-c35f-4aa6-9f23-cc61ccd331fc-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-fh4sz" (UID: "53aa1e5a-c35f-4aa6-9f23-cc61ccd331fc") : secret "networking-console-plugin-cert" not found Nov 02 06:16:16.014381 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:16:16.014099 2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 02 06:16:16.014381 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:16:16.014157 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 06:16:16.014381 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:16:16.014189 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf8e8e65-8525-484b-adfe-f56517ccdc1b-samples-operator-tls podName:cf8e8e65-8525-484b-adfe-f56517ccdc1b nodeName:}" failed. No retries permitted until 2026-11-02 06:16:20.014170987 +0000 UTC m=+41.905989249 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/cf8e8e65-8525-484b-adfe-f56517ccdc1b-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-d6bc4" (UID: "cf8e8e65-8525-484b-adfe-f56517ccdc1b") : secret "samples-operator-tls" not found Nov 02 06:16:16.014381 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:16:16.014212 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ba9b2b6-4be9-4303-8eab-cd464c5a32c7-metrics-tls podName:4ba9b2b6-4be9-4303-8eab-cd464c5a32c7 nodeName:}" failed. No retries permitted until 2026-11-02 06:16:20.014201377 +0000 UTC m=+41.906019624 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4ba9b2b6-4be9-4303-8eab-cd464c5a32c7-metrics-tls") pod "dns-default-gf726" (UID: "4ba9b2b6-4be9-4303-8eab-cd464c5a32c7") : secret "dns-default-metrics-tls" not found Nov 02 06:16:16.855145 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:16.854552 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-8fbcn" event={"ID":"121e38c0-6fa3-46aa-9191-5c6f91c8764c","Type":"ContainerStarted","Data":"483d50163b7bebfd218e328d3df6bb85939f0286a76377f91e7e7a6a99f98b62"} Nov 02 06:16:16.964671 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:16.964609 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-8fbcn" podStartSLOduration=235.041216003 podStartE2EDuration="3m57.964591311s" totalImagesPullingTime="2.923375308s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:12:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:16:16.963167511 +0000 UTC m=+38.854985775" watchObservedRunningTime="2026-11-02 06:16:16.964591311 +0000 UTC m=+38.856409574" Nov 02 06:16:16.967492 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:16.965653 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-kd68m" podStartSLOduration=271.965638522 podStartE2EDuration="4m31.965638522s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:11:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:16:14.367425981 +0000 UTC m=+36.259244237" watchObservedRunningTime="2026-11-02 06:16:16.965638522 +0000 UTC m=+38.857456786" Nov 02 06:16:18.868771 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:18.868551 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-csfhg_17dee72b-2f47-4016-8ffe-3a89934e66ee/console-operator/0.log" Nov 02 06:16:18.869518 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:18.868800 2579 generic.go:356] "Generic (PLEG): container finished" podID="17dee72b-2f47-4016-8ffe-3a89934e66ee" containerID="e105724101b90d7d0d3bfc67f75a1d0d3ac6bd499b0f368ee6fcfc3bb6df1647" exitCode=255 Nov 02 06:16:18.869518 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:18.868837 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-csfhg" event={"ID":"17dee72b-2f47-4016-8ffe-3a89934e66ee","Type":"ContainerDied","Data":"e105724101b90d7d0d3bfc67f75a1d0d3ac6bd499b0f368ee6fcfc3bb6df1647"} Nov 02 06:16:18.869518 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:18.869118 2579 scope.go:122] "RemoveContainer" containerID="e105724101b90d7d0d3bfc67f75a1d0d3ac6bd499b0f368ee6fcfc3bb6df1647" Nov 02 06:16:18.871956 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:18.871484 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-6tt27" event={"ID":"0d2b53c4-9d39-47fc-9367-c9a8e7f5da81","Type":"ContainerStarted","Data":"00507297e3924737a5595ffa4a809eb9c214b769465fa65eaa1dad75983cd9ef"} Nov 02 06:16:18.873775 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:18.873753 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-c8dcq" event={"ID":"37982aa4-bdbd-40e8-a2dd-5094209f3c96","Type":"ContainerStarted","Data":"7fb19f26c0ab0cab47b874832c74cf70aafb4ff88d00575d84440952233bd3e2"} Nov 02 06:16:19.093244 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:19.093110 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-c8dcq" podStartSLOduration=263.607260558 podStartE2EDuration="4m29.093091191s" totalImagesPullingTime="5.485830633s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:11:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:16:19.092370404 +0000 UTC m=+40.984188667" watchObservedRunningTime="2026-11-02 06:16:19.093091191 +0000 UTC m=+40.984909454" Nov 02 06:16:19.201222 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:19.201173 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-6tt27" podStartSLOduration=1.74557459 podStartE2EDuration="7.20115989s" totalImagesPullingTime="5.4555853s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:16:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:16:19.200730996 +0000 UTC m=+41.092549260" watchObservedRunningTime="2026-11-02 06:16:19.20115989 +0000 UTC m=+41.092978152" Nov 02 06:16:19.660935 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:19.660874 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-p98cq"] Nov 02 06:16:19.692832 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:19.692809 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-p98cq" Nov 02 06:16:19.746347 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:19.746317 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 02 06:16:19.746476 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:19.746370 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 02 06:16:19.746476 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:19.746395 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-75dck\"" Nov 02 06:16:19.747647 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:19.747625 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-p98cq"] Nov 02 06:16:19.855606 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:19.855516 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8b491a46-9f53-4a4f-97ed-d8b32e3a03fa-data-volume\") pod \"insights-runtime-extractor-p98cq\" (UID: \"8b491a46-9f53-4a4f-97ed-d8b32e3a03fa\") " pod="openshift-insights/insights-runtime-extractor-p98cq" Nov 02 06:16:19.855606 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:19.855588 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8b491a46-9f53-4a4f-97ed-d8b32e3a03fa-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p98cq\" (UID: \"8b491a46-9f53-4a4f-97ed-d8b32e3a03fa\") " pod="openshift-insights/insights-runtime-extractor-p98cq" Nov 02 06:16:19.858245 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:19.858214 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8b491a46-9f53-4a4f-97ed-d8b32e3a03fa-crio-socket\") pod \"insights-runtime-extractor-p98cq\" (UID: \"8b491a46-9f53-4a4f-97ed-d8b32e3a03fa\") " pod="openshift-insights/insights-runtime-extractor-p98cq" Nov 02 06:16:19.858384 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:19.858250 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpt4c\" (UniqueName: \"kubernetes.io/projected/8b491a46-9f53-4a4f-97ed-d8b32e3a03fa-kube-api-access-dpt4c\") pod \"insights-runtime-extractor-p98cq\" (UID: \"8b491a46-9f53-4a4f-97ed-d8b32e3a03fa\") " pod="openshift-insights/insights-runtime-extractor-p98cq" Nov 02 06:16:19.858384 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:19.858368 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8b491a46-9f53-4a4f-97ed-d8b32e3a03fa-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p98cq\" (UID: \"8b491a46-9f53-4a4f-97ed-d8b32e3a03fa\") " pod="openshift-insights/insights-runtime-extractor-p98cq" Nov 02 06:16:19.879013 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:19.878977 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-k9hcs" event={"ID":"20cb7690-c976-471c-b459-a9ab15cc51bf","Type":"ContainerStarted","Data":"8f6dd3529788666640dea61ff1d0c25eb6a809af44d16d6b71e80276e8809d87"} Nov 02 06:16:19.882736 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:19.882716 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-csfhg_17dee72b-2f47-4016-8ffe-3a89934e66ee/console-operator/1.log" Nov 02 06:16:19.883170 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:19.883154 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-csfhg_17dee72b-2f47-4016-8ffe-3a89934e66ee/console-operator/0.log" Nov 02 06:16:19.883223 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:19.883193 2579 generic.go:356] "Generic (PLEG): container finished" podID="17dee72b-2f47-4016-8ffe-3a89934e66ee" containerID="29a73f05a2b0c2acf34262be37902431d1000d396a0e844130d9fc3edea64b7b" exitCode=255 Nov 02 06:16:19.883298 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:19.883275 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-csfhg" event={"ID":"17dee72b-2f47-4016-8ffe-3a89934e66ee","Type":"ContainerDied","Data":"29a73f05a2b0c2acf34262be37902431d1000d396a0e844130d9fc3edea64b7b"} Nov 02 06:16:19.883351 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:19.883341 2579 scope.go:122] "RemoveContainer" containerID="e105724101b90d7d0d3bfc67f75a1d0d3ac6bd499b0f368ee6fcfc3bb6df1647" Nov 02 06:16:19.883562 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:19.883543 2579 scope.go:122] "RemoveContainer" containerID="29a73f05a2b0c2acf34262be37902431d1000d396a0e844130d9fc3edea64b7b" Nov 02 06:16:19.883787 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:16:19.883765 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-csfhg_openshift-console-operator(17dee72b-2f47-4016-8ffe-3a89934e66ee)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-csfhg" podUID="17dee72b-2f47-4016-8ffe-3a89934e66ee" Nov 02 06:16:19.884874 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:19.884854 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wcrtl" event={"ID":"cc33996a-0d4c-49c4-ba67-283bd6a2de14","Type":"ContainerStarted","Data":"0fcc6142d3a9a4c78a2751ebba8f71bcd77e10e57db430c4f043e35836afcda2"} Nov 02 06:16:19.907747 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:19.907716 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-rrgb2-xp5b2"] Nov 02 06:16:19.922784 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:19.922757 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-rrgb2-xp5b2" Nov 02 06:16:19.959200 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:19.959164 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fa9c5a91-d480-4d7b-9044-be6437a76a02-registry-tls\") pod \"image-registry-746899d75-dtjhs\" (UID: \"fa9c5a91-d480-4d7b-9044-be6437a76a02\") " pod="openshift-image-registry/image-registry-746899d75-dtjhs" Nov 02 06:16:19.959200 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:19.959199 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8b491a46-9f53-4a4f-97ed-d8b32e3a03fa-crio-socket\") pod \"insights-runtime-extractor-p98cq\" (UID: \"8b491a46-9f53-4a4f-97ed-d8b32e3a03fa\") " pod="openshift-insights/insights-runtime-extractor-p98cq" Nov 02 06:16:19.959440 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:16:19.959327 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 06:16:19.959440 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:16:19.959348 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-746899d75-dtjhs: secret "image-registry-tls" not found Nov 02 06:16:19.959440 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:19.959400 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpt4c\" (UniqueName: \"kubernetes.io/projected/8b491a46-9f53-4a4f-97ed-d8b32e3a03fa-kube-api-access-dpt4c\") pod \"insights-runtime-extractor-p98cq\" (UID: \"8b491a46-9f53-4a4f-97ed-d8b32e3a03fa\") " pod="openshift-insights/insights-runtime-extractor-p98cq" Nov 02 06:16:19.959440 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:19.959343 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8b491a46-9f53-4a4f-97ed-d8b32e3a03fa-crio-socket\") pod \"insights-runtime-extractor-p98cq\" (UID: \"8b491a46-9f53-4a4f-97ed-d8b32e3a03fa\") " pod="openshift-insights/insights-runtime-extractor-p98cq" Nov 02 06:16:19.959440 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:16:19.959421 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fa9c5a91-d480-4d7b-9044-be6437a76a02-registry-tls podName:fa9c5a91-d480-4d7b-9044-be6437a76a02 nodeName:}" failed. No retries permitted until 2026-11-02 06:16:27.959401466 +0000 UTC m=+49.851219720 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fa9c5a91-d480-4d7b-9044-be6437a76a02-registry-tls") pod "image-registry-746899d75-dtjhs" (UID: "fa9c5a91-d480-4d7b-9044-be6437a76a02") : secret "image-registry-tls" not found Nov 02 06:16:19.959679 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:19.959579 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45e5ce56-5cab-4546-8b31-02fc06cfa0cc-service-ca-bundle\") pod \"router-default-5f5c58667c-29wnm\" (UID: \"45e5ce56-5cab-4546-8b31-02fc06cfa0cc\") " pod="openshift-ingress/router-default-5f5c58667c-29wnm" Nov 02 06:16:19.959679 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:19.959607 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3fcb7621-86c8-448f-b4fa-b5bd76be7356-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-2rs6c\" (UID: \"3fcb7621-86c8-448f-b4fa-b5bd76be7356\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2rs6c" Nov 02 06:16:19.959679 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:19.959626 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8b491a46-9f53-4a4f-97ed-d8b32e3a03fa-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p98cq\" (UID: \"8b491a46-9f53-4a4f-97ed-d8b32e3a03fa\") " pod="openshift-insights/insights-runtime-extractor-p98cq" Nov 02 06:16:19.959679 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:19.959680 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45e5ce56-5cab-4546-8b31-02fc06cfa0cc-metrics-certs\") pod \"router-default-5f5c58667c-29wnm\" (UID: \"45e5ce56-5cab-4546-8b31-02fc06cfa0cc\") " pod="openshift-ingress/router-default-5f5c58667c-29wnm" Nov 02 06:16:19.959890 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:19.959696 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8b491a46-9f53-4a4f-97ed-d8b32e3a03fa-data-volume\") pod \"insights-runtime-extractor-p98cq\" (UID: \"8b491a46-9f53-4a4f-97ed-d8b32e3a03fa\") " pod="openshift-insights/insights-runtime-extractor-p98cq" Nov 02 06:16:19.959890 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:19.959733 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8b491a46-9f53-4a4f-97ed-d8b32e3a03fa-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p98cq\" (UID: \"8b491a46-9f53-4a4f-97ed-d8b32e3a03fa\") " pod="openshift-insights/insights-runtime-extractor-p98cq" Nov 02 06:16:19.959890 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:16:19.959741 2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 02 06:16:19.959890 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:16:19.959751 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/45e5ce56-5cab-4546-8b31-02fc06cfa0cc-service-ca-bundle podName:45e5ce56-5cab-4546-8b31-02fc06cfa0cc nodeName:}" failed. No retries permitted until 2026-11-02 06:16:27.959733068 +0000 UTC m=+49.851551324 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/45e5ce56-5cab-4546-8b31-02fc06cfa0cc-service-ca-bundle") pod "router-default-5f5c58667c-29wnm" (UID: "45e5ce56-5cab-4546-8b31-02fc06cfa0cc") : configmap references non-existent config key: service-ca.crt Nov 02 06:16:19.959890 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:16:19.959800 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3fcb7621-86c8-448f-b4fa-b5bd76be7356-cluster-monitoring-operator-tls podName:3fcb7621-86c8-448f-b4fa-b5bd76be7356 nodeName:}" failed. No retries permitted until 2026-11-02 06:16:27.959788437 +0000 UTC m=+49.851606681 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/3fcb7621-86c8-448f-b4fa-b5bd76be7356-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-2rs6c" (UID: "3fcb7621-86c8-448f-b4fa-b5bd76be7356") : secret "cluster-monitoring-operator-tls" not found Nov 02 06:16:19.959890 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:16:19.959823 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 06:16:19.959890 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:19.959839 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f0531b4d-c954-4830-85f6-fe876c76437a-registry-tls\") pod \"image-registry-6689c6d46b-vw558\" (UID: \"f0531b4d-c954-4830-85f6-fe876c76437a\") " pod="openshift-image-registry/image-registry-6689c6d46b-vw558" Nov 02 06:16:19.959890 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:16:19.959847 2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 02 06:16:19.959890 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:16:19.959854 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b491a46-9f53-4a4f-97ed-d8b32e3a03fa-insights-runtime-extractor-tls podName:8b491a46-9f53-4a4f-97ed-d8b32e3a03fa nodeName:}" failed. No retries permitted until 2026-11-02 06:16:20.459845095 +0000 UTC m=+42.351663367 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8b491a46-9f53-4a4f-97ed-d8b32e3a03fa-insights-runtime-extractor-tls") pod "insights-runtime-extractor-p98cq" (UID: "8b491a46-9f53-4a4f-97ed-d8b32e3a03fa") : secret "insights-runtime-extractor-tls" not found Nov 02 06:16:19.959890 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:16:19.959883 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45e5ce56-5cab-4546-8b31-02fc06cfa0cc-metrics-certs podName:45e5ce56-5cab-4546-8b31-02fc06cfa0cc nodeName:}" failed. No retries permitted until 2026-11-02 06:16:27.959870392 +0000 UTC m=+49.851688672 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/45e5ce56-5cab-4546-8b31-02fc06cfa0cc-metrics-certs") pod "router-default-5f5c58667c-29wnm" (UID: "45e5ce56-5cab-4546-8b31-02fc06cfa0cc") : secret "router-metrics-certs-default" not found Nov 02 06:16:19.959890 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:16:19.959887 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 02 06:16:19.960341 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:16:19.959962 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 06:16:19.960341 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:16:19.959975 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6689c6d46b-vw558: secret "image-registry-tls" not found Nov 02 06:16:19.960341 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:16:19.959954 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8b491a46-9f53-4a4f-97ed-d8b32e3a03fa-kube-rbac-proxy-cm podName:8b491a46-9f53-4a4f-97ed-d8b32e3a03fa nodeName:}" failed. No retries permitted until 2026-11-02 06:16:20.459940795 +0000 UTC m=+42.351759040 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8b491a46-9f53-4a4f-97ed-d8b32e3a03fa-kube-rbac-proxy-cm") pod "insights-runtime-extractor-p98cq" (UID: "8b491a46-9f53-4a4f-97ed-d8b32e3a03fa") : configmap "kube-rbac-proxy" not found Nov 02 06:16:19.960341 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:16:19.960012 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f0531b4d-c954-4830-85f6-fe876c76437a-registry-tls podName:f0531b4d-c954-4830-85f6-fe876c76437a nodeName:}" failed. No retries permitted until 2026-11-02 06:16:27.959998813 +0000 UTC m=+49.851817079 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f0531b4d-c954-4830-85f6-fe876c76437a-registry-tls") pod "image-registry-6689c6d46b-vw558" (UID: "f0531b4d-c954-4830-85f6-fe876c76437a") : secret "image-registry-tls" not found Nov 02 06:16:19.960341 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:19.960108 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8b491a46-9f53-4a4f-97ed-d8b32e3a03fa-data-volume\") pod \"insights-runtime-extractor-p98cq\" (UID: \"8b491a46-9f53-4a4f-97ed-d8b32e3a03fa\") " pod="openshift-insights/insights-runtime-extractor-p98cq" Nov 02 06:16:19.965045 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:19.965019 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-rrgb2-xp5b2"] Nov 02 06:16:19.973066 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:19.973048 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-mf7sr\"" Nov 02 06:16:19.989151 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:19.989101 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-k9hcs" podStartSLOduration=236.156981833 podStartE2EDuration="4m1.989089169s" totalImagesPullingTime="5.832107336s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:12:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:16:19.987738009 +0000 UTC m=+41.879556273" watchObservedRunningTime="2026-11-02 06:16:19.989089169 +0000 UTC m=+41.880907432" Nov 02 06:16:20.016190 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:20.016159 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpt4c\" (UniqueName: \"kubernetes.io/projected/8b491a46-9f53-4a4f-97ed-d8b32e3a03fa-kube-api-access-dpt4c\") pod \"insights-runtime-extractor-p98cq\" (UID: \"8b491a46-9f53-4a4f-97ed-d8b32e3a03fa\") " pod="openshift-insights/insights-runtime-extractor-p98cq" Nov 02 06:16:20.061143 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:20.061114 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/53aa1e5a-c35f-4aa6-9f23-cc61ccd331fc-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-fh4sz\" (UID: \"53aa1e5a-c35f-4aa6-9f23-cc61ccd331fc\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-fh4sz" Nov 02 06:16:20.061301 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:20.061208 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a4319db2-eda9-47a0-b742-985c8bdd7ea2-service-ca-bundle\") pod \"periodic-gathering-rrgb2-xp5b2\" (UID: \"a4319db2-eda9-47a0-b742-985c8bdd7ea2\") " pod="openshift-insights/periodic-gathering-rrgb2-xp5b2" Nov 02 06:16:20.061301 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:20.061254 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwdcw\" (UniqueName: \"kubernetes.io/projected/a4319db2-eda9-47a0-b742-985c8bdd7ea2-kube-api-access-qwdcw\") pod \"periodic-gathering-rrgb2-xp5b2\" (UID: \"a4319db2-eda9-47a0-b742-985c8bdd7ea2\") " pod="openshift-insights/periodic-gathering-rrgb2-xp5b2" Nov 02 06:16:20.061301 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:16:20.061260 2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 02 06:16:20.061301 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:20.061279 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/a4319db2-eda9-47a0-b742-985c8bdd7ea2-archives-path\") pod \"periodic-gathering-rrgb2-xp5b2\" (UID: \"a4319db2-eda9-47a0-b742-985c8bdd7ea2\") " pod="openshift-insights/periodic-gathering-rrgb2-xp5b2" Nov 02 06:16:20.061493 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:16:20.061349 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53aa1e5a-c35f-4aa6-9f23-cc61ccd331fc-networking-console-plugin-cert podName:53aa1e5a-c35f-4aa6-9f23-cc61ccd331fc nodeName:}" failed. No retries permitted until 2026-11-02 06:16:28.061331387 +0000 UTC m=+49.953149626 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/53aa1e5a-c35f-4aa6-9f23-cc61ccd331fc-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-fh4sz" (UID: "53aa1e5a-c35f-4aa6-9f23-cc61ccd331fc") : secret "networking-console-plugin-cert" not found Nov 02 06:16:20.061493 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:20.061402 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/cf8e8e65-8525-484b-adfe-f56517ccdc1b-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-d6bc4\" (UID: \"cf8e8e65-8525-484b-adfe-f56517ccdc1b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-d6bc4" Nov 02 06:16:20.061599 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:20.061495 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4ba9b2b6-4be9-4303-8eab-cd464c5a32c7-metrics-tls\") pod \"dns-default-gf726\" (UID: \"4ba9b2b6-4be9-4303-8eab-cd464c5a32c7\") " pod="openshift-dns/dns-default-gf726" Nov 02 06:16:20.061599 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:16:20.061584 2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 02 06:16:20.061687 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:16:20.061645 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf8e8e65-8525-484b-adfe-f56517ccdc1b-samples-operator-tls podName:cf8e8e65-8525-484b-adfe-f56517ccdc1b nodeName:}" failed. No retries permitted until 2026-11-02 06:16:28.061629194 +0000 UTC m=+49.953447434 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/cf8e8e65-8525-484b-adfe-f56517ccdc1b-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-d6bc4" (UID: "cf8e8e65-8525-484b-adfe-f56517ccdc1b") : secret "samples-operator-tls" not found Nov 02 06:16:20.061741 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:20.061702 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a4319db2-eda9-47a0-b742-985c8bdd7ea2-trusted-ca-bundle\") pod \"periodic-gathering-rrgb2-xp5b2\" (UID: \"a4319db2-eda9-47a0-b742-985c8bdd7ea2\") " pod="openshift-insights/periodic-gathering-rrgb2-xp5b2" Nov 02 06:16:20.061741 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:16:20.061714 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 06:16:20.061836 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:16:20.061759 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ba9b2b6-4be9-4303-8eab-cd464c5a32c7-metrics-tls podName:4ba9b2b6-4be9-4303-8eab-cd464c5a32c7 nodeName:}" failed. No retries permitted until 2026-11-02 06:16:28.06174404 +0000 UTC m=+49.953562286 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4ba9b2b6-4be9-4303-8eab-cd464c5a32c7-metrics-tls") pod "dns-default-gf726" (UID: "4ba9b2b6-4be9-4303-8eab-cd464c5a32c7") : secret "dns-default-metrics-tls" not found Nov 02 06:16:20.166185 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:20.166149 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a4319db2-eda9-47a0-b742-985c8bdd7ea2-trusted-ca-bundle\") pod \"periodic-gathering-rrgb2-xp5b2\" (UID: \"a4319db2-eda9-47a0-b742-985c8bdd7ea2\") " pod="openshift-insights/periodic-gathering-rrgb2-xp5b2" Nov 02 06:16:20.166349 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:20.166328 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a4319db2-eda9-47a0-b742-985c8bdd7ea2-service-ca-bundle\") pod \"periodic-gathering-rrgb2-xp5b2\" (UID: \"a4319db2-eda9-47a0-b742-985c8bdd7ea2\") " pod="openshift-insights/periodic-gathering-rrgb2-xp5b2" Nov 02 06:16:20.166414 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:20.166361 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwdcw\" (UniqueName: \"kubernetes.io/projected/a4319db2-eda9-47a0-b742-985c8bdd7ea2-kube-api-access-qwdcw\") pod \"periodic-gathering-rrgb2-xp5b2\" (UID: \"a4319db2-eda9-47a0-b742-985c8bdd7ea2\") " pod="openshift-insights/periodic-gathering-rrgb2-xp5b2" Nov 02 06:16:20.166414 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:20.166387 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/a4319db2-eda9-47a0-b742-985c8bdd7ea2-archives-path\") pod \"periodic-gathering-rrgb2-xp5b2\" (UID: \"a4319db2-eda9-47a0-b742-985c8bdd7ea2\") " pod="openshift-insights/periodic-gathering-rrgb2-xp5b2" Nov 02 06:16:20.166753 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:20.166731 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/a4319db2-eda9-47a0-b742-985c8bdd7ea2-archives-path\") pod \"periodic-gathering-rrgb2-xp5b2\" (UID: \"a4319db2-eda9-47a0-b742-985c8bdd7ea2\") " pod="openshift-insights/periodic-gathering-rrgb2-xp5b2" Nov 02 06:16:20.166849 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:20.166826 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a4319db2-eda9-47a0-b742-985c8bdd7ea2-service-ca-bundle\") pod \"periodic-gathering-rrgb2-xp5b2\" (UID: \"a4319db2-eda9-47a0-b742-985c8bdd7ea2\") " pod="openshift-insights/periodic-gathering-rrgb2-xp5b2" Nov 02 06:16:20.166884 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:20.166858 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a4319db2-eda9-47a0-b742-985c8bdd7ea2-trusted-ca-bundle\") pod \"periodic-gathering-rrgb2-xp5b2\" (UID: \"a4319db2-eda9-47a0-b742-985c8bdd7ea2\") " pod="openshift-insights/periodic-gathering-rrgb2-xp5b2" Nov 02 06:16:20.205799 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:20.205755 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wcrtl" podStartSLOduration=235.534974345 podStartE2EDuration="4m2.205743576s" totalImagesPullingTime="6.670769231s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:12:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:16:20.205006117 +0000 UTC m=+42.096824379" watchObservedRunningTime="2026-11-02 06:16:20.205743576 +0000 UTC m=+42.097561838" Nov 02 06:16:20.223517 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:20.223487 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwdcw\" (UniqueName: \"kubernetes.io/projected/a4319db2-eda9-47a0-b742-985c8bdd7ea2-kube-api-access-qwdcw\") pod \"periodic-gathering-rrgb2-xp5b2\" (UID: \"a4319db2-eda9-47a0-b742-985c8bdd7ea2\") " pod="openshift-insights/periodic-gathering-rrgb2-xp5b2" Nov 02 06:16:20.254054 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:20.254029 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-rrgb2-xp5b2" Nov 02 06:16:20.470765 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:20.470544 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8b491a46-9f53-4a4f-97ed-d8b32e3a03fa-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p98cq\" (UID: \"8b491a46-9f53-4a4f-97ed-d8b32e3a03fa\") " pod="openshift-insights/insights-runtime-extractor-p98cq" Nov 02 06:16:20.470923 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:16:20.470693 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 06:16:20.470923 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:20.470829 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8b491a46-9f53-4a4f-97ed-d8b32e3a03fa-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p98cq\" (UID: \"8b491a46-9f53-4a4f-97ed-d8b32e3a03fa\") " pod="openshift-insights/insights-runtime-extractor-p98cq" Nov 02 06:16:20.470923 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:16:20.470844 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b491a46-9f53-4a4f-97ed-d8b32e3a03fa-insights-runtime-extractor-tls podName:8b491a46-9f53-4a4f-97ed-d8b32e3a03fa nodeName:}" failed. No retries permitted until 2026-11-02 06:16:21.470829774 +0000 UTC m=+43.362648015 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8b491a46-9f53-4a4f-97ed-d8b32e3a03fa-insights-runtime-extractor-tls") pod "insights-runtime-extractor-p98cq" (UID: "8b491a46-9f53-4a4f-97ed-d8b32e3a03fa") : secret "insights-runtime-extractor-tls" not found Nov 02 06:16:20.470923 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:16:20.470874 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 02 06:16:20.471077 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:16:20.470934 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8b491a46-9f53-4a4f-97ed-d8b32e3a03fa-kube-rbac-proxy-cm podName:8b491a46-9f53-4a4f-97ed-d8b32e3a03fa nodeName:}" failed. No retries permitted until 2026-11-02 06:16:21.470900352 +0000 UTC m=+43.362718592 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8b491a46-9f53-4a4f-97ed-d8b32e3a03fa-kube-rbac-proxy-cm") pod "insights-runtime-extractor-p98cq" (UID: "8b491a46-9f53-4a4f-97ed-d8b32e3a03fa") : configmap "kube-rbac-proxy" not found Nov 02 06:16:20.643309 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:20.643276 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-rrgb2-xp5b2"] Nov 02 06:16:20.650185 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:16:20.650158 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4319db2_eda9_47a0_b742_985c8bdd7ea2.slice/crio-6f2af3023df9565bf072c1569930184b0aa5c7af28b5d5da6f6d32982f397446 WatchSource:0}: Error finding container 6f2af3023df9565bf072c1569930184b0aa5c7af28b5d5da6f6d32982f397446: Status 404 returned error can't find the container with id 6f2af3023df9565bf072c1569930184b0aa5c7af28b5d5da6f6d32982f397446 Nov 02 06:16:20.889453 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:20.889396 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-rrgb2-xp5b2" event={"ID":"a4319db2-eda9-47a0-b742-985c8bdd7ea2","Type":"ContainerStarted","Data":"5cc58db3667246c8ed607f7c7c5dd57762103332123a56f39b752d4af4280288"} Nov 02 06:16:20.890222 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:20.889470 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-rrgb2-xp5b2" event={"ID":"a4319db2-eda9-47a0-b742-985c8bdd7ea2","Type":"ContainerStarted","Data":"6f2af3023df9565bf072c1569930184b0aa5c7af28b5d5da6f6d32982f397446"} Nov 02 06:16:20.890975 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:20.890959 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-csfhg_17dee72b-2f47-4016-8ffe-3a89934e66ee/console-operator/1.log" Nov 02 06:16:20.891211 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:20.891199 2579 scope.go:122] "RemoveContainer" containerID="29a73f05a2b0c2acf34262be37902431d1000d396a0e844130d9fc3edea64b7b" Nov 02 06:16:20.891360 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:16:20.891343 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-csfhg_openshift-console-operator(17dee72b-2f47-4016-8ffe-3a89934e66ee)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-csfhg" podUID="17dee72b-2f47-4016-8ffe-3a89934e66ee" Nov 02 06:16:20.998589 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:20.998545 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-rrgb2-xp5b2" podStartSLOduration=1.998531735 podStartE2EDuration="1.998531735s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:16:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:16:20.997969826 +0000 UTC m=+42.889788108" watchObservedRunningTime="2026-11-02 06:16:20.998531735 +0000 UTC m=+42.890349998" Nov 02 06:16:21.481199 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:21.481166 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8b491a46-9f53-4a4f-97ed-d8b32e3a03fa-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p98cq\" (UID: \"8b491a46-9f53-4a4f-97ed-d8b32e3a03fa\") " pod="openshift-insights/insights-runtime-extractor-p98cq" Nov 02 06:16:21.481369 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:21.481272 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8b491a46-9f53-4a4f-97ed-d8b32e3a03fa-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p98cq\" (UID: \"8b491a46-9f53-4a4f-97ed-d8b32e3a03fa\") " pod="openshift-insights/insights-runtime-extractor-p98cq" Nov 02 06:16:21.481369 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:16:21.481285 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 02 06:16:21.481369 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:16:21.481348 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8b491a46-9f53-4a4f-97ed-d8b32e3a03fa-kube-rbac-proxy-cm podName:8b491a46-9f53-4a4f-97ed-d8b32e3a03fa nodeName:}" failed. No retries permitted until 2026-11-02 06:16:23.481331667 +0000 UTC m=+45.373149929 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8b491a46-9f53-4a4f-97ed-d8b32e3a03fa-kube-rbac-proxy-cm") pod "insights-runtime-extractor-p98cq" (UID: "8b491a46-9f53-4a4f-97ed-d8b32e3a03fa") : configmap "kube-rbac-proxy" not found Nov 02 06:16:21.481501 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:16:21.481377 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 06:16:21.481501 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:16:21.481412 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b491a46-9f53-4a4f-97ed-d8b32e3a03fa-insights-runtime-extractor-tls podName:8b491a46-9f53-4a4f-97ed-d8b32e3a03fa nodeName:}" failed. No retries permitted until 2026-11-02 06:16:23.481401476 +0000 UTC m=+45.373219719 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8b491a46-9f53-4a4f-97ed-d8b32e3a03fa-insights-runtime-extractor-tls") pod "insights-runtime-extractor-p98cq" (UID: "8b491a46-9f53-4a4f-97ed-d8b32e3a03fa") : secret "insights-runtime-extractor-tls" not found Nov 02 06:16:22.334308 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:22.334274 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-96jpc"] Nov 02 06:16:22.353450 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:22.353422 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-96jpc" Nov 02 06:16:22.390181 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:22.390153 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-96jpc"] Nov 02 06:16:22.390955 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:22.390900 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgkrw\" (UniqueName: \"kubernetes.io/projected/ac6faf80-42b2-4c7a-8e55-abf6435b3d1e-kube-api-access-pgkrw\") pod \"service-ca-5fffc97bdf-96jpc\" (UID: \"ac6faf80-42b2-4c7a-8e55-abf6435b3d1e\") " pod="openshift-service-ca/service-ca-5fffc97bdf-96jpc" Nov 02 06:16:22.391091 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:22.391024 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/ac6faf80-42b2-4c7a-8e55-abf6435b3d1e-signing-key\") pod \"service-ca-5fffc97bdf-96jpc\" (UID: \"ac6faf80-42b2-4c7a-8e55-abf6435b3d1e\") " pod="openshift-service-ca/service-ca-5fffc97bdf-96jpc" Nov 02 06:16:22.391169 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:22.391134 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/ac6faf80-42b2-4c7a-8e55-abf6435b3d1e-signing-cabundle\") pod \"service-ca-5fffc97bdf-96jpc\" (UID: \"ac6faf80-42b2-4c7a-8e55-abf6435b3d1e\") " pod="openshift-service-ca/service-ca-5fffc97bdf-96jpc" Nov 02 06:16:22.436142 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:22.436114 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Nov 02 06:16:22.436306 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:22.436195 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-v49gc\"" Nov 02 06:16:22.450703 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:22.450680 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Nov 02 06:16:22.450932 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:22.450887 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Nov 02 06:16:22.451028 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:22.451012 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Nov 02 06:16:22.492149 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:22.492115 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgkrw\" (UniqueName: \"kubernetes.io/projected/ac6faf80-42b2-4c7a-8e55-abf6435b3d1e-kube-api-access-pgkrw\") pod \"service-ca-5fffc97bdf-96jpc\" (UID: \"ac6faf80-42b2-4c7a-8e55-abf6435b3d1e\") " pod="openshift-service-ca/service-ca-5fffc97bdf-96jpc" Nov 02 06:16:22.492149 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:22.492155 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/ac6faf80-42b2-4c7a-8e55-abf6435b3d1e-signing-key\") pod \"service-ca-5fffc97bdf-96jpc\" (UID: \"ac6faf80-42b2-4c7a-8e55-abf6435b3d1e\") " pod="openshift-service-ca/service-ca-5fffc97bdf-96jpc" Nov 02 06:16:22.492335 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:22.492199 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/ac6faf80-42b2-4c7a-8e55-abf6435b3d1e-signing-cabundle\") pod \"service-ca-5fffc97bdf-96jpc\" (UID: \"ac6faf80-42b2-4c7a-8e55-abf6435b3d1e\") " pod="openshift-service-ca/service-ca-5fffc97bdf-96jpc" Nov 02 06:16:22.492861 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:22.492839 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/ac6faf80-42b2-4c7a-8e55-abf6435b3d1e-signing-cabundle\") pod \"service-ca-5fffc97bdf-96jpc\" (UID: \"ac6faf80-42b2-4c7a-8e55-abf6435b3d1e\") " pod="openshift-service-ca/service-ca-5fffc97bdf-96jpc" Nov 02 06:16:22.496133 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:22.496113 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/ac6faf80-42b2-4c7a-8e55-abf6435b3d1e-signing-key\") pod \"service-ca-5fffc97bdf-96jpc\" (UID: \"ac6faf80-42b2-4c7a-8e55-abf6435b3d1e\") " pod="openshift-service-ca/service-ca-5fffc97bdf-96jpc" Nov 02 06:16:22.547630 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:22.547594 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-csfhg" Nov 02 06:16:22.548075 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:22.548056 2579 scope.go:122] "RemoveContainer" containerID="29a73f05a2b0c2acf34262be37902431d1000d396a0e844130d9fc3edea64b7b" Nov 02 06:16:22.548610 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:16:22.548247 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-csfhg_openshift-console-operator(17dee72b-2f47-4016-8ffe-3a89934e66ee)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-csfhg" podUID="17dee72b-2f47-4016-8ffe-3a89934e66ee" Nov 02 06:16:22.550086 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:22.550065 2579 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-csfhg" Nov 02 06:16:22.550546 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:22.550525 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgkrw\" (UniqueName: \"kubernetes.io/projected/ac6faf80-42b2-4c7a-8e55-abf6435b3d1e-kube-api-access-pgkrw\") pod \"service-ca-5fffc97bdf-96jpc\" (UID: \"ac6faf80-42b2-4c7a-8e55-abf6435b3d1e\") " pod="openshift-service-ca/service-ca-5fffc97bdf-96jpc" Nov 02 06:16:22.680359 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:22.680329 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-96jpc" Nov 02 06:16:22.785275 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:22.785105 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-2p95f_d8d87317-3f41-46d4-a58e-3c7601e0d445/dns-node-resolver/0.log" Nov 02 06:16:22.901185 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:22.901161 2579 scope.go:122] "RemoveContainer" containerID="29a73f05a2b0c2acf34262be37902431d1000d396a0e844130d9fc3edea64b7b" Nov 02 06:16:22.901337 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:16:22.901319 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-csfhg_openshift-console-operator(17dee72b-2f47-4016-8ffe-3a89934e66ee)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-csfhg" podUID="17dee72b-2f47-4016-8ffe-3a89934e66ee" Nov 02 06:16:23.054958 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:23.054848 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-96jpc"] Nov 02 06:16:23.060286 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:16:23.060258 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac6faf80_42b2_4c7a_8e55_abf6435b3d1e.slice/crio-4b603936d475f6f974e687dc0306c917a65be18ac6c747bade66b4f9b09d5a3e WatchSource:0}: Error finding container 4b603936d475f6f974e687dc0306c917a65be18ac6c747bade66b4f9b09d5a3e: Status 404 returned error can't find the container with id 4b603936d475f6f974e687dc0306c917a65be18ac6c747bade66b4f9b09d5a3e Nov 02 06:16:23.501653 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:23.501623 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8b491a46-9f53-4a4f-97ed-d8b32e3a03fa-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p98cq\" (UID: \"8b491a46-9f53-4a4f-97ed-d8b32e3a03fa\") " pod="openshift-insights/insights-runtime-extractor-p98cq" Nov 02 06:16:23.502129 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:23.501808 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8b491a46-9f53-4a4f-97ed-d8b32e3a03fa-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p98cq\" (UID: \"8b491a46-9f53-4a4f-97ed-d8b32e3a03fa\") " pod="openshift-insights/insights-runtime-extractor-p98cq" Nov 02 06:16:23.502129 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:16:23.501818 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 06:16:23.502129 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:16:23.501923 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b491a46-9f53-4a4f-97ed-d8b32e3a03fa-insights-runtime-extractor-tls podName:8b491a46-9f53-4a4f-97ed-d8b32e3a03fa nodeName:}" failed. No retries permitted until 2026-11-02 06:16:27.50188666 +0000 UTC m=+49.393704922 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8b491a46-9f53-4a4f-97ed-d8b32e3a03fa-insights-runtime-extractor-tls") pod "insights-runtime-extractor-p98cq" (UID: "8b491a46-9f53-4a4f-97ed-d8b32e3a03fa") : secret "insights-runtime-extractor-tls" not found Nov 02 06:16:23.502414 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:23.502394 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8b491a46-9f53-4a4f-97ed-d8b32e3a03fa-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p98cq\" (UID: \"8b491a46-9f53-4a4f-97ed-d8b32e3a03fa\") " pod="openshift-insights/insights-runtime-extractor-p98cq" Nov 02 06:16:23.905757 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:23.905719 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-sktt2" event={"ID":"12b6cc1c-537d-4125-92ff-fe5f6927c53e","Type":"ContainerStarted","Data":"4191b5d75eba96c8e2cd049f0f075d84f7480825b2c51619e1348610d941d6e9"} Nov 02 06:16:23.907217 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:23.907176 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-96jpc" event={"ID":"ac6faf80-42b2-4c7a-8e55-abf6435b3d1e","Type":"ContainerStarted","Data":"dc3a898024c6dcd88744f4284bd5d221497dd6a0f585fa1bbf754b47b37ba9a9"} Nov 02 06:16:23.907374 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:23.907227 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-96jpc" event={"ID":"ac6faf80-42b2-4c7a-8e55-abf6435b3d1e","Type":"ContainerStarted","Data":"4b603936d475f6f974e687dc0306c917a65be18ac6c747bade66b4f9b09d5a3e"} Nov 02 06:16:23.968231 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:23.968201 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-qm6wn_3eec1082-df0b-420b-b4ff-5a1c0fab1d1d/node-ca/0.log" Nov 02 06:16:24.015887 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:24.015840 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-sktt2" podStartSLOduration=236.178193804 podStartE2EDuration="4m6.015827817s" totalImagesPullingTime="9.837634013s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:12:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:16:24.01525809 +0000 UTC m=+45.907076372" watchObservedRunningTime="2026-11-02 06:16:24.015827817 +0000 UTC m=+45.907646078" Nov 02 06:16:24.128104 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:24.128054 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca/service-ca-5fffc97bdf-96jpc" podStartSLOduration=2.128040403 podStartE2EDuration="2.128040403s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:16:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:16:24.126819146 +0000 UTC m=+46.018637408" watchObservedRunningTime="2026-11-02 06:16:24.128040403 +0000 UTC m=+46.019858662" Nov 02 06:16:24.579342 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:24.579313 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-6tt27_0d2b53c4-9d39-47fc-9367-c9a8e7f5da81/serve-healthcheck-canary/0.log" Nov 02 06:16:27.541899 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:27.541857 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8b491a46-9f53-4a4f-97ed-d8b32e3a03fa-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p98cq\" (UID: \"8b491a46-9f53-4a4f-97ed-d8b32e3a03fa\") " pod="openshift-insights/insights-runtime-extractor-p98cq" Nov 02 06:16:27.542376 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:16:27.542334 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 06:16:27.542456 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:16:27.542419 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b491a46-9f53-4a4f-97ed-d8b32e3a03fa-insights-runtime-extractor-tls podName:8b491a46-9f53-4a4f-97ed-d8b32e3a03fa nodeName:}" failed. No retries permitted until 2026-11-02 06:16:35.542391787 +0000 UTC m=+57.434210031 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8b491a46-9f53-4a4f-97ed-d8b32e3a03fa-insights-runtime-extractor-tls") pod "insights-runtime-extractor-p98cq" (UID: "8b491a46-9f53-4a4f-97ed-d8b32e3a03fa") : secret "insights-runtime-extractor-tls" not found Nov 02 06:16:28.047001 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:28.046968 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f0531b4d-c954-4830-85f6-fe876c76437a-registry-tls\") pod \"image-registry-6689c6d46b-vw558\" (UID: \"f0531b4d-c954-4830-85f6-fe876c76437a\") " pod="openshift-image-registry/image-registry-6689c6d46b-vw558" Nov 02 06:16:28.047159 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:28.047027 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fa9c5a91-d480-4d7b-9044-be6437a76a02-registry-tls\") pod \"image-registry-746899d75-dtjhs\" (UID: \"fa9c5a91-d480-4d7b-9044-be6437a76a02\") " pod="openshift-image-registry/image-registry-746899d75-dtjhs" Nov 02 06:16:28.047159 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:28.047078 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45e5ce56-5cab-4546-8b31-02fc06cfa0cc-service-ca-bundle\") pod \"router-default-5f5c58667c-29wnm\" (UID: \"45e5ce56-5cab-4546-8b31-02fc06cfa0cc\") " pod="openshift-ingress/router-default-5f5c58667c-29wnm" Nov 02 06:16:28.047159 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:28.047094 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3fcb7621-86c8-448f-b4fa-b5bd76be7356-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-2rs6c\" (UID: \"3fcb7621-86c8-448f-b4fa-b5bd76be7356\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2rs6c" Nov 02 06:16:28.047159 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:28.047141 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45e5ce56-5cab-4546-8b31-02fc06cfa0cc-metrics-certs\") pod \"router-default-5f5c58667c-29wnm\" (UID: \"45e5ce56-5cab-4546-8b31-02fc06cfa0cc\") " pod="openshift-ingress/router-default-5f5c58667c-29wnm" Nov 02 06:16:28.047381 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:16:28.047245 2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 02 06:16:28.047381 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:16:28.047323 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3fcb7621-86c8-448f-b4fa-b5bd76be7356-cluster-monitoring-operator-tls podName:3fcb7621-86c8-448f-b4fa-b5bd76be7356 nodeName:}" failed. No retries permitted until 2026-11-02 06:16:44.047302428 +0000 UTC m=+65.939120672 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/3fcb7621-86c8-448f-b4fa-b5bd76be7356-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-2rs6c" (UID: "3fcb7621-86c8-448f-b4fa-b5bd76be7356") : secret "cluster-monitoring-operator-tls" not found Nov 02 06:16:28.047745 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:28.047728 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45e5ce56-5cab-4546-8b31-02fc06cfa0cc-service-ca-bundle\") pod \"router-default-5f5c58667c-29wnm\" (UID: \"45e5ce56-5cab-4546-8b31-02fc06cfa0cc\") " pod="openshift-ingress/router-default-5f5c58667c-29wnm" Nov 02 06:16:28.050746 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:28.050724 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45e5ce56-5cab-4546-8b31-02fc06cfa0cc-metrics-certs\") pod \"router-default-5f5c58667c-29wnm\" (UID: \"45e5ce56-5cab-4546-8b31-02fc06cfa0cc\") " pod="openshift-ingress/router-default-5f5c58667c-29wnm" Nov 02 06:16:28.050830 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:28.050724 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fa9c5a91-d480-4d7b-9044-be6437a76a02-registry-tls\") pod \"image-registry-746899d75-dtjhs\" (UID: \"fa9c5a91-d480-4d7b-9044-be6437a76a02\") " pod="openshift-image-registry/image-registry-746899d75-dtjhs" Nov 02 06:16:28.050875 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:28.050819 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f0531b4d-c954-4830-85f6-fe876c76437a-registry-tls\") pod \"image-registry-6689c6d46b-vw558\" (UID: \"f0531b4d-c954-4830-85f6-fe876c76437a\") " pod="openshift-image-registry/image-registry-6689c6d46b-vw558" Nov 02 06:16:28.082628 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:28.082593 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6689c6d46b-vw558" Nov 02 06:16:28.098639 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:28.098611 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-746899d75-dtjhs" Nov 02 06:16:28.112468 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:28.112443 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5f5c58667c-29wnm" Nov 02 06:16:28.150821 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:28.148553 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/cf8e8e65-8525-484b-adfe-f56517ccdc1b-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-d6bc4\" (UID: \"cf8e8e65-8525-484b-adfe-f56517ccdc1b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-d6bc4" Nov 02 06:16:28.150821 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:16:28.148732 2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 02 06:16:28.150821 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:28.148745 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4ba9b2b6-4be9-4303-8eab-cd464c5a32c7-metrics-tls\") pod \"dns-default-gf726\" (UID: \"4ba9b2b6-4be9-4303-8eab-cd464c5a32c7\") " pod="openshift-dns/dns-default-gf726" Nov 02 06:16:28.150821 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:16:28.148810 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf8e8e65-8525-484b-adfe-f56517ccdc1b-samples-operator-tls podName:cf8e8e65-8525-484b-adfe-f56517ccdc1b nodeName:}" failed. No retries permitted until 2026-11-02 06:16:44.148789516 +0000 UTC m=+66.040607776 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/cf8e8e65-8525-484b-adfe-f56517ccdc1b-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-d6bc4" (UID: "cf8e8e65-8525-484b-adfe-f56517ccdc1b") : secret "samples-operator-tls" not found Nov 02 06:16:28.150821 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:28.148944 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/53aa1e5a-c35f-4aa6-9f23-cc61ccd331fc-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-fh4sz\" (UID: \"53aa1e5a-c35f-4aa6-9f23-cc61ccd331fc\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-fh4sz" Nov 02 06:16:28.153508 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:28.153461 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4ba9b2b6-4be9-4303-8eab-cd464c5a32c7-metrics-tls\") pod \"dns-default-gf726\" (UID: \"4ba9b2b6-4be9-4303-8eab-cd464c5a32c7\") " pod="openshift-dns/dns-default-gf726" Nov 02 06:16:28.155275 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:28.155247 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/53aa1e5a-c35f-4aa6-9f23-cc61ccd331fc-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-fh4sz\" (UID: \"53aa1e5a-c35f-4aa6-9f23-cc61ccd331fc\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-fh4sz" Nov 02 06:16:28.199172 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:28.199143 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-fh4sz" Nov 02 06:16:28.239432 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:28.239390 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-gf726" Nov 02 06:16:28.559811 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:28.559785 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-746899d75-dtjhs"] Nov 02 06:16:28.565667 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:16:28.565642 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa9c5a91_d480_4d7b_9044_be6437a76a02.slice/crio-1309dfc627fd0eddb9d56bfeb55c303d58564eed884481f3f3119988a12fbe55 WatchSource:0}: Error finding container 1309dfc627fd0eddb9d56bfeb55c303d58564eed884481f3f3119988a12fbe55: Status 404 returned error can't find the container with id 1309dfc627fd0eddb9d56bfeb55c303d58564eed884481f3f3119988a12fbe55 Nov 02 06:16:28.607861 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:28.607834 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6689c6d46b-vw558"] Nov 02 06:16:28.609004 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:28.608986 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5f5c58667c-29wnm"] Nov 02 06:16:28.625039 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:16:28.625011 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45e5ce56_5cab_4546_8b31_02fc06cfa0cc.slice/crio-12a8615ab34e8cb0bbe17c3b60704970d0bcd860a9a74b7984b41986462505ea WatchSource:0}: Error finding container 12a8615ab34e8cb0bbe17c3b60704970d0bcd860a9a74b7984b41986462505ea: Status 404 returned error can't find the container with id 12a8615ab34e8cb0bbe17c3b60704970d0bcd860a9a74b7984b41986462505ea Nov 02 06:16:28.625318 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:16:28.625299 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0531b4d_c954_4830_85f6_fe876c76437a.slice/crio-a471a58ea4a732d84e89fb09f437def55b26113096801bc4a4e213c7894fbe5b WatchSource:0}: Error finding container a471a58ea4a732d84e89fb09f437def55b26113096801bc4a4e213c7894fbe5b: Status 404 returned error can't find the container with id a471a58ea4a732d84e89fb09f437def55b26113096801bc4a4e213c7894fbe5b Nov 02 06:16:28.670531 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:28.670506 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-fh4sz"] Nov 02 06:16:28.672691 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:28.672664 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-gf726"] Nov 02 06:16:28.679211 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:16:28.679187 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53aa1e5a_c35f_4aa6_9f23_cc61ccd331fc.slice/crio-d319bf628cd7bcbfc1dcc73a4d9d12b8dfb00fc85e674cf3e9929dca6b16bfc3 WatchSource:0}: Error finding container d319bf628cd7bcbfc1dcc73a4d9d12b8dfb00fc85e674cf3e9929dca6b16bfc3: Status 404 returned error can't find the container with id d319bf628cd7bcbfc1dcc73a4d9d12b8dfb00fc85e674cf3e9929dca6b16bfc3 Nov 02 06:16:28.679872 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:16:28.679848 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ba9b2b6_4be9_4303_8eab_cd464c5a32c7.slice/crio-8b4aca9cba52d27e69b569155b9d1855545be2e53e75210dc3ffe05d64dd23df WatchSource:0}: Error finding container 8b4aca9cba52d27e69b569155b9d1855545be2e53e75210dc3ffe05d64dd23df: Status 404 returned error can't find the container with id 8b4aca9cba52d27e69b569155b9d1855545be2e53e75210dc3ffe05d64dd23df Nov 02 06:16:28.925607 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:28.925569 2579 generic.go:356] "Generic (PLEG): container finished" podID="12b6cc1c-537d-4125-92ff-fe5f6927c53e" containerID="4191b5d75eba96c8e2cd049f0f075d84f7480825b2c51619e1348610d941d6e9" exitCode=0 Nov 02 06:16:28.925798 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:28.925658 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-sktt2" event={"ID":"12b6cc1c-537d-4125-92ff-fe5f6927c53e","Type":"ContainerDied","Data":"4191b5d75eba96c8e2cd049f0f075d84f7480825b2c51619e1348610d941d6e9"} Nov 02 06:16:28.926126 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:28.926104 2579 scope.go:122] "RemoveContainer" containerID="4191b5d75eba96c8e2cd049f0f075d84f7480825b2c51619e1348610d941d6e9" Nov 02 06:16:28.927049 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:28.927016 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gf726" event={"ID":"4ba9b2b6-4be9-4303-8eab-cd464c5a32c7","Type":"ContainerStarted","Data":"8b4aca9cba52d27e69b569155b9d1855545be2e53e75210dc3ffe05d64dd23df"} Nov 02 06:16:28.928256 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:28.928220 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-fh4sz" event={"ID":"53aa1e5a-c35f-4aa6-9f23-cc61ccd331fc","Type":"ContainerStarted","Data":"d319bf628cd7bcbfc1dcc73a4d9d12b8dfb00fc85e674cf3e9929dca6b16bfc3"} Nov 02 06:16:28.929918 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:28.929874 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5f5c58667c-29wnm" event={"ID":"45e5ce56-5cab-4546-8b31-02fc06cfa0cc","Type":"ContainerStarted","Data":"0aa76bbbc1c98942981e1be3a5d55f3462bc2fdceefe68052b93bd731efd86d1"} Nov 02 06:16:28.930010 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:28.929947 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5f5c58667c-29wnm" event={"ID":"45e5ce56-5cab-4546-8b31-02fc06cfa0cc","Type":"ContainerStarted","Data":"12a8615ab34e8cb0bbe17c3b60704970d0bcd860a9a74b7984b41986462505ea"} Nov 02 06:16:28.935016 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:28.934991 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-746899d75-dtjhs" event={"ID":"fa9c5a91-d480-4d7b-9044-be6437a76a02","Type":"ContainerStarted","Data":"8ec81ab82bca12552c048070bb8e6d8403afabfda731aa44831bb2fb8e97cf17"} Nov 02 06:16:28.935121 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:28.935032 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-746899d75-dtjhs" event={"ID":"fa9c5a91-d480-4d7b-9044-be6437a76a02","Type":"ContainerStarted","Data":"1309dfc627fd0eddb9d56bfeb55c303d58564eed884481f3f3119988a12fbe55"} Nov 02 06:16:28.935287 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:28.935122 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-746899d75-dtjhs" Nov 02 06:16:28.937168 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:28.937149 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6689c6d46b-vw558" event={"ID":"f0531b4d-c954-4830-85f6-fe876c76437a","Type":"ContainerStarted","Data":"8d5f9daaf11f23bb864f8bc6039339327cfe071ed1a68abec1f2b1a7919f8a3d"} Nov 02 06:16:28.937275 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:28.937183 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6689c6d46b-vw558" event={"ID":"f0531b4d-c954-4830-85f6-fe876c76437a","Type":"ContainerStarted","Data":"a471a58ea4a732d84e89fb09f437def55b26113096801bc4a4e213c7894fbe5b"} Nov 02 06:16:28.937275 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:28.937208 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6689c6d46b-vw558" Nov 02 06:16:29.113269 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:29.113240 2579 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-5f5c58667c-29wnm" Nov 02 06:16:29.116096 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:29.116061 2579 patch_prober.go:29] interesting pod/router-default-5f5c58667c-29wnm container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-proxy-http failed: reason withheld Nov 02 06:16:29.116096 ip-10-0-130-93 kubenswrapper[2579]: [-]has-synced failed: reason withheld Nov 02 06:16:29.116096 ip-10-0-130-93 kubenswrapper[2579]: [+]process-running ok Nov 02 06:16:29.116096 ip-10-0-130-93 kubenswrapper[2579]: healthz check failed Nov 02 06:16:29.116287 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:29.116115 2579 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5f5c58667c-29wnm" podUID="45e5ce56-5cab-4546-8b31-02fc06cfa0cc" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Nov 02 06:16:29.149376 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:29.149317 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/router-default-5f5c58667c-29wnm" podStartSLOduration=276.149295718 podStartE2EDuration="4m36.149295718s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:11:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:16:29.14746112 +0000 UTC m=+51.039279383" watchObservedRunningTime="2026-11-02 06:16:29.149295718 +0000 UTC m=+51.041113980" Nov 02 06:16:29.263245 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:29.263152 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-746899d75-dtjhs" podStartSLOduration=49.263139112 podStartE2EDuration="49.263139112s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:15:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:16:29.261698996 +0000 UTC m=+51.153517270" watchObservedRunningTime="2026-11-02 06:16:29.263139112 +0000 UTC m=+51.154957425" Nov 02 06:16:29.372891 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:29.372842 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6689c6d46b-vw558" podStartSLOduration=234.372829488 podStartE2EDuration="3m54.372829488s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:12:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:16:29.372353305 +0000 UTC m=+51.264171567" watchObservedRunningTime="2026-11-02 06:16:29.372829488 +0000 UTC m=+51.264647750" Nov 02 06:16:29.648192 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:29.648158 2579 scope.go:122] "RemoveContainer" containerID="29a73f05a2b0c2acf34262be37902431d1000d396a0e844130d9fc3edea64b7b" Nov 02 06:16:29.943640 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:29.943558 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-csfhg_17dee72b-2f47-4016-8ffe-3a89934e66ee/console-operator/2.log" Nov 02 06:16:29.944146 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:29.944119 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-csfhg_17dee72b-2f47-4016-8ffe-3a89934e66ee/console-operator/1.log" Nov 02 06:16:29.944286 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:29.944177 2579 generic.go:356] "Generic (PLEG): container finished" podID="17dee72b-2f47-4016-8ffe-3a89934e66ee" containerID="8bd3233465a5560a8e5e0dc6a2c9a27ed2a1f235d342c92357cf3a9a272a46f8" exitCode=255 Nov 02 06:16:29.944355 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:29.944286 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-csfhg" event={"ID":"17dee72b-2f47-4016-8ffe-3a89934e66ee","Type":"ContainerDied","Data":"8bd3233465a5560a8e5e0dc6a2c9a27ed2a1f235d342c92357cf3a9a272a46f8"} Nov 02 06:16:29.944355 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:29.944340 2579 scope.go:122] "RemoveContainer" containerID="29a73f05a2b0c2acf34262be37902431d1000d396a0e844130d9fc3edea64b7b" Nov 02 06:16:29.944934 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:29.944741 2579 scope.go:122] "RemoveContainer" containerID="8bd3233465a5560a8e5e0dc6a2c9a27ed2a1f235d342c92357cf3a9a272a46f8" Nov 02 06:16:29.945057 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:16:29.944994 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-csfhg_openshift-console-operator(17dee72b-2f47-4016-8ffe-3a89934e66ee)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-csfhg" podUID="17dee72b-2f47-4016-8ffe-3a89934e66ee" Nov 02 06:16:29.948702 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:29.948639 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-sktt2" event={"ID":"12b6cc1c-537d-4125-92ff-fe5f6927c53e","Type":"ContainerStarted","Data":"07c143145dedb6529f2e52587b5cef573fe50ec0d1ed9d214a39a8c4b4f2ebf5"} Nov 02 06:16:30.114481 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:30.114448 2579 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-5f5c58667c-29wnm" Nov 02 06:16:30.954249 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:30.954180 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-csfhg_17dee72b-2f47-4016-8ffe-3a89934e66ee/console-operator/2.log" Nov 02 06:16:30.955947 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:30.955915 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-fh4sz" event={"ID":"53aa1e5a-c35f-4aa6-9f23-cc61ccd331fc","Type":"ContainerStarted","Data":"c22616662dbb50b96fd3e842d95debccfb350b859db4509a77a34a0374688e63"} Nov 02 06:16:30.956263 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:30.956240 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-5f5c58667c-29wnm" Nov 02 06:16:30.957265 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:30.957242 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-5f5c58667c-29wnm" Nov 02 06:16:31.064265 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:31.064206 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-86c478bb88-fh4sz" podStartSLOduration=246.111723419 podStartE2EDuration="4m8.064190927s" totalImagesPullingTime="1.952467508s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:12:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:16:31.063992744 +0000 UTC m=+52.955811007" watchObservedRunningTime="2026-11-02 06:16:31.064190927 +0000 UTC m=+52.956009188" Nov 02 06:16:31.961375 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:31.961339 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gf726" event={"ID":"4ba9b2b6-4be9-4303-8eab-cd464c5a32c7","Type":"ContainerStarted","Data":"272e71ebe472f4b44fffd8e4975073e26e782b1fab98a79005df11dbf5b7d00e"} Nov 02 06:16:31.961830 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:31.961434 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gf726" event={"ID":"4ba9b2b6-4be9-4303-8eab-cd464c5a32c7","Type":"ContainerStarted","Data":"d1e11c96f1f5e61f3b4d31f5bb53205d73abb09c6e87d892aa68c95146abdb0c"} Nov 02 06:16:31.961830 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:31.961465 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-gf726" Nov 02 06:16:32.070796 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:32.070742 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-gf726" podStartSLOduration=17.265526757 podStartE2EDuration="20.07072824s" totalImagesPullingTime="2.805201483s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:16:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:16:32.069794356 +0000 UTC m=+53.961612618" watchObservedRunningTime="2026-11-02 06:16:32.07072824 +0000 UTC m=+53.962546500" Nov 02 06:16:32.548042 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:32.548001 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-csfhg" Nov 02 06:16:32.548424 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:32.548406 2579 scope.go:122] "RemoveContainer" containerID="8bd3233465a5560a8e5e0dc6a2c9a27ed2a1f235d342c92357cf3a9a272a46f8" Nov 02 06:16:32.548647 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:16:32.548624 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-csfhg_openshift-console-operator(17dee72b-2f47-4016-8ffe-3a89934e66ee)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-csfhg" podUID="17dee72b-2f47-4016-8ffe-3a89934e66ee" Nov 02 06:16:32.549439 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:32.549416 2579 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-csfhg" Nov 02 06:16:32.965109 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:32.965077 2579 scope.go:122] "RemoveContainer" containerID="8bd3233465a5560a8e5e0dc6a2c9a27ed2a1f235d342c92357cf3a9a272a46f8" Nov 02 06:16:32.965611 ip-10-0-130-93 kubenswrapper[2579]: E1102 06:16:32.965287 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-csfhg_openshift-console-operator(17dee72b-2f47-4016-8ffe-3a89934e66ee)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-csfhg" podUID="17dee72b-2f47-4016-8ffe-3a89934e66ee" Nov 02 06:16:35.624032 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:35.623938 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8b491a46-9f53-4a4f-97ed-d8b32e3a03fa-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p98cq\" (UID: \"8b491a46-9f53-4a4f-97ed-d8b32e3a03fa\") " pod="openshift-insights/insights-runtime-extractor-p98cq" Nov 02 06:16:35.626300 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:35.626274 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8b491a46-9f53-4a4f-97ed-d8b32e3a03fa-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p98cq\" (UID: \"8b491a46-9f53-4a4f-97ed-d8b32e3a03fa\") " pod="openshift-insights/insights-runtime-extractor-p98cq" Nov 02 06:16:35.676720 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:35.676691 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-p98cq" Nov 02 06:16:36.043987 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:36.043949 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-p98cq"] Nov 02 06:16:36.049769 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:16:36.049741 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b491a46_9f53_4a4f_97ed_d8b32e3a03fa.slice/crio-fdbc98481353628068c730cc914f394bf2248182b7dc1d598e90142adad72ecf WatchSource:0}: Error finding container fdbc98481353628068c730cc914f394bf2248182b7dc1d598e90142adad72ecf: Status 404 returned error can't find the container with id fdbc98481353628068c730cc914f394bf2248182b7dc1d598e90142adad72ecf Nov 02 06:16:36.815868 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:36.815842 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7wb4x" Nov 02 06:16:36.980340 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:36.980309 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-p98cq" event={"ID":"8b491a46-9f53-4a4f-97ed-d8b32e3a03fa","Type":"ContainerStarted","Data":"46f4f4d5b5c5e112c194f38c4ac5ffcd592fb31bb43697b0365bed3be9bf68c2"} Nov 02 06:16:36.980510 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:36.980362 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-p98cq" event={"ID":"8b491a46-9f53-4a4f-97ed-d8b32e3a03fa","Type":"ContainerStarted","Data":"fdbc98481353628068c730cc914f394bf2248182b7dc1d598e90142adad72ecf"} Nov 02 06:16:37.985848 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:37.985812 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-p98cq" event={"ID":"8b491a46-9f53-4a4f-97ed-d8b32e3a03fa","Type":"ContainerStarted","Data":"e1632f72b5faaa05709332d3e8a737145f3b477822a7486f0a25fb5e33185ab6"} Nov 02 06:16:41.001177 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:41.001134 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-p98cq" event={"ID":"8b491a46-9f53-4a4f-97ed-d8b32e3a03fa","Type":"ContainerStarted","Data":"8866635865bdd30e384844da1bc8f93a3558f4406a6839b987bdf5dec772e45a"} Nov 02 06:16:41.095136 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:41.095105 2579 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6689c6d46b-vw558"] Nov 02 06:16:41.102548 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:41.102520 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-mddfd"] Nov 02 06:16:41.105733 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:41.105712 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-mddfd" Nov 02 06:16:41.161680 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:41.161650 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Nov 02 06:16:41.165799 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:41.165753 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-p98cq" podStartSLOduration=18.200655599 podStartE2EDuration="22.165739845s" totalImagesPullingTime="3.965084246s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 06:16:19 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:16:41.164724966 +0000 UTC m=+63.056543229" watchObservedRunningTime="2026-11-02 06:16:41.165739845 +0000 UTC m=+63.057558106" Nov 02 06:16:41.169600 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:41.169572 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-mddfd"] Nov 02 06:16:41.174357 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:41.174327 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7jwb\" (UniqueName: \"kubernetes.io/projected/49f1bd67-26ad-4fc9-b3d5-7bf7f6404aaa-kube-api-access-v7jwb\") pod \"migrator-66df5d496b-mddfd\" (UID: \"49f1bd67-26ad-4fc9-b3d5-7bf7f6404aaa\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-mddfd" Nov 02 06:16:41.178146 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:41.178127 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Nov 02 06:16:41.227889 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:41.227862 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-kqjnz\"" Nov 02 06:16:41.275850 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:41.275788 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7jwb\" (UniqueName: \"kubernetes.io/projected/49f1bd67-26ad-4fc9-b3d5-7bf7f6404aaa-kube-api-access-v7jwb\") pod \"migrator-66df5d496b-mddfd\" (UID: \"49f1bd67-26ad-4fc9-b3d5-7bf7f6404aaa\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-mddfd" Nov 02 06:16:41.332814 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:41.332789 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7jwb\" (UniqueName: \"kubernetes.io/projected/49f1bd67-26ad-4fc9-b3d5-7bf7f6404aaa-kube-api-access-v7jwb\") pod \"migrator-66df5d496b-mddfd\" (UID: \"49f1bd67-26ad-4fc9-b3d5-7bf7f6404aaa\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-mddfd" Nov 02 06:16:41.417256 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:41.417225 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-mddfd" Nov 02 06:16:41.782991 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:41.782957 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-mddfd"] Nov 02 06:16:41.788744 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:16:41.788710 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49f1bd67_26ad_4fc9_b3d5_7bf7f6404aaa.slice/crio-418136c5b736666c736e9cbf8be032be97dc9918c1dd664aae798cd2baf3649b WatchSource:0}: Error finding container 418136c5b736666c736e9cbf8be032be97dc9918c1dd664aae798cd2baf3649b: Status 404 returned error can't find the container with id 418136c5b736666c736e9cbf8be032be97dc9918c1dd664aae798cd2baf3649b Nov 02 06:16:41.967094 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:41.967038 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-gf726" Nov 02 06:16:42.005849 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:42.005819 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-mddfd" event={"ID":"49f1bd67-26ad-4fc9-b3d5-7bf7f6404aaa","Type":"ContainerStarted","Data":"418136c5b736666c736e9cbf8be032be97dc9918c1dd664aae798cd2baf3649b"} Nov 02 06:16:43.597824 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:43.597788 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b6432a8-b0ce-4954-8e3d-8eb8782557eb-metrics-certs\") pod \"network-metrics-daemon-sb4nt\" (UID: \"3b6432a8-b0ce-4954-8e3d-8eb8782557eb\") " pod="openshift-multus/network-metrics-daemon-sb4nt" Nov 02 06:16:43.598353 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:43.597961 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b65bb247-83f7-4749-9aa1-9b2677dd1b3d-original-pull-secret\") pod \"global-pull-secret-syncer-ggpsr\" (UID: \"b65bb247-83f7-4749-9aa1-9b2677dd1b3d\") " pod="kube-system/global-pull-secret-syncer-ggpsr" Nov 02 06:16:43.647312 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:43.647286 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 02 06:16:43.650349 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:43.650319 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b6432a8-b0ce-4954-8e3d-8eb8782557eb-metrics-certs\") pod \"network-metrics-daemon-sb4nt\" (UID: \"3b6432a8-b0ce-4954-8e3d-8eb8782557eb\") " pod="openshift-multus/network-metrics-daemon-sb4nt" Nov 02 06:16:43.695118 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:43.695098 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 02 06:16:43.698397 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:43.698375 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnq6k\" (UniqueName: \"kubernetes.io/projected/3ced47b0-9cd0-4710-8f75-2a333437ed01-kube-api-access-gnq6k\") pod \"network-check-target-ft7gs\" (UID: \"3ced47b0-9cd0-4710-8f75-2a333437ed01\") " pod="openshift-network-diagnostics/network-check-target-ft7gs" Nov 02 06:16:43.700350 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:43.700328 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b65bb247-83f7-4749-9aa1-9b2677dd1b3d-original-pull-secret\") pod \"global-pull-secret-syncer-ggpsr\" (UID: \"b65bb247-83f7-4749-9aa1-9b2677dd1b3d\") " pod="kube-system/global-pull-secret-syncer-ggpsr" Nov 02 06:16:43.700621 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:43.700602 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnq6k\" (UniqueName: \"kubernetes.io/projected/3ced47b0-9cd0-4710-8f75-2a333437ed01-kube-api-access-gnq6k\") pod \"network-check-target-ft7gs\" (UID: \"3ced47b0-9cd0-4710-8f75-2a333437ed01\") " pod="openshift-network-diagnostics/network-check-target-ft7gs" Nov 02 06:16:43.776760 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:43.776731 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-9bfjz\"" Nov 02 06:16:43.777734 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:43.777717 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ggpsr" Nov 02 06:16:43.797393 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:43.797373 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-cdk7k\"" Nov 02 06:16:43.798897 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:43.798871 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ft7gs" Nov 02 06:16:43.838174 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:43.836129 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-2646z\"" Nov 02 06:16:43.838584 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:43.838352 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sb4nt" Nov 02 06:16:44.014276 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:44.014246 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-mddfd" event={"ID":"49f1bd67-26ad-4fc9-b3d5-7bf7f6404aaa","Type":"ContainerStarted","Data":"e5c5d3dfa98c4ba4630e9ac7c5f5e620890af4a23892437f1a54e34c69b23c9b"} Nov 02 06:16:44.101948 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:44.101858 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3fcb7621-86c8-448f-b4fa-b5bd76be7356-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-2rs6c\" (UID: \"3fcb7621-86c8-448f-b4fa-b5bd76be7356\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2rs6c" Nov 02 06:16:44.104246 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:44.104218 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3fcb7621-86c8-448f-b4fa-b5bd76be7356-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-2rs6c\" (UID: \"3fcb7621-86c8-448f-b4fa-b5bd76be7356\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2rs6c" Nov 02 06:16:44.122816 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:44.122772 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-mddfd" podStartSLOduration=1.240768545 podStartE2EDuration="3.122759324s" totalImagesPullingTime="1.881990779s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:16:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:16:44.121607039 +0000 UTC m=+66.013425300" watchObservedRunningTime="2026-11-02 06:16:44.122759324 +0000 UTC m=+66.014577585" Nov 02 06:16:44.145678 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:44.145653 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-ggpsr"] Nov 02 06:16:44.178544 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:16:44.178514 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb65bb247_83f7_4749_9aa1_9b2677dd1b3d.slice/crio-05a094dc530991421555eea671fb2cf05126ac45b8261be9db29c7b4f57ad03b WatchSource:0}: Error finding container 05a094dc530991421555eea671fb2cf05126ac45b8261be9db29c7b4f57ad03b: Status 404 returned error can't find the container with id 05a094dc530991421555eea671fb2cf05126ac45b8261be9db29c7b4f57ad03b Nov 02 06:16:44.196524 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:44.196498 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-ft7gs"] Nov 02 06:16:44.201976 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:16:44.201951 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ced47b0_9cd0_4710_8f75_2a333437ed01.slice/crio-5165263c016e50839257e4586fc7f9fc1b015d1999d655cc835b1d5e978df321 WatchSource:0}: Error finding container 5165263c016e50839257e4586fc7f9fc1b015d1999d655cc835b1d5e978df321: Status 404 returned error can't find the container with id 5165263c016e50839257e4586fc7f9fc1b015d1999d655cc835b1d5e978df321 Nov 02 06:16:44.203006 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:44.202984 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/cf8e8e65-8525-484b-adfe-f56517ccdc1b-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-d6bc4\" (UID: \"cf8e8e65-8525-484b-adfe-f56517ccdc1b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-d6bc4" Nov 02 06:16:44.205541 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:44.205521 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/cf8e8e65-8525-484b-adfe-f56517ccdc1b-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-d6bc4\" (UID: \"cf8e8e65-8525-484b-adfe-f56517ccdc1b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-d6bc4" Nov 02 06:16:44.246326 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:44.246304 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-sb4nt"] Nov 02 06:16:44.251767 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:16:44.251739 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b6432a8_b0ce_4954_8e3d_8eb8782557eb.slice/crio-fadf7f58c12e185657a35443e5641a2c59dc461cb89e0db18ce1be9b812892b7 WatchSource:0}: Error finding container fadf7f58c12e185657a35443e5641a2c59dc461cb89e0db18ce1be9b812892b7: Status 404 returned error can't find the container with id fadf7f58c12e185657a35443e5641a2c59dc461cb89e0db18ce1be9b812892b7 Nov 02 06:16:44.321231 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:44.321198 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2rs6c" Nov 02 06:16:44.360644 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:44.360578 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-d6bc4" Nov 02 06:16:44.685406 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:44.685378 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-2rs6c"] Nov 02 06:16:44.690804 ip-10-0-130-93 kubenswrapper[2579]: W1102 06:16:44.690777 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fcb7621_86c8_448f_b4fa_b5bd76be7356.slice/crio-60673025a4b89a0ee5f4cb8c3345979e787c7da063a0e2faa54ef8740d8f0781 WatchSource:0}: Error finding container 60673025a4b89a0ee5f4cb8c3345979e787c7da063a0e2faa54ef8740d8f0781: Status 404 returned error can't find the container with id 60673025a4b89a0ee5f4cb8c3345979e787c7da063a0e2faa54ef8740d8f0781 Nov 02 06:16:44.737262 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:44.737233 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-d6bc4"] Nov 02 06:16:45.018856 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:45.018762 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-sb4nt" event={"ID":"3b6432a8-b0ce-4954-8e3d-8eb8782557eb","Type":"ContainerStarted","Data":"fadf7f58c12e185657a35443e5641a2c59dc461cb89e0db18ce1be9b812892b7"} Nov 02 06:16:45.019879 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:45.019856 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2rs6c" event={"ID":"3fcb7621-86c8-448f-b4fa-b5bd76be7356","Type":"ContainerStarted","Data":"60673025a4b89a0ee5f4cb8c3345979e787c7da063a0e2faa54ef8740d8f0781"} Nov 02 06:16:45.020988 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:45.020968 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-d6bc4" event={"ID":"cf8e8e65-8525-484b-adfe-f56517ccdc1b","Type":"ContainerStarted","Data":"7e7bb32b8719598da92f32821b935863ee8427e701ef1c413fe40777eed58690"} Nov 02 06:16:45.021980 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:45.021961 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-ggpsr" event={"ID":"b65bb247-83f7-4749-9aa1-9b2677dd1b3d","Type":"ContainerStarted","Data":"05a094dc530991421555eea671fb2cf05126ac45b8261be9db29c7b4f57ad03b"} Nov 02 06:16:45.023367 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:45.023347 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-ft7gs" event={"ID":"3ced47b0-9cd0-4710-8f75-2a333437ed01","Type":"ContainerStarted","Data":"526d333459e99f4f2e229944023722ede77e76e782a6ccc4cac98d9da39914ed"} Nov 02 06:16:45.023451 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:45.023381 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-ft7gs" event={"ID":"3ced47b0-9cd0-4710-8f75-2a333437ed01","Type":"ContainerStarted","Data":"5165263c016e50839257e4586fc7f9fc1b015d1999d655cc835b1d5e978df321"} Nov 02 06:16:45.133783 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:45.133740 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-ft7gs" podStartSLOduration=67.133727548 podStartE2EDuration="1m7.133727548s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:15:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:16:45.13247522 +0000 UTC m=+67.024293482" watchObservedRunningTime="2026-11-02 06:16:45.133727548 +0000 UTC m=+67.025545809" Nov 02 06:16:46.032501 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:46.032462 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-ft7gs" Nov 02 06:16:47.038283 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:47.038239 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-sb4nt" event={"ID":"3b6432a8-b0ce-4954-8e3d-8eb8782557eb","Type":"ContainerStarted","Data":"132e83ce971ee1a740e70e7b784e97f2859f015ce29d718c7f04ea2fb59b8546"} Nov 02 06:16:47.038745 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:47.038310 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-sb4nt" event={"ID":"3b6432a8-b0ce-4954-8e3d-8eb8782557eb","Type":"ContainerStarted","Data":"a9293a4420ed3f14360990f18fc2448a0c2be0730bdc2bb2274fb86ee7ce2377"} Nov 02 06:16:47.148097 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:47.148050 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-sb4nt" podStartSLOduration=67.423181032 podStartE2EDuration="1m9.148035793s" totalImagesPullingTime="1.724854761s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:15:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:16:47.146945062 +0000 UTC m=+69.038763324" watchObservedRunningTime="2026-11-02 06:16:47.148035793 +0000 UTC m=+69.039854054" Nov 02 06:16:48.044064 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:48.044038 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2rs6c" event={"ID":"3fcb7621-86c8-448f-b4fa-b5bd76be7356","Type":"ContainerStarted","Data":"aaf35b65b80d74d38bfd1ac50e5747673eb41fddcc9bca5629646e8d98c6f5bf"} Nov 02 06:16:48.045866 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:48.045841 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-d6bc4" event={"ID":"cf8e8e65-8525-484b-adfe-f56517ccdc1b","Type":"ContainerStarted","Data":"48ea947d520d5c2536800b0594c38c4851bf18900390203e0534ec01ace5a9b5"} Nov 02 06:16:48.103443 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:48.103366 2579 patch_prober.go:29] interesting pod/image-registry-746899d75-dtjhs container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Nov 02 06:16:48.103443 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:48.103416 2579 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-746899d75-dtjhs" podUID="fa9c5a91-d480-4d7b-9044-be6437a76a02" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 06:16:48.159874 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:48.159824 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2rs6c" podStartSLOduration=267.012661035 podStartE2EDuration="4m30.159808773s" totalImagesPullingTime="3.147147738s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:12:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:16:48.158683276 +0000 UTC m=+70.050501541" watchObservedRunningTime="2026-11-02 06:16:48.159808773 +0000 UTC m=+70.051627034" Nov 02 06:16:49.053678 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:49.053626 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-d6bc4" event={"ID":"cf8e8e65-8525-484b-adfe-f56517ccdc1b","Type":"ContainerStarted","Data":"98da6247e8854d94dd6221496368dfdb0f95615e79a3ee9e37ab8cc0aceb3a6c"} Nov 02 06:16:49.165535 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:49.165469 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-d6bc4" podStartSLOduration=268.039104103 podStartE2EDuration="4m31.165450565s" totalImagesPullingTime="3.126346462s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:12:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:16:49.164672888 +0000 UTC m=+71.056491151" watchObservedRunningTime="2026-11-02 06:16:49.165450565 +0000 UTC m=+71.057268827" Nov 02 06:16:49.954103 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:49.954068 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-746899d75-dtjhs" Nov 02 06:16:50.648627 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:50.648592 2579 scope.go:122] "RemoveContainer" containerID="8bd3233465a5560a8e5e0dc6a2c9a27ed2a1f235d342c92357cf3a9a272a46f8" Nov 02 06:16:51.064606 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:51.064532 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-csfhg_17dee72b-2f47-4016-8ffe-3a89934e66ee/console-operator/2.log" Nov 02 06:16:51.064767 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:51.064638 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-csfhg" event={"ID":"17dee72b-2f47-4016-8ffe-3a89934e66ee","Type":"ContainerStarted","Data":"61188d47b2710872109c8f7e295f975308d7592b7ac4df5693715824d8514f44"} Nov 02 06:16:51.064982 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:51.064960 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-csfhg" Nov 02 06:16:51.101435 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:51.101406 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6689c6d46b-vw558" Nov 02 06:16:51.175472 ip-10-0-130-93 kubenswrapper[2579]: I1102 06:16:51.175419 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console-operator/console-operator-7cd56b8dbc-csfhg" podStartSLOduration=267.620224635 podStartE2EDuration="4m33.175406965s" totalImagesPullingTime="5.55518233s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:12:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:16:51.174212141 +0000 UTC m=+73.066030399" watchObservedRunningTime="2026-11-02 06:16:51.175406965 +0000 UTC m=+73.067225260" Nov 02 06:16:51.667613 ip-10-0-130-93 systemd[1]: Stopping Kubernetes Kubelet... Nov 02 06:16:51.687280 ip-10-0-130-93 systemd[1]: kubelet.service: Deactivated successfully. Nov 02 06:16:51.687450 ip-10-0-130-93 systemd[1]: Stopped Kubernetes Kubelet. Nov 02 06:16:51.687622 ip-10-0-130-93 systemd[1]: kubelet.service: Consumed 4.657s CPU time. Nov 02 06:16:51.696545 ip-10-0-130-93 systemd[1]: Starting Kubernetes Kubelet... Nov 02 06:16:51.757618 ip-10-0-130-93 kubenswrapper[5916]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 06:16:51.757618 ip-10-0-130-93 kubenswrapper[5916]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 02 06:16:51.757618 ip-10-0-130-93 kubenswrapper[5916]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 06:16:51.757618 ip-10-0-130-93 kubenswrapper[5916]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 06:16:51.759770 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.759684 5916 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 06:16:51.759817 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.759779 5916 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 06:16:51.759817 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.759783 5916 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 06:16:51.759817 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.759787 5916 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 06:16:51.759817 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.759790 5916 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 06:16:51.759817 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.759793 5916 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 06:16:51.759817 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.759796 5916 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 06:16:51.759817 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.759799 5916 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 06:16:51.759817 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.759801 5916 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 06:16:51.759817 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.759804 5916 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 06:16:51.759817 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.759808 5916 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 06:16:51.759817 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.759812 5916 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 06:16:51.759817 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.759815 5916 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 06:16:51.759817 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.759819 5916 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 06:16:51.759817 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.759822 5916 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 06:16:51.759817 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.759825 5916 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 06:16:51.760272 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.759828 5916 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 06:16:51.760272 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.759830 5916 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 06:16:51.760272 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.759833 5916 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 06:16:51.760272 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.759836 5916 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 06:16:51.760272 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.759843 5916 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 06:16:51.760272 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.759846 5916 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 06:16:51.760272 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.759849 5916 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 06:16:51.760272 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.759852 5916 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 06:16:51.760272 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.759854 5916 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 06:16:51.760272 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.759857 5916 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 06:16:51.760272 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.759859 5916 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 06:16:51.760272 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.759862 5916 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 06:16:51.760272 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.759865 5916 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 06:16:51.760272 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.759868 5916 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 06:16:51.760272 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.759870 5916 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 06:16:51.760272 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.759873 5916 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 06:16:51.760272 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.759875 5916 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 06:16:51.760272 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.759878 5916 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 06:16:51.760272 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.759881 5916 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 06:16:51.760745 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.759884 5916 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 06:16:51.760745 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.759886 5916 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 06:16:51.760745 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.759889 5916 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 06:16:51.760745 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.759891 5916 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 06:16:51.760745 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.759894 5916 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 06:16:51.760745 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.759896 5916 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 06:16:51.760745 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.759899 5916 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 06:16:51.760745 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.759916 5916 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 06:16:51.760745 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.759920 5916 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 06:16:51.760745 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.759923 5916 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 06:16:51.760745 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.759926 5916 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 06:16:51.760745 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.759928 5916 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 06:16:51.760745 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.759932 5916 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 06:16:51.760745 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.759935 5916 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 06:16:51.760745 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.759938 5916 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 06:16:51.760745 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.759941 5916 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 06:16:51.760745 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.759944 5916 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 06:16:51.760745 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.759953 5916 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 06:16:51.760745 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.759956 5916 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 06:16:51.760745 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.759959 5916 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 06:16:51.761278 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.759962 5916 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 06:16:51.761278 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.759964 5916 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 06:16:51.761278 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.759967 5916 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 06:16:51.761278 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.759970 5916 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 06:16:51.761278 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.759973 5916 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 06:16:51.761278 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.759975 5916 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 06:16:51.761278 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.759978 5916 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 06:16:51.761278 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.759981 5916 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 06:16:51.761278 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.759983 5916 feature_gate.go:386] unrecognized feature gate: Example Nov 02 06:16:51.761278 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.759986 5916 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 06:16:51.761278 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.759988 5916 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 06:16:51.761278 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.759991 5916 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 06:16:51.761278 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.759994 5916 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 06:16:51.761278 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.759996 5916 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 06:16:51.761278 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.759999 5916 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 06:16:51.761278 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.760002 5916 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 06:16:51.761278 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.760004 5916 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 06:16:51.761278 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.760007 5916 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 06:16:51.761278 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.760009 5916 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 06:16:51.761278 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.760012 5916 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 06:16:51.761813 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.760015 5916 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 06:16:51.761813 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.760018 5916 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 06:16:51.761813 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.760020 5916 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 06:16:51.761813 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.760023 5916 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 06:16:51.761813 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.760026 5916 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 06:16:51.761813 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.760029 5916 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 06:16:51.761813 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.760032 5916 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 06:16:51.761813 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.760034 5916 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 06:16:51.761813 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.760037 5916 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 06:16:51.761813 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.760039 5916 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 06:16:51.761813 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.760048 5916 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 06:16:51.761813 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.760051 5916 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 06:16:51.761813 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.760054 5916 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 06:16:51.761813 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.760056 5916 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 06:16:51.761813 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.760059 5916 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 06:16:51.761813 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.760061 5916 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 06:16:51.761813 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.760064 5916 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 06:16:51.761813 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.760067 5916 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 06:16:51.761813 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.760069 5916 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 06:16:51.761813 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.760071 5916 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 06:16:51.762342 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.760074 5916 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 06:16:51.762342 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.760076 5916 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 06:16:51.762342 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.760079 5916 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 06:16:51.762342 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.760082 5916 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 06:16:51.762342 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.760087 5916 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 06:16:51.762342 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.760091 5916 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 06:16:51.762342 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.760095 5916 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 06:16:51.762342 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.760098 5916 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 06:16:51.762342 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.760101 5916 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 06:16:51.762342 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.760104 5916 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 06:16:51.762342 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.760106 5916 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 06:16:51.762342 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.760917 5916 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 06:16:51.762342 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.760926 5916 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 06:16:51.762342 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.760930 5916 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 06:16:51.762342 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.760935 5916 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 06:16:51.762342 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.760939 5916 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 06:16:51.762342 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.760942 5916 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 06:16:51.762342 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.760945 5916 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 06:16:51.762342 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.760948 5916 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 06:16:51.762846 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.760951 5916 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 06:16:51.762846 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.760954 5916 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 06:16:51.762846 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.760957 5916 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 06:16:51.762846 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.760960 5916 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 06:16:51.762846 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.760964 5916 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 06:16:51.762846 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.760967 5916 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 06:16:51.762846 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.760970 5916 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 06:16:51.762846 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.760973 5916 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 06:16:51.762846 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.760976 5916 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 06:16:51.762846 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.760978 5916 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 06:16:51.762846 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.760983 5916 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 06:16:51.762846 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.760987 5916 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 06:16:51.762846 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.760990 5916 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 06:16:51.762846 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.760993 5916 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 06:16:51.762846 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.760996 5916 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 06:16:51.762846 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.760999 5916 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 06:16:51.762846 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761001 5916 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 06:16:51.762846 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761004 5916 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 06:16:51.762846 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761006 5916 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 06:16:51.763345 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761009 5916 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 06:16:51.763345 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761012 5916 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 06:16:51.763345 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761014 5916 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 06:16:51.763345 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761018 5916 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 06:16:51.763345 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761021 5916 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 06:16:51.763345 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761023 5916 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 06:16:51.763345 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761026 5916 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 06:16:51.763345 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761029 5916 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 06:16:51.763345 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761031 5916 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 06:16:51.763345 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761033 5916 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 06:16:51.763345 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761037 5916 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 06:16:51.763345 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761040 5916 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 06:16:51.763345 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761043 5916 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 06:16:51.763345 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761045 5916 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 06:16:51.763345 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761048 5916 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 06:16:51.763345 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761050 5916 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 06:16:51.763345 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761053 5916 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 06:16:51.763345 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761055 5916 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 06:16:51.763345 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761059 5916 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 06:16:51.763345 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761062 5916 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 06:16:51.763849 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761064 5916 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 06:16:51.763849 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761067 5916 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 06:16:51.763849 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761069 5916 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 06:16:51.763849 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761072 5916 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 06:16:51.763849 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761074 5916 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 06:16:51.763849 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761077 5916 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 06:16:51.763849 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761079 5916 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 06:16:51.763849 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761082 5916 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 06:16:51.763849 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761085 5916 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 06:16:51.763849 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761087 5916 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 06:16:51.763849 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761090 5916 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 06:16:51.763849 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761092 5916 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 06:16:51.763849 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761095 5916 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 06:16:51.763849 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761098 5916 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 06:16:51.763849 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761100 5916 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 06:16:51.763849 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761103 5916 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 06:16:51.763849 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761106 5916 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 06:16:51.763849 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761108 5916 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 06:16:51.763849 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761111 5916 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 06:16:51.764348 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761114 5916 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 06:16:51.764348 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761116 5916 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 06:16:51.764348 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761119 5916 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 06:16:51.764348 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761122 5916 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 06:16:51.764348 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761125 5916 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 06:16:51.764348 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761127 5916 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 06:16:51.764348 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761130 5916 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 06:16:51.764348 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761133 5916 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 06:16:51.764348 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761136 5916 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 06:16:51.764348 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761139 5916 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 06:16:51.764348 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761141 5916 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 06:16:51.764348 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761144 5916 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 06:16:51.764348 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761147 5916 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 06:16:51.764348 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761150 5916 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 06:16:51.764348 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761152 5916 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 06:16:51.764348 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761155 5916 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 06:16:51.764348 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761157 5916 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 06:16:51.764348 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761160 5916 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 06:16:51.764348 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761162 5916 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 06:16:51.764348 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761165 5916 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 06:16:51.764857 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761168 5916 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 06:16:51.764857 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761171 5916 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 06:16:51.764857 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761173 5916 feature_gate.go:386] unrecognized feature gate: Example Nov 02 06:16:51.764857 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761176 5916 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 06:16:51.764857 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761179 5916 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 06:16:51.764857 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761181 5916 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 06:16:51.764857 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761184 5916 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 06:16:51.764857 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761187 5916 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 06:16:51.764857 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761189 5916 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 06:16:51.764857 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761192 5916 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 06:16:51.764857 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761194 5916 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 06:16:51.764857 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761197 5916 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 06:16:51.764857 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761199 5916 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 06:16:51.764857 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761202 5916 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 06:16:51.764857 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761204 5916 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 06:16:51.764857 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761207 5916 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 06:16:51.764857 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761215 5916 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 06:16:51.764857 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761219 5916 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 06:16:51.764857 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761222 5916 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 06:16:51.765334 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761225 5916 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 06:16:51.765334 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761713 5916 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 06:16:51.765334 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761719 5916 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 06:16:51.765334 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761722 5916 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 06:16:51.765334 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761725 5916 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 06:16:51.765334 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761730 5916 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 06:16:51.765334 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761733 5916 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 06:16:51.765334 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761736 5916 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 06:16:51.765334 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761739 5916 feature_gate.go:386] unrecognized feature gate: Example Nov 02 06:16:51.765334 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761742 5916 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 06:16:51.765334 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761745 5916 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 06:16:51.765334 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761747 5916 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 06:16:51.765334 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761750 5916 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 06:16:51.765334 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761753 5916 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 06:16:51.765334 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761756 5916 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 06:16:51.765334 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761758 5916 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 06:16:51.765334 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761761 5916 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 06:16:51.765334 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761764 5916 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 06:16:51.765334 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761767 5916 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 06:16:51.765334 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761770 5916 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 06:16:51.765828 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761772 5916 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 06:16:51.765828 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761775 5916 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 06:16:51.765828 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761778 5916 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 06:16:51.765828 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761780 5916 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 06:16:51.765828 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761783 5916 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 06:16:51.765828 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761786 5916 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 06:16:51.765828 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761788 5916 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 06:16:51.765828 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761791 5916 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 06:16:51.765828 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761794 5916 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 06:16:51.765828 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761797 5916 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 06:16:51.765828 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761799 5916 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 06:16:51.765828 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761802 5916 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 06:16:51.765828 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761804 5916 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 06:16:51.765828 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761807 5916 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 06:16:51.765828 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761809 5916 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 06:16:51.765828 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761812 5916 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 06:16:51.765828 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761814 5916 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 06:16:51.765828 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761817 5916 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 06:16:51.765828 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761820 5916 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 06:16:51.765828 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761823 5916 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 06:16:51.766650 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761827 5916 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 06:16:51.766650 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761831 5916 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 06:16:51.766650 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761835 5916 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 06:16:51.766650 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761837 5916 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 06:16:51.766650 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761840 5916 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 06:16:51.766650 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761842 5916 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 06:16:51.766650 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761845 5916 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 06:16:51.766650 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761847 5916 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 06:16:51.766650 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761853 5916 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 06:16:51.766650 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761856 5916 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 06:16:51.766650 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761858 5916 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 06:16:51.766650 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761861 5916 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 06:16:51.766650 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761864 5916 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 06:16:51.766650 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761866 5916 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 06:16:51.766650 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761869 5916 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 06:16:51.766650 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761871 5916 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 06:16:51.766650 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761874 5916 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 06:16:51.766650 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761877 5916 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 06:16:51.766650 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761880 5916 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 06:16:51.767224 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761882 5916 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 06:16:51.767224 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761885 5916 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 06:16:51.767224 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761887 5916 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 06:16:51.767224 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761890 5916 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 06:16:51.767224 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761892 5916 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 06:16:51.767224 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761895 5916 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 06:16:51.767224 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761897 5916 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 06:16:51.767224 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761900 5916 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 06:16:51.767224 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761918 5916 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 06:16:51.767224 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761922 5916 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 06:16:51.767224 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761926 5916 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 06:16:51.767224 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761929 5916 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 06:16:51.767224 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761933 5916 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 06:16:51.767224 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761935 5916 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 06:16:51.767224 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761938 5916 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 06:16:51.767224 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761941 5916 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 06:16:51.767224 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761943 5916 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 06:16:51.767224 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761946 5916 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 06:16:51.767224 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761948 5916 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 06:16:51.767224 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761951 5916 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 06:16:51.767896 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761956 5916 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 06:16:51.767896 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761959 5916 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 06:16:51.767896 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761962 5916 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 06:16:51.767896 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761965 5916 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 06:16:51.767896 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761968 5916 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 06:16:51.767896 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761970 5916 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 06:16:51.767896 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761973 5916 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 06:16:51.767896 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761976 5916 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 06:16:51.767896 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761980 5916 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 06:16:51.767896 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761983 5916 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 06:16:51.767896 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761986 5916 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 06:16:51.767896 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761989 5916 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 06:16:51.767896 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761992 5916 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 06:16:51.767896 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761995 5916 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 06:16:51.767896 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.761997 5916 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 06:16:51.767896 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.762000 5916 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 06:16:51.767896 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.762002 5916 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 06:16:51.767896 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.762005 5916 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 06:16:51.767896 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.762007 5916 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 06:16:51.768389 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.762010 5916 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 06:16:51.768389 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.762013 5916 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 06:16:51.768389 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.762015 5916 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 06:16:51.768389 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.762018 5916 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 06:16:51.768389 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.762020 5916 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 06:16:51.768389 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.762022 5916 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 06:16:51.768389 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.762026 5916 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 06:16:51.768389 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.762028 5916 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 06:16:51.768389 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.762031 5916 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 06:16:51.768389 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.762139 5916 flags.go:64] FLAG: --address="0.0.0.0" Nov 02 06:16:51.768389 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.762157 5916 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 02 06:16:51.768389 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.762165 5916 flags.go:64] FLAG: --anonymous-auth="true" Nov 02 06:16:51.768389 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.762169 5916 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 02 06:16:51.768389 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.762176 5916 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 02 06:16:51.768389 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.762181 5916 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 02 06:16:51.768389 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.762185 5916 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 02 06:16:51.768389 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.762189 5916 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 02 06:16:51.768389 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.762193 5916 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 02 06:16:51.768389 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.762196 5916 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 02 06:16:51.768389 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.762199 5916 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 02 06:16:51.768389 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.762203 5916 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 02 06:16:51.768921 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.762206 5916 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 02 06:16:51.768921 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.762209 5916 flags.go:64] FLAG: --cgroup-root="" Nov 02 06:16:51.768921 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.762213 5916 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 02 06:16:51.768921 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.762216 5916 flags.go:64] FLAG: --client-ca-file="" Nov 02 06:16:51.768921 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.762219 5916 flags.go:64] FLAG: --cloud-provider="external" Nov 02 06:16:51.768921 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.762222 5916 flags.go:64] FLAG: --cluster-dns="[]" Nov 02 06:16:51.768921 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.762226 5916 flags.go:64] FLAG: --cluster-domain="" Nov 02 06:16:51.768921 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.762229 5916 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 02 06:16:51.768921 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.762232 5916 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 02 06:16:51.768921 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.762235 5916 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 02 06:16:51.768921 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.762238 5916 flags.go:64] FLAG: --container-log-max-files="5" Nov 02 06:16:51.768921 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.762242 5916 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 02 06:16:51.768921 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.762245 5916 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 02 06:16:51.768921 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.762248 5916 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 02 06:16:51.768921 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.762252 5916 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 02 06:16:51.768921 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.762255 5916 flags.go:64] FLAG: --contention-profiling="false" Nov 02 06:16:51.768921 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.762258 5916 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 02 06:16:51.768921 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.762261 5916 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 02 06:16:51.768921 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.762265 5916 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 02 06:16:51.768921 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.762268 5916 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 02 06:16:51.768921 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.762272 5916 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 02 06:16:51.768921 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.762275 5916 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 02 06:16:51.768921 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.762278 5916 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 02 06:16:51.768921 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.762281 5916 flags.go:64] FLAG: --enable-load-reader="false" Nov 02 06:16:51.768921 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.762285 5916 flags.go:64] FLAG: --enable-server="true" Nov 02 06:16:51.769538 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.762290 5916 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 02 06:16:51.769538 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.762295 5916 flags.go:64] FLAG: --event-burst="100" Nov 02 06:16:51.769538 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.762298 5916 flags.go:64] FLAG: --event-qps="50" Nov 02 06:16:51.769538 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.762301 5916 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 02 06:16:51.769538 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.762305 5916 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 02 06:16:51.769538 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.762308 5916 flags.go:64] FLAG: --eviction-hard="" Nov 02 06:16:51.769538 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.762312 5916 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 02 06:16:51.769538 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.762316 5916 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 02 06:16:51.769538 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.762319 5916 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 02 06:16:51.769538 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.762322 5916 flags.go:64] FLAG: --eviction-soft="" Nov 02 06:16:51.769538 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.762325 5916 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 02 06:16:51.769538 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.762328 5916 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 02 06:16:51.769538 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.762331 5916 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 02 06:16:51.769538 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.762334 5916 flags.go:64] FLAG: --experimental-mounter-path="" Nov 02 06:16:51.769538 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.762338 5916 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 02 06:16:51.769538 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.762341 5916 flags.go:64] FLAG: --fail-swap-on="true" Nov 02 06:16:51.769538 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.762344 5916 flags.go:64] FLAG: --feature-gates="" Nov 02 06:16:51.769538 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.762348 5916 flags.go:64] FLAG: --file-check-frequency="20s" Nov 02 06:16:51.769538 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.762352 5916 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 02 06:16:51.769538 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.762355 5916 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 02 06:16:51.769538 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.762358 5916 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 02 06:16:51.769538 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.762361 5916 flags.go:64] FLAG: --healthz-port="10248" Nov 02 06:16:51.769538 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.762364 5916 flags.go:64] FLAG: --help="false" Nov 02 06:16:51.769538 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.762367 5916 flags.go:64] FLAG: --hostname-override="ip-10-0-130-93.us-west-2.compute.internal" Nov 02 06:16:51.769538 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.762371 5916 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 02 06:16:51.770172 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.762374 5916 flags.go:64] FLAG: --http-check-frequency="20s" Nov 02 06:16:51.770172 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.762378 5916 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 02 06:16:51.770172 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.762382 5916 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 02 06:16:51.770172 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.762385 5916 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 02 06:16:51.770172 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.762388 5916 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 02 06:16:51.770172 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.762391 5916 flags.go:64] FLAG: --image-service-endpoint="" Nov 02 06:16:51.770172 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.762395 5916 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 02 06:16:51.770172 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.762400 5916 flags.go:64] FLAG: --kube-api-burst="100" Nov 02 06:16:51.770172 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.762403 5916 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 02 06:16:51.770172 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.762406 5916 flags.go:64] FLAG: --kube-api-qps="50" Nov 02 06:16:51.770172 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.762409 5916 flags.go:64] FLAG: --kube-reserved="" Nov 02 06:16:51.770172 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.762412 5916 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 02 06:16:51.770172 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.762415 5916 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 02 06:16:51.770172 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.762418 5916 flags.go:64] FLAG: --kubelet-cgroups="" Nov 02 06:16:51.770172 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.762421 5916 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 02 06:16:51.770172 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.762424 5916 flags.go:64] FLAG: --lock-file="" Nov 02 06:16:51.770172 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.762426 5916 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 02 06:16:51.770172 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.762430 5916 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 02 06:16:51.770172 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.762433 5916 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 02 06:16:51.770172 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.762438 5916 flags.go:64] FLAG: --log-json-split-stream="false" Nov 02 06:16:51.770172 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.762441 5916 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 02 06:16:51.770172 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.762444 5916 flags.go:64] FLAG: --log-text-split-stream="false" Nov 02 06:16:51.770172 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.762447 5916 flags.go:64] FLAG: --logging-format="text" Nov 02 06:16:51.770757 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.762450 5916 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 02 06:16:51.770757 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.762454 5916 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 02 06:16:51.770757 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.762457 5916 flags.go:64] FLAG: --manifest-url="" Nov 02 06:16:51.770757 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.762459 5916 flags.go:64] FLAG: --manifest-url-header="" Nov 02 06:16:51.770757 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.762464 5916 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 02 06:16:51.770757 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.762467 5916 flags.go:64] FLAG: --max-open-files="1000000" Nov 02 06:16:51.770757 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.762471 5916 flags.go:64] FLAG: --max-pods="110" Nov 02 06:16:51.770757 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.762474 5916 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 02 06:16:51.770757 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.762477 5916 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 02 06:16:51.770757 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.762481 5916 flags.go:64] FLAG: --memory-manager-policy="None" Nov 02 06:16:51.770757 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.762483 5916 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 02 06:16:51.770757 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.762488 5916 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 02 06:16:51.770757 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.762491 5916 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 02 06:16:51.770757 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.762495 5916 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 02 06:16:51.770757 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.762501 5916 flags.go:64] FLAG: --node-status-max-images="50" Nov 02 06:16:51.770757 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.762506 5916 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 02 06:16:51.770757 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.762512 5916 flags.go:64] FLAG: --oom-score-adj="-999" Nov 02 06:16:51.770757 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.762515 5916 flags.go:64] FLAG: --pod-cidr="" Nov 02 06:16:51.770757 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.762518 5916 flags.go:64] FLAG: --pod-manifest-path="" Nov 02 06:16:51.770757 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.762521 5916 flags.go:64] FLAG: --pod-max-pids="-1" Nov 02 06:16:51.770757 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.762524 5916 flags.go:64] FLAG: --pods-per-core="0" Nov 02 06:16:51.770757 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.762527 5916 flags.go:64] FLAG: --port="10250" Nov 02 06:16:51.770757 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.762530 5916 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 02 06:16:51.770757 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.762534 5916 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-01095da248285f49f" Nov 02 06:16:51.770757 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.762537 5916 flags.go:64] FLAG: --qos-reserved="" Nov 02 06:16:51.771476 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.762540 5916 flags.go:64] FLAG: --read-only-port="10255" Nov 02 06:16:51.771476 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.762543 5916 flags.go:64] FLAG: --register-node="true" Nov 02 06:16:51.771476 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.762546 5916 flags.go:64] FLAG: --register-with-taints="" Nov 02 06:16:51.771476 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.762550 5916 flags.go:64] FLAG: --registry-burst="10" Nov 02 06:16:51.771476 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.762553 5916 flags.go:64] FLAG: --registry-qps="5" Nov 02 06:16:51.771476 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.762556 5916 flags.go:64] FLAG: --reserved-cpus="" Nov 02 06:16:51.771476 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.762559 5916 flags.go:64] FLAG: --reserved-memory="" Nov 02 06:16:51.771476 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.762563 5916 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 02 06:16:51.771476 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.762566 5916 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 02 06:16:51.771476 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.762569 5916 flags.go:64] FLAG: --rotate-certificates="false" Nov 02 06:16:51.771476 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.762572 5916 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 02 06:16:51.771476 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.762575 5916 flags.go:64] FLAG: --runonce="false" Nov 02 06:16:51.771476 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.762578 5916 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 02 06:16:51.771476 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.762582 5916 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 02 06:16:51.771476 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.762585 5916 flags.go:64] FLAG: --seccomp-default="false" Nov 02 06:16:51.771476 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.762588 5916 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 02 06:16:51.771476 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.762591 5916 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 02 06:16:51.771476 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.762594 5916 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 02 06:16:51.771476 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.762597 5916 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 02 06:16:51.771476 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.762601 5916 flags.go:64] FLAG: --storage-driver-password="root" Nov 02 06:16:51.771476 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.762604 5916 flags.go:64] FLAG: --storage-driver-secure="false" Nov 02 06:16:51.771476 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.762607 5916 flags.go:64] FLAG: --storage-driver-table="stats" Nov 02 06:16:51.771476 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.762611 5916 flags.go:64] FLAG: --storage-driver-user="root" Nov 02 06:16:51.771476 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.762615 5916 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 02 06:16:51.771476 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.762618 5916 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 02 06:16:51.771476 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.762621 5916 flags.go:64] FLAG: --system-cgroups="" Nov 02 06:16:51.772129 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.762624 5916 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 02 06:16:51.772129 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.762629 5916 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 02 06:16:51.772129 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.762632 5916 flags.go:64] FLAG: --tls-cert-file="" Nov 02 06:16:51.772129 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.762635 5916 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 02 06:16:51.772129 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.762639 5916 flags.go:64] FLAG: --tls-min-version="" Nov 02 06:16:51.772129 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.762648 5916 flags.go:64] FLAG: --tls-private-key-file="" Nov 02 06:16:51.772129 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.762652 5916 flags.go:64] FLAG: --topology-manager-policy="none" Nov 02 06:16:51.772129 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.762655 5916 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 02 06:16:51.772129 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.762658 5916 flags.go:64] FLAG: --topology-manager-scope="container" Nov 02 06:16:51.772129 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.762661 5916 flags.go:64] FLAG: --v="2" Nov 02 06:16:51.772129 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.762665 5916 flags.go:64] FLAG: --version="false" Nov 02 06:16:51.772129 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.762669 5916 flags.go:64] FLAG: --vmodule="" Nov 02 06:16:51.772129 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.762673 5916 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 02 06:16:51.772129 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.762677 5916 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 02 06:16:51.772129 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.762811 5916 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 06:16:51.772129 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.762815 5916 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 06:16:51.772129 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.762819 5916 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 06:16:51.772129 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.762822 5916 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 06:16:51.772129 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.762826 5916 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 06:16:51.772129 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.762829 5916 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 06:16:51.772129 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.762831 5916 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 06:16:51.772129 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.762834 5916 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 06:16:51.772129 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.762837 5916 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 06:16:51.772770 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.762840 5916 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 06:16:51.772770 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.762843 5916 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 06:16:51.772770 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.762845 5916 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 06:16:51.772770 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.762848 5916 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 06:16:51.772770 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.762851 5916 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 06:16:51.772770 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.762855 5916 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 06:16:51.772770 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.762859 5916 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 06:16:51.772770 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.762862 5916 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 06:16:51.772770 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.762865 5916 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 06:16:51.772770 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.762867 5916 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 06:16:51.772770 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.762870 5916 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 06:16:51.772770 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.762873 5916 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 06:16:51.772770 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.762875 5916 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 06:16:51.772770 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.762878 5916 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 06:16:51.772770 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.762880 5916 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 06:16:51.772770 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.762883 5916 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 06:16:51.772770 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.762886 5916 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 06:16:51.772770 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.762888 5916 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 06:16:51.772770 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.762891 5916 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 06:16:51.773261 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.762893 5916 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 06:16:51.773261 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.762896 5916 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 06:16:51.773261 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.762898 5916 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 06:16:51.773261 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.762916 5916 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 06:16:51.773261 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.762920 5916 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 06:16:51.773261 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.762922 5916 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 06:16:51.773261 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.762925 5916 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 06:16:51.773261 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.762928 5916 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 06:16:51.773261 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.762930 5916 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 06:16:51.773261 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.762933 5916 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 06:16:51.773261 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.762935 5916 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 06:16:51.773261 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.762938 5916 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 06:16:51.773261 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.762940 5916 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 06:16:51.773261 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.762943 5916 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 06:16:51.773261 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.762945 5916 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 06:16:51.773261 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.762948 5916 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 06:16:51.773261 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.762951 5916 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 06:16:51.773261 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.762954 5916 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 06:16:51.773261 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.762958 5916 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 06:16:51.773261 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.762962 5916 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 06:16:51.773261 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.762965 5916 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 06:16:51.773773 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.762968 5916 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 06:16:51.773773 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.762972 5916 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 06:16:51.773773 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.762976 5916 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 06:16:51.773773 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.762979 5916 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 06:16:51.773773 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.762982 5916 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 06:16:51.773773 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.762985 5916 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 06:16:51.773773 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.762988 5916 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 06:16:51.773773 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.762991 5916 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 06:16:51.773773 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.762993 5916 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 06:16:51.773773 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.762996 5916 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 06:16:51.773773 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.762999 5916 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 06:16:51.773773 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.763001 5916 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 06:16:51.773773 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.763004 5916 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 06:16:51.773773 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.763007 5916 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 06:16:51.773773 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.763009 5916 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 06:16:51.773773 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.763012 5916 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 06:16:51.773773 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.763014 5916 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 06:16:51.773773 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.763017 5916 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 06:16:51.774255 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.763020 5916 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 06:16:51.774255 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.763022 5916 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 06:16:51.774255 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.763025 5916 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 06:16:51.774255 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.763027 5916 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 06:16:51.774255 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.763030 5916 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 06:16:51.774255 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.763033 5916 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 06:16:51.774255 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.763035 5916 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 06:16:51.774255 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.763038 5916 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 06:16:51.774255 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.763040 5916 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 06:16:51.774255 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.763043 5916 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 06:16:51.774255 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.763046 5916 feature_gate.go:386] unrecognized feature gate: Example Nov 02 06:16:51.774255 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.763050 5916 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 06:16:51.774255 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.763054 5916 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 06:16:51.774255 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.763057 5916 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 06:16:51.774255 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.763060 5916 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 06:16:51.774255 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.763064 5916 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 06:16:51.774255 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.763068 5916 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 06:16:51.774255 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.763071 5916 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 06:16:51.774255 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.763074 5916 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 06:16:51.774738 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.763076 5916 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 06:16:51.774738 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.763079 5916 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 06:16:51.774738 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.763082 5916 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 06:16:51.774738 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.763084 5916 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 06:16:51.774738 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.763087 5916 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 06:16:51.774738 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.763089 5916 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 06:16:51.774738 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.763092 5916 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 06:16:51.774738 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.763094 5916 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 06:16:51.774738 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.763097 5916 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 06:16:51.774738 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.763099 5916 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 06:16:51.774738 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.763101 5916 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 06:16:51.774738 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.763104 5916 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 06:16:51.774738 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.763107 5916 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 06:16:51.774738 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.763109 5916 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 06:16:51.774738 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.763112 5916 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 06:16:51.774738 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.763115 5916 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 06:16:51.774738 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.763117 5916 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 06:16:51.774738 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.763120 5916 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 06:16:51.774738 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.763122 5916 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 06:16:51.775226 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.763125 5916 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 06:16:51.775226 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.763152 5916 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 06:16:51.775226 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.763192 5916 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 06:16:51.775226 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.763196 5916 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 06:16:51.775226 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.763199 5916 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 06:16:51.775226 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.763203 5916 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 06:16:51.775226 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.763210 5916 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 06:16:51.775226 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.763215 5916 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 06:16:51.775226 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.763218 5916 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 06:16:51.775226 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.763220 5916 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 06:16:51.775226 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.763223 5916 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 06:16:51.775226 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.763226 5916 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 06:16:51.775226 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.763229 5916 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 06:16:51.775226 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.763232 5916 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 06:16:51.775226 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.763235 5916 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 06:16:51.775226 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.763237 5916 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 06:16:51.775226 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.763240 5916 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 06:16:51.775643 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.763244 5916 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 06:16:51.775643 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.763249 5916 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 06:16:51.775643 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.763252 5916 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 06:16:51.775643 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.763254 5916 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 06:16:51.775643 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.763257 5916 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 06:16:51.775643 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.763260 5916 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 06:16:51.775643 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.763263 5916 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 06:16:51.775643 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.763265 5916 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 06:16:51.775643 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.763268 5916 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 06:16:51.775643 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.763270 5916 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 06:16:51.775643 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.763273 5916 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 06:16:51.775643 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.763276 5916 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 06:16:51.775643 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.763278 5916 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 06:16:51.775643 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.763281 5916 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 06:16:51.775643 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.763283 5916 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 06:16:51.775643 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.763286 5916 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 06:16:51.775643 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.763288 5916 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 06:16:51.775643 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.763291 5916 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 06:16:51.776101 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.763294 5916 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 06:16:51.776101 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.763296 5916 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 06:16:51.776101 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.763301 5916 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 06:16:51.776101 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.763304 5916 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 06:16:51.776101 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.763308 5916 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 06:16:51.776101 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.763311 5916 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 06:16:51.776101 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.763315 5916 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 06:16:51.776101 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.763317 5916 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 06:16:51.776101 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.763320 5916 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 06:16:51.776101 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.763323 5916 feature_gate.go:386] unrecognized feature gate: Example Nov 02 06:16:51.776101 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.763326 5916 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 06:16:51.776101 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.763329 5916 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 06:16:51.776101 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.763331 5916 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 06:16:51.776101 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.763334 5916 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 06:16:51.776101 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.763338 5916 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 06:16:51.776101 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.763342 5916 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 06:16:51.776101 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.763345 5916 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 06:16:51.776101 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.763348 5916 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 06:16:51.776101 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.763352 5916 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 06:16:51.776101 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.763355 5916 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 06:16:51.776661 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.763358 5916 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 06:16:51.776661 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.763360 5916 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 06:16:51.776661 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.763363 5916 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 06:16:51.776661 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.763366 5916 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 06:16:51.776661 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.763368 5916 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 06:16:51.776661 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.763371 5916 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 06:16:51.776661 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.763374 5916 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 06:16:51.776661 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.763376 5916 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 06:16:51.776661 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.763379 5916 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 06:16:51.776661 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.763381 5916 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 06:16:51.776661 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.763384 5916 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 06:16:51.776661 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.763386 5916 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 06:16:51.776661 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.763389 5916 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 06:16:51.776661 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.763392 5916 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 06:16:51.776661 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.763395 5916 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 06:16:51.776661 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.763398 5916 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 06:16:51.776661 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.763401 5916 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 06:16:51.776661 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.763404 5916 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 06:16:51.776661 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.763407 5916 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 06:16:51.776661 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.763409 5916 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 06:16:51.777195 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.763412 5916 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 06:16:51.777195 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.763415 5916 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 06:16:51.777195 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.763417 5916 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 06:16:51.777195 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.763419 5916 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 06:16:51.777195 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.763422 5916 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 06:16:51.777195 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.763425 5916 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 06:16:51.777195 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.763427 5916 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 06:16:51.777195 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.763430 5916 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 06:16:51.777195 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.763432 5916 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 06:16:51.777195 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.763435 5916 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 06:16:51.777195 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.763437 5916 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 06:16:51.777195 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.763440 5916 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 06:16:51.777195 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.763442 5916 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 06:16:51.777195 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.763445 5916 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 06:16:51.777195 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.763448 5916 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 06:16:51.777195 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.763450 5916 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 06:16:51.777195 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.763453 5916 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 06:16:51.777195 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.763455 5916 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 06:16:51.777195 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.763458 5916 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 06:16:51.777655 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.763460 5916 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 06:16:51.777655 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.763463 5916 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 06:16:51.777655 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.763465 5916 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 06:16:51.777655 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.763468 5916 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 06:16:51.777655 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.763471 5916 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 06:16:51.777655 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.763473 5916 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 06:16:51.777655 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.763476 5916 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 06:16:51.777655 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.763479 5916 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 06:16:51.777655 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.763482 5916 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 06:16:51.777655 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.763484 5916 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 06:16:51.777655 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.763487 5916 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 06:16:51.777655 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.763491 5916 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 06:16:51.777655 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.763494 5916 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 06:16:51.777655 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.763496 5916 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 06:16:51.777655 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.768374 5916 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 02 06:16:51.777655 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.768391 5916 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 02 06:16:51.777655 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.768410 5916 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 02 06:16:51.777655 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.768415 5916 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 02 06:16:51.777655 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.768475 5916 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 06:16:51.777655 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.768480 5916 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 06:16:51.778174 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.768484 5916 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 06:16:51.778174 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.768487 5916 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 06:16:51.778174 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.768490 5916 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 06:16:51.778174 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.768493 5916 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 06:16:51.778174 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.768496 5916 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 06:16:51.778174 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.768499 5916 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 06:16:51.778174 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.768502 5916 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 06:16:51.778174 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.768504 5916 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 06:16:51.778174 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.768507 5916 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 06:16:51.778174 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.768511 5916 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 06:16:51.778174 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.768514 5916 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 06:16:51.778174 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.768517 5916 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 06:16:51.778174 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.768519 5916 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 06:16:51.778174 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.768522 5916 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 06:16:51.778174 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.768526 5916 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 06:16:51.778174 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.768530 5916 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 06:16:51.778174 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.768534 5916 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 06:16:51.778174 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.768537 5916 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 06:16:51.778710 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.768540 5916 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 06:16:51.778710 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.768543 5916 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 06:16:51.778710 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.768546 5916 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 06:16:51.778710 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.768549 5916 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 06:16:51.778710 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.768552 5916 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 06:16:51.778710 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.768554 5916 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 06:16:51.778710 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.768557 5916 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 06:16:51.778710 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.768560 5916 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 06:16:51.778710 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.768563 5916 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 06:16:51.778710 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.768566 5916 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 06:16:51.778710 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.768568 5916 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 06:16:51.778710 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.768571 5916 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 06:16:51.778710 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.768574 5916 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 06:16:51.778710 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.768577 5916 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 06:16:51.778710 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.768579 5916 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 06:16:51.778710 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.768582 5916 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 06:16:51.778710 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.768584 5916 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 06:16:51.778710 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.768587 5916 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 06:16:51.778710 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.768590 5916 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 06:16:51.778710 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.768592 5916 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 06:16:51.779404 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.768595 5916 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 06:16:51.779404 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.768597 5916 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 06:16:51.779404 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.768600 5916 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 06:16:51.779404 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.768602 5916 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 06:16:51.779404 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.768606 5916 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 06:16:51.779404 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.768609 5916 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 06:16:51.779404 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.768612 5916 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 06:16:51.779404 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.768615 5916 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 06:16:51.779404 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.768618 5916 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 06:16:51.779404 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.768621 5916 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 06:16:51.779404 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.768623 5916 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 06:16:51.779404 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.768626 5916 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 06:16:51.779404 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.768629 5916 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 06:16:51.779404 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.768632 5916 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 06:16:51.779404 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.768635 5916 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 06:16:51.779404 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.768637 5916 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 06:16:51.779404 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.768640 5916 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 06:16:51.779404 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.768642 5916 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 06:16:51.779404 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.768645 5916 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 06:16:51.779404 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.768648 5916 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 06:16:51.779916 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.768651 5916 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 06:16:51.779916 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.768654 5916 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 06:16:51.779916 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.768656 5916 feature_gate.go:386] unrecognized feature gate: Example Nov 02 06:16:51.779916 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.768659 5916 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 06:16:51.779916 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.768662 5916 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 06:16:51.779916 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.768665 5916 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 06:16:51.779916 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.768667 5916 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 06:16:51.779916 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.768670 5916 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 06:16:51.779916 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.768673 5916 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 06:16:51.779916 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.768676 5916 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 06:16:51.779916 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.768678 5916 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 06:16:51.779916 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.768681 5916 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 06:16:51.779916 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.768684 5916 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 06:16:51.779916 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.768686 5916 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 06:16:51.779916 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.768689 5916 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 06:16:51.779916 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.768691 5916 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 06:16:51.779916 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.768694 5916 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 06:16:51.779916 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.768697 5916 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 06:16:51.779916 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.768699 5916 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 06:16:51.779916 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.768703 5916 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 06:16:51.780440 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.768706 5916 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 06:16:51.780440 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.768708 5916 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 06:16:51.780440 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.768711 5916 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 06:16:51.780440 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.768714 5916 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 06:16:51.780440 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.768717 5916 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 06:16:51.780440 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.768719 5916 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 06:16:51.780440 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.768722 5916 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 06:16:51.780440 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.768725 5916 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 06:16:51.780440 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.768727 5916 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 06:16:51.780440 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.768730 5916 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 06:16:51.780440 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.768733 5916 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 06:16:51.780440 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.768735 5916 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 06:16:51.780440 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.768738 5916 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 06:16:51.780440 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.768742 5916 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 06:16:51.780440 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.768746 5916 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 06:16:51.780440 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.768749 5916 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 06:16:51.780440 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.768752 5916 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 06:16:51.780440 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.768755 5916 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 06:16:51.780440 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.768758 5916 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 06:16:51.780440 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.768760 5916 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 06:16:51.781226 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.768763 5916 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 06:16:51.781226 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.768766 5916 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 06:16:51.781226 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.768769 5916 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 06:16:51.781226 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.768772 5916 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 06:16:51.781226 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.768774 5916 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 06:16:51.781226 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.768777 5916 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 06:16:51.781226 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.768807 5916 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 06:16:51.781226 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.768985 5916 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 06:16:51.781226 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.768990 5916 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 06:16:51.781226 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.768993 5916 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 06:16:51.781226 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.768997 5916 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 06:16:51.781226 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.768999 5916 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 06:16:51.781226 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769003 5916 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 06:16:51.781226 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769006 5916 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 06:16:51.781226 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769009 5916 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 06:16:51.781790 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769012 5916 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 06:16:51.781790 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769014 5916 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 06:16:51.781790 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769017 5916 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 06:16:51.781790 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769019 5916 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 06:16:51.781790 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769022 5916 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 06:16:51.781790 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769025 5916 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 06:16:51.781790 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769027 5916 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 06:16:51.781790 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769031 5916 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 06:16:51.781790 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769033 5916 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 06:16:51.781790 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769036 5916 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 06:16:51.781790 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769039 5916 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 06:16:51.781790 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769041 5916 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 06:16:51.781790 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769044 5916 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 06:16:51.781790 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769046 5916 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 06:16:51.781790 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769049 5916 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 06:16:51.781790 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769051 5916 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 06:16:51.781790 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769054 5916 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 06:16:51.781790 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769056 5916 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 06:16:51.781790 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769059 5916 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 06:16:51.781790 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769062 5916 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 06:16:51.782378 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769064 5916 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 06:16:51.782378 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769066 5916 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 06:16:51.782378 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769069 5916 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 06:16:51.782378 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769071 5916 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 06:16:51.782378 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769074 5916 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 06:16:51.782378 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769076 5916 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 06:16:51.782378 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769079 5916 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 06:16:51.782378 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769081 5916 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 06:16:51.782378 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769084 5916 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 06:16:51.782378 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769088 5916 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 06:16:51.782378 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769093 5916 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 06:16:51.782378 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769098 5916 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 06:16:51.782378 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769100 5916 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 06:16:51.782378 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769103 5916 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 06:16:51.782378 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769105 5916 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 06:16:51.782378 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769108 5916 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 06:16:51.782378 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769111 5916 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 06:16:51.782378 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769113 5916 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 06:16:51.782378 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769116 5916 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 06:16:51.782914 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769119 5916 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 06:16:51.782914 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769121 5916 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 06:16:51.782914 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769124 5916 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 06:16:51.782914 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769127 5916 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 06:16:51.782914 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769129 5916 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 06:16:51.782914 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769132 5916 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 06:16:51.782914 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769135 5916 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 06:16:51.782914 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769137 5916 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 06:16:51.782914 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769140 5916 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 06:16:51.782914 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769143 5916 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 06:16:51.782914 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769145 5916 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 06:16:51.782914 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769148 5916 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 06:16:51.782914 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769150 5916 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 06:16:51.782914 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769153 5916 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 06:16:51.782914 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769155 5916 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 06:16:51.782914 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769158 5916 feature_gate.go:386] unrecognized feature gate: Example Nov 02 06:16:51.782914 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769161 5916 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 06:16:51.782914 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769163 5916 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 06:16:51.782914 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769166 5916 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 06:16:51.782914 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769168 5916 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 06:16:51.783457 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769171 5916 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 06:16:51.783457 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769173 5916 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 06:16:51.783457 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769176 5916 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 06:16:51.783457 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769179 5916 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 06:16:51.783457 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769182 5916 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 06:16:51.783457 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769184 5916 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 06:16:51.783457 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769187 5916 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 06:16:51.783457 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769189 5916 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 06:16:51.783457 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769192 5916 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 06:16:51.783457 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769195 5916 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 06:16:51.783457 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769197 5916 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 06:16:51.783457 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769200 5916 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 06:16:51.783457 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769202 5916 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 06:16:51.783457 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769205 5916 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 06:16:51.783457 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769208 5916 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 06:16:51.783457 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769210 5916 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 06:16:51.783457 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769213 5916 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 06:16:51.783457 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769216 5916 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 06:16:51.783457 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769218 5916 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 06:16:51.783457 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769221 5916 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 06:16:51.784083 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769223 5916 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 06:16:51.784083 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769227 5916 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 06:16:51.784083 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769230 5916 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 06:16:51.784083 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769234 5916 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 06:16:51.784083 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769237 5916 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 06:16:51.784083 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769239 5916 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 06:16:51.784083 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769242 5916 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 06:16:51.784083 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769245 5916 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 06:16:51.784083 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769247 5916 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 06:16:51.784083 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769250 5916 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 06:16:51.784083 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769255 5916 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 06:16:51.784083 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769258 5916 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 06:16:51.784083 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769260 5916 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 06:16:51.784083 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769263 5916 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 06:16:51.784083 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769266 5916 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 06:16:51.784083 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769268 5916 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 06:16:51.784083 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769271 5916 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 06:16:51.784083 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769274 5916 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 06:16:51.784083 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769276 5916 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 06:16:51.784564 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.769299 5916 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 06:16:51.784564 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769331 5916 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 06:16:51.784564 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769335 5916 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 06:16:51.784564 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769338 5916 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 06:16:51.784564 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769341 5916 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 06:16:51.784564 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769344 5916 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 06:16:51.784564 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769347 5916 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 06:16:51.784564 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769350 5916 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 06:16:51.784564 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769353 5916 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 06:16:51.784564 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769355 5916 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 06:16:51.784564 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769358 5916 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 06:16:51.784564 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769361 5916 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 06:16:51.784564 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769363 5916 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 06:16:51.784564 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769367 5916 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 06:16:51.784564 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769369 5916 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 06:16:51.784564 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769372 5916 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 06:16:51.785018 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769375 5916 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 06:16:51.785018 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769377 5916 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 06:16:51.785018 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769380 5916 feature_gate.go:386] unrecognized feature gate: Example Nov 02 06:16:51.785018 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769383 5916 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 06:16:51.785018 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769385 5916 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 06:16:51.785018 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769388 5916 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 06:16:51.785018 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769390 5916 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 06:16:51.785018 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769393 5916 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 06:16:51.785018 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769398 5916 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 06:16:51.785018 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769402 5916 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 06:16:51.785018 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769404 5916 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 06:16:51.785018 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769407 5916 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 06:16:51.785018 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769410 5916 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 06:16:51.785018 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769412 5916 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 06:16:51.785018 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769415 5916 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 06:16:51.785018 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769418 5916 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 06:16:51.785018 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769421 5916 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 06:16:51.785018 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769424 5916 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 06:16:51.785018 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769427 5916 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 06:16:51.785018 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769429 5916 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 06:16:51.785520 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769432 5916 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 06:16:51.785520 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769435 5916 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 06:16:51.785520 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769437 5916 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 06:16:51.785520 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769440 5916 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 06:16:51.785520 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769442 5916 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 06:16:51.785520 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769445 5916 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 06:16:51.785520 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769448 5916 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 06:16:51.785520 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769450 5916 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 06:16:51.785520 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769454 5916 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 06:16:51.785520 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769458 5916 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 06:16:51.785520 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769461 5916 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 06:16:51.785520 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769463 5916 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 06:16:51.785520 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769466 5916 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 06:16:51.785520 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769469 5916 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 06:16:51.785520 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769472 5916 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 06:16:51.785520 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769474 5916 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 06:16:51.785520 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769477 5916 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 06:16:51.785520 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769479 5916 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 06:16:51.785520 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769482 5916 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 06:16:51.786068 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769484 5916 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 06:16:51.786068 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769489 5916 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 06:16:51.786068 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769493 5916 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 06:16:51.786068 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769495 5916 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 06:16:51.786068 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769498 5916 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 06:16:51.786068 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769500 5916 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 06:16:51.786068 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769503 5916 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 06:16:51.786068 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769506 5916 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 06:16:51.786068 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769508 5916 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 06:16:51.786068 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769511 5916 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 06:16:51.786068 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769514 5916 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 06:16:51.786068 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769517 5916 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 06:16:51.786068 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769519 5916 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 06:16:51.786068 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769522 5916 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 06:16:51.786068 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769524 5916 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 06:16:51.786068 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769527 5916 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 06:16:51.786068 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769529 5916 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 06:16:51.786068 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769532 5916 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 06:16:51.786068 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769535 5916 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 06:16:51.786585 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769537 5916 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 06:16:51.786585 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769540 5916 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 06:16:51.786585 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769543 5916 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 06:16:51.786585 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769546 5916 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 06:16:51.786585 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769549 5916 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 06:16:51.786585 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769551 5916 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 06:16:51.786585 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769554 5916 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 06:16:51.786585 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769557 5916 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 06:16:51.786585 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769559 5916 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 06:16:51.786585 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769562 5916 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 06:16:51.786585 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769565 5916 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 06:16:51.786585 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769567 5916 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 06:16:51.786585 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769570 5916 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 06:16:51.786585 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769572 5916 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 06:16:51.786585 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769576 5916 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 06:16:51.786585 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769580 5916 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 06:16:51.786585 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769583 5916 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 06:16:51.786585 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769585 5916 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 06:16:51.786585 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769588 5916 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 06:16:51.786585 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769590 5916 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 06:16:51.787114 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769593 5916 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 06:16:51.787114 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769596 5916 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 06:16:51.787114 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769598 5916 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 06:16:51.787114 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769602 5916 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 06:16:51.787114 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769604 5916 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 06:16:51.787114 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769608 5916 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 06:16:51.787114 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769612 5916 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 06:16:51.787114 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769616 5916 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 06:16:51.787114 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769618 5916 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 06:16:51.787114 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769621 5916 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 06:16:51.787114 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769624 5916 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 06:16:51.787114 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769626 5916 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 06:16:51.787114 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:16:51.769629 5916 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 06:16:51.787114 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.769688 5916 server.go:957] "Client rotation is on, will bootstrap in background" Nov 02 06:16:51.787114 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.770640 5916 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Nov 02 06:16:51.787114 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.770697 5916 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Nov 02 06:16:51.787114 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.770871 5916 server.go:1014] "Starting client certificate rotation" Nov 02 06:16:51.787114 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.771025 5916 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 02 06:16:51.787562 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.771079 5916 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 06:10:38 +0000 UTC" deadline="2028-07-31 02:56:46.874382142 +0000 UTC" Nov 02 06:16:51.787562 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.771107 5916 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15284h39m55.103277461s" Nov 02 06:16:51.787562 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.771340 5916 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 06:16:51.787562 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.772424 5916 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 06:16:51.787562 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.773574 5916 log.go:25] "Validated CRI v1 runtime API" Nov 02 06:16:51.787562 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.775481 5916 log.go:25] "Validated CRI v1 image API" Nov 02 06:16:51.787562 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.776280 5916 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 02 06:16:51.787562 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.778748 5916 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 a3052cc1-5d54-4510-a252-8750d465bd42:/dev/nvme0n1p4 b74f8c3a-d9d7-412c-b5ea-ebb494b55370:/dev/nvme0n1p3] Nov 02 06:16:51.787803 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.778766 5916 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/05a094dc530991421555eea671fb2cf05126ac45b8261be9db29c7b4f57ad03b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/05a094dc530991421555eea671fb2cf05126ac45b8261be9db29c7b4f57ad03b/userdata/shm major:0 minor:437 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/119ed3531b52ab511c43385dc22917968f984e5c99e56eb48bdcb85ee43db915/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/119ed3531b52ab511c43385dc22917968f984e5c99e56eb48bdcb85ee43db915/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/12a8615ab34e8cb0bbe17c3b60704970d0bcd860a9a74b7984b41986462505ea/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/12a8615ab34e8cb0bbe17c3b60704970d0bcd860a9a74b7984b41986462505ea/userdata/shm major:0 minor:353 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1309dfc627fd0eddb9d56bfeb55c303d58564eed884481f3f3119988a12fbe55/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1309dfc627fd0eddb9d56bfeb55c303d58564eed884481f3f3119988a12fbe55/userdata/shm major:0 minor:351 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1e1cdd6d7235fef42913f68e6790d9df4741bd50deb1e6f2c87c1978b3d17f19/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1e1cdd6d7235fef42913f68e6790d9df4741bd50deb1e6f2c87c1978b3d17f19/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/216fc5ad7a00312abf4a46ba90063240b9601015205ec36f60ddec22af761f9c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/216fc5ad7a00312abf4a46ba90063240b9601015205ec36f60ddec22af761f9c/userdata/shm major:0 minor:244 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/23685e5c401a72636d0a3f2994a72afdb729f870e680b5dbd7d42a469c46b89e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/23685e5c401a72636d0a3f2994a72afdb729f870e680b5dbd7d42a469c46b89e/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3fe2b451a165e20914129200c0e1ed9201cfbb89b1eb8d68742f2ca731715d93/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3fe2b451a165e20914129200c0e1ed9201cfbb89b1eb8d68742f2ca731715d93/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/418136c5b736666c736e9cbf8be032be97dc9918c1dd664aae798cd2baf3649b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/418136c5b736666c736e9cbf8be032be97dc9918c1dd664aae798cd2baf3649b/userdata/shm major:0 minor:428 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4963902e93a3be86d04c977ac33a031fb4a55dbdfd7d021a6e3debdefa6f663a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4963902e93a3be86d04c977ac33a031fb4a55dbdfd7d021a6e3debdefa6f663a/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4b603936d475f6f974e687dc0306c917a65be18ac6c747bade66b4f9b09d5a3e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4b603936d475f6f974e687dc0306c917a65be18ac6c747bade66b4f9b09d5a3e/userdata/shm major:0 minor:330 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5165263c016e50839257e4586fc7f9fc1b015d1999d655cc835b1d5e978df321/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5165263c016e50839257e4586fc7f9fc1b015d1999d655cc835b1d5e978df321/userdata/shm major:0 minor:443 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/52cfcf77d0a8a8269f6a1d6364aa5d81ce424af636a65e86e9902c6b1b9ee167/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/52cfcf77d0a8a8269f6a1d6364aa5d81ce424af636a65e86e9902c6b1b9ee167/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5edecaa33e325dbc20c6c14bc24482a1098e38251bf6e78f228cc2b199f6fc9b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5edecaa33e325dbc20c6c14bc24482a1098e38251bf6e78f228cc2b199f6fc9b/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/60673025a4b89a0ee5f4cb8c3345979e787c7da063a0e2faa54ef8740d8f0781/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/60673025a4b89a0ee5f4cb8c3345979e787c7da063a0e2faa54ef8740d8f0781/userdata/shm major:0 minor:455 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/62d6ac9953134850fb004827c8a962446a4b2f6330e743b941096a9ab30409f5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/62d6ac9953134850fb004827c8a962446a4b2f6330e743b941096a9ab30409f5/userdata/shm major:0 minor:246 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6f2af3023df9565bf072c1569930184b0aa5c7af28b5d5da6f6d32982f397446/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6f2af3023df9565bf072c1569930184b0aa5c7af28b5d5da6f6d32982f397446/userdata/shm major:0 minor:318 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/755856f4f662e5878f1383579803c0114371ba096156e09fb4df933877698348/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/755856f4f662e5878f1383579803c0114371ba096156e09fb4df933877698348/userdata/shm major:0 minor:221 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7e7bb32b8719598da92f32821b935863ee8427e701ef1c413fe40777eed58690/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7e7bb32b8719598da92f32821b935863ee8427e701ef1c413fe40777eed58690/userdata/shm major:0 minor:463 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7e9463781594333c81318e1426f180685a00493bdbfb6deb6f58e1e836dffb13/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7e9463781594333c81318e1426f180685a00493bdbfb6deb6f58e1e836dffb13/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/84ef672c27542c8ee2cd9aa8315ffdac87f5adea3ca97472f1d9343ae269bc89/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/84ef672c27542c8ee2cd9aa8315ffdac87f5adea3ca97472f1d9343ae269bc89/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/860a45d95193ea55c54e30a2f0d13b46a725340587fe8a32c4c6a6be79374f1d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/860a45d95193ea55c54e30a2f0d13b46a725340587fe8a32c4c6a6be79374f1d/userdata/shm major:0 minor:223 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8b4aca9cba52d27e69b569155b9d1855545be2e53e75210dc3ffe05d64dd23df/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8b4aca9cba52d27e69b569155b9d1855545be2e53e75210dc3ffe05d64dd23df/userdata/shm major:0 minor:359 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/91abcd1f2335c5276a0294d88a772c710c698e73c3a2b82f1a7042821d779a32/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/91abcd1f2335c5276a0294d88a772c710c698e73c3a2b82f1a7042821d779a32/userdata/shm major:0 minor:240 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a471a58ea4a732d84e89fb09f437def55b26113096801bc4a4e213c7894fbe5b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a471a58ea4a732d84e89fb09f437def55b26113096801bc4a4e213c7894fbe5b/userdata/shm major:0 minor:349 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ab0ebd2db011ef10e8f93786016b5cc45593e7dc0f17ead3905307dd1d9fcd2d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ab0ebd2db011ef10e8f93786016b5cc45593e7dc0f17ead3905307dd1d9fcd2d/userdata/shm major:0 minor:236 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b5d368d976619eaa837c56d4e22406db6bf7495a6d9575c227809c8db3f208ac/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b5d368d976619eaa837c56d4e22406db6bf7495a6d9575c227809c8db3f208ac/userdata/shm major:0 minor:60 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c1ac0d913b943b018616c75971989a95da11db3c09c0128e4db983e8b0d0eaa3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c1ac0d913b943b018616c75971989a95da11db3c09c0128e4db983e8b0d0eaa3/userdata/shm major:0 minor:62 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c53b73150c37aad5859309630eab56b7a553276fb27542d80ef212490baaeb30/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c53b73150c37aad5859309630eab56b7a553276fb27542d80ef212490baaeb30/userdata/shm major:0 minor:238 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d1b287d3147a3d6fbc93b84f0c5883cfd49ac31f846e5d01f59df87df61be411/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d1b287d3147a3d6fbc93b84f0c5883cfd49ac31f846e5d01f59df87df61be411/userdata/shm major:0 minor:234 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d319bf628cd7bcbfc1dcc73a4d9d12b8dfb00fc85e674cf3e9929dca6b16bfc3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d319bf628cd7bcbfc1dcc73a4d9d12b8dfb00fc85e674cf3e9929dca6b16bfc3/userdata/shm major:0 minor:357 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e0370d945d4e54d879de7ca1d43138dcadfb5673f42e318252a19874123f5c6c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e0370d945d4e54d879de7ca1d43138dcadfb5673f42e318252a19874123f5c6c/userdata/shm major:0 minor:214 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fadf7f58c12e185657a35443e5641a2c59dc461cb89e0db18ce1be9b812892b7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fadf7f58c12e185657a35443e5641a2c59dc461cb89e0db18ce1be9b812892b7/userdata/shm major:0 minor:445 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fdbc98481353628068c730cc914f394bf2248182b7dc1d598e90142adad72ecf/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fdbc98481353628068c730cc914f394bf2248182b7dc1d598e90142adad72ecf/userdata/shm major:0 minor:407 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0d2b53c4-9d39-47fc-9367-c9a8e7f5da81/volumes/kubernetes.io~projected/kube-api-access-8mplc:{mountpoint:/var/lib/kubelet/pods/0d2b53c4-9d39-47fc-9367-c9a8e7f5da81/volumes/kubernetes.io~projected/kube-api-access-8mplc major:0 minor:229 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0d2b53c4-9d39-47fc-9367-c9a8e7f5da81/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/0d2b53c4-9d39-47fc-9367-c9a8e7f5da81/volumes/kubernetes.io~secret/cert major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/121e38c0-6fa3-46aa-9191-5c6f91c8764c/volumes/kubernetes.io~projected/kube-api-access-fnw94:{mountpoint:/var/lib/kubelet/pods/121e38c0-6fa3-46aa-9191-5c6f91c8764c/volumes/kubernetes.io~projected/kube-api-access-fnw94 major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/12b6cc1c-537d-4125-92ff-fe5f6927c53e/volumes/kubernetes.io~projected/kube-api-access-bmsjq:{mountpoint:/var/lib/kubelet/pods/12b6cc1c-537d-4125-92ff-fe5f6927c53e/volumes/kubernetes.io~projected/kube-api-access-bmsjq major:0 minor:233 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/12b6cc1c-537d-4125-92ff-fe5f6927c53e/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/12b6cc1c-537d-4125-92ff-fe5f6927c53e/volumes/kubernetes.io~secret/serving-cert major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/17dee72b-2f47-4016-8ffe-3a89934e66ee/volumes/kubernetes.io~projected/kube-api-access-9tz6t:{mountpoint:/var/lib/kubelet/pods/17dee72b-2f47-4016-8ffe-3a89934e66ee/volumes/kubernetes.io~projected/kube-api-access-9tz6t major:0 minor:228 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/17dee72b-2f47-4016-8ffe-3a89934e66ee/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/17dee72b-2f47-4016-8ffe-3a89934e66ee/volumes/kubernetes.io~secret/serving-cert major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/20cb7690-c976-471c-b459-a9ab15cc51bf/volumes/kubernetes.io~projected/kube-api-access-5989k:{mountpoint:/var/lib/kubelet/pods/20cb7690-c976-471c-b459-a9ab15cc51bf/volumes/kubernetes.io~projected/kube-api-access-5989k major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3419bd74-22aa-4841-87d2-c4340e01dae9/volumes/kubernetes.io~projected/kube-api-access-vz9mf:{mountpoint:/var/lib/kubelet/pods/3419bd74-22aa-4841-87d2-c4340e01dae9/volumes/kubernetes.io~projected/kube-api-access-vz9mf major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3500e682-2f15-45a2-92d6-bbc35dae3231/volumes/kubernetes.io~projected/kube-api-access-v8lqt:{mountpoint:/var/lib/kubelet/pods/3500e682-2f15-45a2-92d6-bbc35dae3231/volumes/kubernetes.io~projected/kube-api-access-v8lqt major:0 minor:53 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/37982aa4-bdbd-40e8-a2dd-5094209f3c96/volumes/kubernetes.io~projected/kube-api-access-h7l5t:{mountpoint:/var/lib/kubelet/pods/37982aa4-bdbd-40e8-a2dd-5094209f3c96/volumes/kubernetes.io~projected/kube-api-access-h7l5t major:0 minor:232 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/37982aa4-bdbd-40e8-a2dd-5094209f3c96/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/37982aa4-bdbd-40e8-a2dd-5094209f3c96/volumes/kubernetes.io~secret/serving-cert major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3b6432a8-b0ce-4954-8e3d-8eb8782557eb/volumes/kubernetes.io~projected/kube-api-access-pvdp9:{mountpoint:/var/lib/kubelet/pods/3b6432a8-b0ce-4954-8e3d-8eb8782557eb/volumes/kubernetes.io~projected/kube-api-access-pvdp9 major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3b6432a8-b0ce-4954-8e3d-8eb8782557eb/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/3b6432a8-b0ce-4954-8e3d-8eb8782557eb/volumes/kubernetes.io~secret/metrics-certs major:0 minor:432 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3ced47b0-9cd0-4710-8f75-2a333437ed01/volumes/kubernetes.io~projected/kube-api-access-gnq6k:{mountpoint:/var/lib/kubelet/pods/3ced47b0-9cd0-4710-8f75-2a333437ed01/volumes/kubernetes.io~projected/kube-api-access-gnq6k major:0 minor:436 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3eec1082-df0b-420b-b4ff-5a1c0fab1d1d/volumes/kubernetes.io~projected/kube-api-access-k8vwr:{mountpoint:/var/lib/kubelet/pods/3eec1082-df0b-420b-b4ff-5a1c0fab1d1d/volumes/kubernetes.io~projected/kube-api-access-k8vwr major:0 minor:51 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3fcb7621-86c8-448f-b4fa-b5bd76be7356/volumes/kubernetes.io~projected/kube-api-access-mwh5f:{mountpoint:/var/lib/kubelet/pods/3fcb7621-86c8-448f-b4fa-b5bd76be7356/volumes/kubernetes.io~projected/kube-api-access-mwh5f major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3fcb7621-86c8-448f-b4fa-b5bd76be7356/volumes/kubernetes.io~secret/cluster-monitoring-operator-tls:{mountpoint:/var/lib/kubelet/pods/3fcb7621-86c8-448f-b4fa-b5bd76be7356/volumes/kubernetes.io~secret/cluster-monitoring-operator-tls major:0 minor:447 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/45e5ce56-5cab-4546-8b31-02fc06cfa0cc/volumes/kubernetes.io~projected/kube-api-access-xvpvr:{mountpoint:/var/lib/kubelet/pods/45e5ce56-5cab-4546-8b31-02fc06cfa0cc/volumes/kubernetes.io~projected/kube-api-access-xvpvr major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/45e5ce56-5cab-4546-8b31-02fc06cfa0cc/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/45e5ce56-5cab-4546-8b31-02fc06cfa0cc/volumes/kubernetes.io~secret/default-certificate major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/45e5ce56-5cab-4546-8b31-02fc06cfa0cc/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/45e5ce56-5cab-4546-8b31-02fc06cfa0cc/volumes/kubernetes.io~secret/metrics-certs major:0 minor:348 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/45e5ce56-5cab-4546-8b31-02fc06cfa0cc/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/45e5ce56-5cab-4546-8b31-02fc06cfa0cc/volumes/kubernetes.io~secret/stats-auth major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/49f1bd67-26ad-4fc9-b3d5-7bf7f6404aaa/volumes/kubernetes.io~projected/kube-api-access-v7jwb:{mountpoint:/var/lib/kubelet/pods/49f1bd67-26ad-4fc9-b3d5-7bf7f6404aaa/volumes/kubernetes.io~projected/kube-api-access-v7jwb major:0 minor:424 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4ba9b2b6-4be9-4303-8eab-cd464c5a32c7/volumes/kubernetes.io~projected/kube-api-access-dv9xj:{mountpoint:/var/lib/kubelet/pods/4ba9b2b6-4be9-4303-8eab-cd464c5a32c7/volumes/kubernetes.io~projected/kube-api-access-dv9xj major:0 minor:230 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4ba9b2b6-4be9-4303-8eab-cd464c5a32c7/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/4ba9b2b6-4be9-4303-8eab-cd464c5a32c7/volumes/kubernetes.io~secret/metrics-tls major:0 minor:355 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/53aa1e5a-c35f-4aa6-9f23-cc61ccd331fc/volumes/kubernetes.io~secret/networking-console-plugin-cert:{mountpoint:/var/lib/kubelet/pods/53aa1e5a-c35f-4aa6-9f23-cc61ccd331fc/volumes/kubernetes.io~secret/networking-console-plugin-cert major:0 minor:356 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7434256f-45c3-47ff-9111-352114b0f6d8/volumes/kubernetes.io~projected/kube-api-access-l5q7t:{mountpoint:/var/lib/kubelet/pods/7434256f-45c3-47ff-9111-352114b0f6d8/volumes/kubernetes.io~projected/kube-api-access-l5q7t major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/81b393f2-42de-4a99-8114-1445de4d6dc8/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/81b393f2-42de-4a99-8114-1445de4d6dc8/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/81b393f2-42de-4a99-8114-1445de4d6dc8/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/81b393f2-42de-4a99-8114-1445de4d6dc8/volumes/kubernetes.io~empty-dir/tmp major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/81b393f2-42de-4a99-8114-1445de4d6dc8/volumes/kubernetes.io~projected/kube-api-access-hrq9j:{mountpoint:/var/lib/kubelet/pods/81b393f2-42de-4a99-8114-1445de4d6dc8/volumes/kubernetes.io~projected/kube-api-access-hrq9j major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/87e797a0-1da6-41ca-af8d-7382afaf3987/volumes/kubernetes.io~projected/kube-api-access-l6vx8:{mountpoint:/var/lib/kubelet/pods/87e797a0-1da6-41ca-af8d-7382afaf3987/volumes/kubernetes.io~projected/kube-api-access-l6vx8 major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8b491a46-9f53-4a4f-97ed-d8b32e3a03fa/volumes/kubernetes.io~projected/kube-api-access-dpt4c:{mountpoint:/var/lib/kubelet/pods/8b491a46-9f53-4a4f-97ed-d8b32e3a03fa/volumes/kubernetes.io~projected/kube-api-access-dpt4c major:0 minor:300 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8b491a46-9f53-4a4f-97ed-d8b32e3a03fa/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/8b491a46-9f53-4a4f-97ed-d8b32e3a03fa/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:403 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8e8f2e60-de1b-4c8b-9285-56568c150d92/volumes/kubernetes.io~projected/kube-api-access-zrh7x:{mountpoint:/var/lib/kubelet/pods/8e8f2e60-de1b-4c8b-9285-56568c150d92/volumes/kubernetes.io~projected/kube-api-access-zrh7x major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a2ef8269-c442-42a8-8041-6c1b6aaf0746/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/a2ef8269-c442-42a8-8041-6c1b6aaf0746/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a2ef8269-c442-42a8-8041-6c1b6aaf0746/volumes/kubernetes.io~projected/kube-api-access-qzxsj:{mountpoint:/var/lib/kubelet/pods/a2ef8269-c442-42a8-8041-6c1b6aaf0746/volumes/kubernetes.io~projected/kube-api-access-qzxsj major:0 minor:52 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a2ef8269-c442-42a8-8041-6c1b6aaf0746/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/a2ef8269-c442-42a8-8041-6c1b6aaf0746/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a4319db2-eda9-47a0-b742-985c8bdd7ea2/volumes/kubernetes.io~projected/kube-api-access-qwdcw:{mountpoint:/var/lib/kubelet/pods/a4319db2-eda9-47a0-b742-985c8bdd7ea2/volumes/kubernetes.io~projected/kube-api-access-qwdcw major:0 minor:317 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ac6faf80-42b2-4c7a-8e55-abf6435b3d1e/volumes/kubernetes.io~projected/kube-api-access-pgkrw:{mountpoint:/var/lib/kubelet/pods/ac6faf80-42b2-4c7a-8e55-abf6435b3d1e/volumes/kubernetes.io~projected/kube-api-access-pgkrw major:0 minor:329 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ac6faf80-42b2-4c7a-8e55-abf6435b3d1e/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/ac6faf80-42b2-4c7a-8e55-abf6435b3d1e/volumes/kubernetes.io~secret/signing-key major:0 minor:324 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b65bb247-83f7-4749-9aa1-9b2677dd1b3d/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/b65bb247-83f7-4749-9aa1-9b2677dd1b3d/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:435 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cc33996a-0d4c-49c4-ba67-283bd6a2de14/volumes/kubernetes.io~projected/kube-api-access-rd8hq:{mountpoint:/var/lib/kubelet/pods/cc33996a-0d4c-49c4-ba67-283bd6a2de14/volumes/kubernetes.io~projected/kube-api-access-rd8hq major:0 minor:227 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cc33996a-0d4c-49c4-ba67-283bd6a2de14/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/cc33996a-0d4c-49c4-ba67-283bd6a2de14/volumes/kubernetes.io~secret/serving-cert major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cf8e8e65-8525-484b-adfe-f56517ccdc1b/volumes/kubernetes.io~projected/kube-api-access-vwqbt:{mountpoint:/var/lib/kubelet/pods/cf8e8e65-8525-484b-adfe-f56517ccdc1b/volumes/kubernetes.io~projected/kube-api-access-vwqbt major:0 minor:231 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cf8e8e65-8525-484b-adfe-f56517ccdc1b/volumes/kubernetes.io~secret/samples-operator-tls:{mountpoint:/var/lib/kubelet/pods/cf8e8e65-8525-484b-adfe-f56517ccdc1b/volumes/kubernetes.io~secret/samples-operator-tls major:0 minor:452 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d4615a9e-da82-4f8b-8225-d60836b1ba80/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/d4615a9e-da82-4f8b-8225-d60836b1ba80/volumes/kubernetes.io~secret/agent-certs major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d8d87317-3f41-46d4-a58e-3c7601e0d445/volumes/kubernetes.io~projected/kube-api-access-5kcrm:{mountpoint:/var/lib/kubelet/pods/d8d87317-3f41-46d4-a58e-3c7601e0d445/volumes/kubernetes.io~projected/kube-api-access-5kcrm major:0 minor:54 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dee3b502-2562-4b21-bc46-2031580c4b0a/volumes/kubernetes.io~projected/kube-api-access-txffk:{mountpoint:/var/lib/kubelet/pods/dee3b502-2562-4b21-bc46-2031580c4b0a/volumes/kubernetes.io~projected/kube-api-access-txffk major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f0531b4d-c954-4830-85f6-fe876c76437a/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/f0531b4d-c954-4830-85f6-fe876c76437a/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f0531b4d-c954-4830-85f6-fe876c76437a/volumes/kubernetes.io~projected/kube-api-access-m89g8:{mountpoint:/var/lib/kubelet/pods/f0531b4d-c954-4830-85f6-fe876c76437a/volumes/kubernetes.io~projected/kube-api-access-m89g8 major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f0531b4d-c954-4830-85f6-fe876c76437a/volumes/kubernetes.io~projected/registry-tls:{mountpoint:/var/lib/kubelet/pods/f0531b4d-c954-4830-85f6-fe876c76437a/volumes/kubernetes.io~projected/registry-tls major:0 minor:338 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f0531b4d-c954-4830-85f6-fe876c76437a/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/f0531b4d-c954-4830-85f6-fe876c76437a/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f0531b4d-c954-4830-85f6-fe876c76437a/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/f0531b4d-c954-4830-85f6-fe876c76437a/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f14e8781-bf82-4810-9909-c126ccdc1ce1/volumes/kubernetes.io~projected/kube-api-access-v7xzh:{mountpoint:/var/lib/kubelet/pods/f14e8781-bf82-4810-9909-c126ccdc1ce1/volumes/kubernetes.io~projected/kube-api-access-v7xzh major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fa9c5a91-d480-4d7b-9044-be6437a76a02/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/fa9c5a91-d480-4d7b-9044-be6437a76a02/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fa9c5a91-d480-4d7b-9044-be6437a76a02/volumes/kubernetes.io~projected/kube-api-access-cctkp:{mountpoint:/var/lib/kubelet/pods/fa9c5a91-d480-4d7b-9044-be6437a76a02/volumes/kubernetes.io~projected/kube-api-access-cctkp major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fa9c5a91-d480-4d7b-9044-be6437a76a02/volumes/kubernetes.io~projected/registry-tls:{mountpoint:/var/lib/kubelet/pods/fa9c5a91-d480-4d7b-9044-be6437a76a02/volumes/kubernetes.io~projected/registry-tls major:0 minor:339 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fa9c5a91-d480-4d7b-9044-be6437a76a02/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/fa9c5a91-d480-4d7b-9044-be6437a76a02/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fa9c5a91-d480-4d7b-9044-be6437a76a02/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/fa9c5a91-d480-4d7b-9044-be6437a76a02/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:202 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/023f21c06900f798d82af6582cf3c9158c0e509516e8d088b1d1acc2cbe369ab/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/74b56096e9820f1639e7adcee28fa24bccc709336432628b19a5018f63ba7681/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/bd719d9c0d3c4c53a6c26bb701477e7c3155d39ed688106ed0861bf157e621b2/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/2935b69e04bbbc90d9b0cbdae06c54fe1c737ed2d99d11b3de4b8167bb56d97a/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-111:{mountpoint:/var/lib/containers/storage/overlay/6c96cbdbf785592447136057e9eaa567491bba2c5b17d00bbe2fcb9997a95286/merged major:0 minor:111 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/b43a5b8f08d4ff6aca404ded278ff9fa09980f583bdb37c6e3e9c92dbcacbfa5/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/8396877242efb175ef5a281860daafc7ed9041bea71d657c3012f549fee7722f/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/6a1f027bbeb72236670fd9958cc26e1e4c9d997607458517611450735bb0a034/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/631097f8f5c8835364f1a5171976a782bc39969b6727820989e43c7a6e45b21d/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/a4e1f3e637b9856549257d18647d62080324f5be065920d336874c5cdc83f6d3/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/2118f66b917e466d3d4aec7b663308d87c16939b44b2b6b9be1008e5755808f8/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-147:{mountpoint:/var/lib/containers/storage/overlay/baab878b5351aa42c2f09c60d8a373fbd9e898244e44ce4dd356b45dbba3328b/merged major:0 minor:147 fsType:overlay blockSize:0} overlay_0-149:{mountpoint:/var/lib/containers/storage/overlay/e6d90b0a7cbf039e22845ffeeb87294de8dbbdaceb3afb0934daee7ad138d594/merged major:0 minor:149 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/86d99338f247ef6d42e31ec8341f814e93218d0ac4d453f09aa4ce1df391f9f4/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-158:{mountpoint:/var/lib/containers/storage/overlay/712cbe4175b6539d45e0a64fa5bfbbaabdeb157e503d17392fad1734563ae970/merged major:0 minor:158 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/724d5868040d331996d2c600c1e61353cd6095e559d1d144c41e1c3435a65188/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/a39d2471423a089466fca5a85c10675411bc8ba2dd648e9e344e5f67d4364668/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/daee36cb1d810347e864f79014c34e95a62c99aa0f25355add3ea985a801ea31/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/7ddaceea42a8d5db7c7768f234bf06fc910fc095d75ec6a8b7ac7e7b43a60b2d/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/adf3f86600f6288755e6b142454d57f8564669c1c1ef01ad9f9fa4f809f9d15d/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/c6d44419a8306a70e85e7d2754e71d64cf4c39898c1f8850e82f6f5886ab8e36/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/28f06ea71ba7746c36f83a61a86fe7580b0acec83921adcb921422284d1e1f1c/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/8b2868912aa63e618ba8109f8fa1356cc2ebc445fa400de56eb032642e61522d/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/eab2bf657ced39330115ade766514982a52ef488a116568d9665a195587123ab/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/61122aeafa89f0a58ed3bb2457b93fe93ebb9c89c595f049ad7b3577b00f0907/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/0c696861054bb43554fe88f388a97660936ce06b8b2b7a0a02610e3547fb697a/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/121a060e603b38a9c0534430de55ef576487cb6e18d5515f67808b46ade8cc6f/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/50e51163cd3df7a96c5d10c61b5e83b9725b51bfb42b637081b85eed21c2c6d7/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/41a000d703e608ba24596e22a1947367e8d2c46d65ed5402a05c94534ab01efe/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/53fd262cee3904fb8305ba1b3426212accb2279b2b1acd41deebb4cfc851366e/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-266:{mountpoint:/var/lib/containers/storage/overlay/0715b809d5aab76a9660d0ad1af07c5c4cb9c8de65a8ae0e3d6b5904ac2dc297/merged major:0 minor:266 fsType:overlay blockSize:0} overlay_0-271:{mountpoint:/var/lib/containers/storage/overlay/44ac929e60fe88980e94011f92b6a7bbad916f631c4104f8c4ae291fae13fa0f/merged major:0 minor:271 fsType:overlay blockSize:0} overlay_0-282:{mountpoint:/var/lib/containers/storage/overlay/c0b0bbbbfdbd6ee63d2ffccd627768fdebb44b340ba7e049939d4f3345cecf3a/merged major:0 minor:282 fsType:overlay blockSize:0} overlay_0-284:{mountpoint:/var/lib/containers/storage/overlay/a2d559f68983a7654b36de6360341d498e51ffc9178500cabf7c09a3fd8662f3/merged major:0 minor:284 fsType:overlay blockSize:0} overlay_0-286:{mountpoint:/var/lib/containers/storage/overlay/6eede0cec37bfb9fa398ec5cdeb6a9bf9b22fc044918539974a83d72ad162761/merged major:0 minor:286 fsType:overlay blockSize:0} overlay_0-288:{mountpoint:/var/lib/containers/storage/overlay/2688adc52b5817552d8e0ae28a7f168adf63d5d35ad120badf2c3eb59dda5cd0/merged major:0 minor:288 fsType:overlay blockSize:0} overlay_0-294:{mountpoint:/var/lib/containers/storage/overlay/ace011ff772a055fe830156c55601d124aac824d70328add5237abdbfd539c89/merged major:0 minor:294 fsType:overlay blockSize:0} overlay_0-296:{mountpoint:/var/lib/containers/storage/overlay/76a2c8f5346ad6debbd17ee96d15a98a727586d82f73cc4fe134252dad4da3c5/merged major:0 minor:296 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/76b9ea2c3c1e1f9c28723721e5850b976a786ded09f526738abaa1397227e7b0/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-320:{mountpoint:/var/lib/containers/storage/overlay/0e0c5809b85ad3c10ddc87d1865df7a846e141383522bbe22a3575a7fa145d57/merged major:0 minor:320 fsType:overlay blockSize:0} overlay_0-322:{mountpoint:/var/lib/containers/storage/overlay/afcc2c48aea7b120d7f8b3814624cc6aca535d62e0e683df8ee8cc569d729bdd/merged major:0 minor:322 fsType:overlay blockSize:0} overlay_0-332:{mountpoint:/var/lib/containers/storage/overlay/914c04e0bea3e3952ed40f1b56aa40c15f4ebe3702ac33d9ca0a7a4bb9a1db10/merged major:0 minor:332 fsType:overlay blockSize:0} overlay_0-334:{mountpoint:/var/lib/containers/storage/overlay/cc3e83a40b4fc43fab8a6ce7e8a8892f65a2515d8286b497b6ef5ce3bb503fed/merged major:0 minor:334 fsType:overlay blockSize:0} overlay_0-336:{mountpoint:/var/lib/containers/storage/overlay/3410727931ac94fa18b8b5f13fa6e9a736b2154c2edeeace236944e97237daa1/merged major:0 minor:336 fsType:overlay blockSize:0} overlay_0-344:{mountpoint:/var/lib/containers/storage/overlay/00b9a4adc56bac2c7e4a106475344cf746ed70d307654e7367f04eff1dbf977e/merged major:0 minor:344 fsType:overlay blockSize:0} overlay_0-346:{mountpoint:/var/lib/containers/storage/overlay/e5d6c8ec5eb4e2dc27870a084206b603cf0b6f6bd3519bf1b86a322563b964aa/merged major:0 minor:346 fsType:overlay blockSize:0} overlay_0-361:{mountpoint:/var/lib/containers/storage/overlay/32f2f75a15721de1b1ac0786985cf80ab128486f7670eefa1924b51cb23b743c/merged major:0 minor:361 fsType:overlay blockSize:0} overlay_0-363:{mountpoint:/var/lib/containers/storage/overlay/c2499d3ba65d942aaf54a58e28754cab883b99277c68e750cb27becec078bf0a/merged major:0 minor:363 fsType:overlay blockSize:0} overlay_0-364:{mountpoint:/var/lib/containers/storage/overlay/b34d50da82d72fc4539d24c347ff8ce4e0d7ace917157bd46d46880a509b245e/merged major:0 minor:364 fsType:overlay blockSize:0} overlay_0-366:{mountpoint:/var/lib/containers/storage/overlay/b2b1c50638d283c3d1ec825f57d72ac6c0bba41d967b499b21d52ada8a8c3cae/merged major:0 minor:366 fsType:overlay blockSize:0} overlay_0-372:{mountpoint:/var/lib/containers/storage/overlay/a46d5465d3c4fc9e1e7603754cce7a2599edde6975a1fba439a238be99744089/merged major:0 minor:372 fsType:overlay blockSize:0} overlay_0-382:{mountpoint:/var/lib/containers/storage/overlay/7488eb58bd1c118a063933fd765ce154c473d62e5b8d5bc70bab6b3d3a0896be/merged major:0 minor:382 fsType:overlay blockSize:0} overlay_0-385:{mountpoint:/var/lib/containers/storage/overlay/538eb46a3b6c14d9e5b6606c6f3a127a2a66d50e08fed0a4a61e75f9cc25bf09/merged major:0 minor:385 fsType:overlay blockSize:0} overlay_0-391:{mountpoint:/var/lib/containers/storage/overlay/4e35e18804ff4703333a6e6c64e54526322619f856162fd72365930663c5b7f9/merged major:0 minor:391 fsType:overlay blockSize:0} overlay_0-397:{mountpoint:/var/lib/containers/storage/overlay/6537227416ef104ce42b864d8fc4674733c93cfdf1b488832ae7c7606288325a/merged major:0 minor:397 fsType:overlay blockSize:0} overlay_0-409:{mountpoint:/var/lib/containers/storage/overlay/2be2f3fd281afde4ce03ac5c6466188ebaf1b53ccc55c34e0518e4447b7f18f5/merged major:0 minor:409 fsType:overlay blockSize:0} overlay_0-411:{mountpoint:/var/lib/containers/storage/overlay/7347f861b181cf175c86077a349b368438608d8182a821be3e2d18f4a7c0ceab/merged major:0 minor:411 fsType:overlay blockSize:0} overlay_0-413:{mountpoint:/var/lib/containers/storage/overlay/59f3615baa2e705149c85c40448c3465e88fc28d06837816a8266327c6ec0161/merged major:0 minor:413 fsType:overlay blockSize:0} overlay_0-419:{mountpoint:/var/lib/containers/storage/overlay/31db912611bf0117a246506dbf475d7d66d69ddd1aa9a3cb822e735820ea5dcd/merged major:0 minor:419 fsType:overlay blockSize:0} overlay_0-430:{mountpoint:/var/lib/containers/storage/overlay/45685967311b34f27c5193f441181b903100da0f942f77f2e151c838a0aa3221/merged major:0 minor:430 fsType:overlay blockSize:0} overlay_0-433:{mountpoint:/var/lib/containers/storage/overlay/81b713387b1735ca8c91c5dcf1f11fa8320e3410e4b740564e6b0231957b7b7a/merged major:0 minor:433 fsType:overlay blockSize:0} overlay_0-448:{mountpoint:/var/lib/containers/storage/overlay/a3924304ce13f53667f6a45b8816c2ef2d34acd384aab3198ea5fd616f3ba700/merged major:0 minor:448 fsType:overlay blockSize:0} overlay_0-450:{mountpoint:/var/lib/containers/storage/overlay/258934df64eaecc5c9cd5d969fa90f067c0fe174bf71d904346de49ad1aa747f/merged major:0 minor:450 fsType:overlay blockSize:0} overlay_0-453:{mountpoint:/var/lib/containers/storage/overlay/0fbebe6834bd0f51ae9b8b90a6a318a635daedb858a2c0ed6b9518435d8180ac/merged major:0 minor:453 fsType:overlay blockSize:0} overlay_0-460:{mountpoint:/var/lib/containers/storage/overlay/419c3641be60cbe58a15bc3d05fcfafc79025ddfc6ed6bff256cd03e07328c6d/merged major:0 minor:460 fsType:overlay blockSize:0} overlay_0-465:{mountpoint:/var/lib/containers/storage/overlay/c2c8ccaccf785a7cae77098074cb577c3291609e9ad64a548272d62961f8d543/merged major:0 minor:465 fsType:overlay blockSize:0} overlay_0-467:{mountpoint:/var/lib/containers/storage/overlay/dd601a3928df0467ad014dbc52febf9b35b623ca03f75dd20db63f6bfaf79571/merged major:0 minor:467 fsType:overlay blockSize:0} overlay_0-469:{mountpoint:/var/lib/containers/storage/overlay/800ab9da9c924eb466815de54d1efbaf4924a503f2604a14a43178fb5a9b78d5/merged major:0 minor:469 fsType:overlay blockSize:0} overlay_0-475:{mountpoint:/var/lib/containers/storage/overlay/43987320ce2dc12325e5f4b3da0c8d20e3da0d09bc7a276972af91dac8cec954/merged major:0 minor:475 fsType:overlay blockSize:0} overlay_0-481:{mountpoint:/var/lib/containers/storage/overlay/c395a2321f3c7f3d81050c9ba67d3df81e46347f42c1fd94feed86f77da9a084/merged major:0 minor:481 fsType:overlay blockSize:0} overlay_0-486:{mountpoint:/var/lib/containers/storage/overlay/c0960dd734bf681bd944407ab4327f9da8b431715d5b2fc2efb0f649a2ff0b4b/merged major:0 minor:486 fsType:overlay blockSize:0} overlay_0-492:{mountpoint:/var/lib/containers/storage/overlay/ff26872a54d3362e04829ea1ee447a41358d24ca6a51ff9c0fa184be94bafb27/merged major:0 minor:492 fsType:overlay blockSize:0} overlay_0-497:{mountpoint:/var/lib/containers/storage/overlay/4cbd99ca1e57fea804c35674ffec16b1d3deae2aa4fab951fce75027070b59a6/merged major:0 minor:497 fsType:overlay blockSize:0} overlay_0-502:{mountpoint:/var/lib/containers/storage/overlay/e08bb7ed3820817ecc71098a33a2e7ec9bc46a8751241223c384f1bea4c2dd6d/merged major:0 minor:502 fsType:overlay blockSize:0} overlay_0-63:{mountpoint:/var/lib/containers/storage/overlay/c2c90bf13b7bc58350b7c12fdf4ccef864b0c74e2d220de883c0e4fea502f8fb/merged major:0 minor:63 fsType:overlay blockSize:0} overlay_0-66:{mountpoint:/var/lib/containers/storage/overlay/1caad85d2971c5fec673f2dc3659c7c5d9e9d5d2c56ff7398e6d6084e78be33b/merged major:0 minor:66 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/9540a2b486c71af3892f6842c6fe2a38f648292deca0c6dc33a5200d207428c3/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/b846f9b63e372a2f8ec71abc772b1b6126ac5af29a983fb88712e43706f3370f/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/bc27f415b7e21d152712a1e78c7120ac2d55856edb8c43589ac806c172387192/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/e1d2f955daa11c796b20f975e44d7fd57d9d7e5fad4ecfd5aa888fe5a714ff13/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/74004d57c7596e3cddd448a2b66c0ef02886e0afc5fbf2f0e2af6a5eb7007604/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/6d8316d7657fc2d3bf5d7c5923350399777e6cee78946015e4735a80f059651c/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/c4f90e08e7cf44de586fd9ea33c41149101398e02701c117999f2607cbb4facd/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/e02ebb2e715ff773de8369b46184cc8110b4dabeac4131625f8c24b2795c996f/merged major:0 minor:96 fsType:overlay blockSize:0}] Nov 02 06:16:51.790438 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.789972 5916 manager.go:217] Machine: {Timestamp:2026-11-02 06:16:51.789485973 +0000 UTC m=+0.061306359 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100645 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2f38e4cdb73e9d854a14e852006bb4 SystemUUID:ec2f38e4-cdb7-3e9d-854a-14e852006bb4 BootID:0257ffa4-a85a-4582-93f8-e3183ee282db Filesystems:[{Device:/var/lib/kubelet/pods/87e797a0-1da6-41ca-af8d-7382afaf3987/volumes/kubernetes.io~projected/kube-api-access-l6vx8 DeviceMajor:0 DeviceMinor:59 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f0531b4d-c954-4830-85f6-fe876c76437a/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:45 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/91abcd1f2335c5276a0294d88a772c710c698e73c3a2b82f1a7042821d779a32/userdata/shm DeviceMajor:0 DeviceMinor:240 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/4ba9b2b6-4be9-4303-8eab-cd464c5a32c7/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:355 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-469 DeviceMajor:0 DeviceMinor:469 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-147 DeviceMajor:0 DeviceMinor:147 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/62d6ac9953134850fb004827c8a962446a4b2f6330e743b941096a9ab30409f5/userdata/shm DeviceMajor:0 DeviceMinor:246 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-334 DeviceMajor:0 DeviceMinor:334 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-382 DeviceMajor:0 DeviceMinor:382 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-391 DeviceMajor:0 DeviceMinor:391 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-448 DeviceMajor:0 DeviceMinor:448 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3500e682-2f15-45a2-92d6-bbc35dae3231/volumes/kubernetes.io~projected/kube-api-access-v8lqt DeviceMajor:0 DeviceMinor:53 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-149 DeviceMajor:0 DeviceMinor:149 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fa9c5a91-d480-4d7b-9044-be6437a76a02/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:205 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/860a45d95193ea55c54e30a2f0d13b46a725340587fe8a32c4c6a6be79374f1d/userdata/shm DeviceMajor:0 DeviceMinor:223 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8b491a46-9f53-4a4f-97ed-d8b32e3a03fa/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:403 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b5d368d976619eaa837c56d4e22406db6bf7495a6d9575c227809c8db3f208ac/userdata/shm DeviceMajor:0 DeviceMinor:60 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-63 DeviceMajor:0 DeviceMinor:63 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-433 DeviceMajor:0 DeviceMinor:433 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7434256f-45c3-47ff-9111-352114b0f6d8/volumes/kubernetes.io~projected/kube-api-access-l5q7t DeviceMajor:0 DeviceMinor:209 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/37982aa4-bdbd-40e8-a2dd-5094209f3c96/volumes/kubernetes.io~projected/kube-api-access-h7l5t DeviceMajor:0 DeviceMinor:232 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-364 DeviceMajor:0 DeviceMinor:364 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3fe2b451a165e20914129200c0e1ed9201cfbb89b1eb8d68742f2ca731715d93/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-284 DeviceMajor:0 DeviceMinor:284 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-465 DeviceMajor:0 DeviceMinor:465 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-475 DeviceMajor:0 DeviceMinor:475 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-158 DeviceMajor:0 DeviceMinor:158 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-296 DeviceMajor:0 DeviceMinor:296 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1309dfc627fd0eddb9d56bfeb55c303d58564eed884481f3f3119988a12fbe55/userdata/shm DeviceMajor:0 DeviceMinor:351 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8b4aca9cba52d27e69b569155b9d1855545be2e53e75210dc3ffe05d64dd23df/userdata/shm DeviceMajor:0 DeviceMinor:359 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-344 DeviceMajor:0 DeviceMinor:344 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cf8e8e65-8525-484b-adfe-f56517ccdc1b/volumes/kubernetes.io~projected/kube-api-access-vwqbt DeviceMajor:0 DeviceMinor:231 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4963902e93a3be86d04c977ac33a031fb4a55dbdfd7d021a6e3debdefa6f663a/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-332 DeviceMajor:0 DeviceMinor:332 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f0531b4d-c954-4830-85f6-fe876c76437a/volumes/kubernetes.io~projected/registry-tls DeviceMajor:0 DeviceMinor:338 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/0d2b53c4-9d39-47fc-9367-c9a8e7f5da81/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:220 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/0d2b53c4-9d39-47fc-9367-c9a8e7f5da81/volumes/kubernetes.io~projected/kube-api-access-8mplc DeviceMajor:0 DeviceMinor:229 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6f2af3023df9565bf072c1569930184b0aa5c7af28b5d5da6f6d32982f397446/userdata/shm DeviceMajor:0 DeviceMinor:318 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/60673025a4b89a0ee5f4cb8c3345979e787c7da063a0e2faa54ef8740d8f0781/userdata/shm DeviceMajor:0 DeviceMinor:455 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-492 DeviceMajor:0 DeviceMinor:492 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3419bd74-22aa-4841-87d2-c4340e01dae9/volumes/kubernetes.io~projected/kube-api-access-vz9mf DeviceMajor:0 DeviceMinor:56 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/52cfcf77d0a8a8269f6a1d6364aa5d81ce424af636a65e86e9902c6b1b9ee167/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/fa9c5a91-d480-4d7b-9044-be6437a76a02/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:203 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/fa9c5a91-d480-4d7b-9044-be6437a76a02/volumes/kubernetes.io~projected/kube-api-access-cctkp DeviceMajor:0 DeviceMinor:212 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ac6faf80-42b2-4c7a-8e55-abf6435b3d1e/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:324 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/49f1bd67-26ad-4fc9-b3d5-7bf7f6404aaa/volumes/kubernetes.io~projected/kube-api-access-v7jwb DeviceMajor:0 DeviceMinor:424 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/cf8e8e65-8525-484b-adfe-f56517ccdc1b/volumes/kubernetes.io~secret/samples-operator-tls DeviceMajor:0 DeviceMinor:452 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/45e5ce56-5cab-4546-8b31-02fc06cfa0cc/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:197 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/37982aa4-bdbd-40e8-a2dd-5094209f3c96/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:216 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-363 DeviceMajor:0 DeviceMinor:363 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5165263c016e50839257e4586fc7f9fc1b015d1999d655cc835b1d5e978df321/userdata/shm DeviceMajor:0 DeviceMinor:443 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-502 DeviceMajor:0 DeviceMinor:502 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7e9463781594333c81318e1426f180685a00493bdbfb6deb6f58e1e836dffb13/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-453 DeviceMajor:0 DeviceMinor:453 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5edecaa33e325dbc20c6c14bc24482a1098e38251bf6e78f228cc2b199f6fc9b/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-336 DeviceMajor:0 DeviceMinor:336 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/216fc5ad7a00312abf4a46ba90063240b9601015205ec36f60ddec22af761f9c/userdata/shm DeviceMajor:0 DeviceMinor:244 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-397 DeviceMajor:0 DeviceMinor:397 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3ced47b0-9cd0-4710-8f75-2a333437ed01/volumes/kubernetes.io~projected/kube-api-access-gnq6k DeviceMajor:0 DeviceMinor:436 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-460 DeviceMajor:0 DeviceMinor:460 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d4615a9e-da82-4f8b-8225-d60836b1ba80/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:50 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/d8d87317-3f41-46d4-a58e-3c7601e0d445/volumes/kubernetes.io~projected/kube-api-access-5kcrm DeviceMajor:0 DeviceMinor:54 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/3b6432a8-b0ce-4954-8e3d-8eb8782557eb/volumes/kubernetes.io~projected/kube-api-access-pvdp9 DeviceMajor:0 DeviceMinor:55 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-294 DeviceMajor:0 DeviceMinor:294 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d319bf628cd7bcbfc1dcc73a4d9d12b8dfb00fc85e674cf3e9929dca6b16bfc3/userdata/shm DeviceMajor:0 DeviceMinor:357 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fdbc98481353628068c730cc914f394bf2248182b7dc1d598e90142adad72ecf/userdata/shm DeviceMajor:0 DeviceMinor:407 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-481 DeviceMajor:0 DeviceMinor:481 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-111 DeviceMajor:0 DeviceMinor:111 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/121e38c0-6fa3-46aa-9191-5c6f91c8764c/volumes/kubernetes.io~projected/kube-api-access-fnw94 DeviceMajor:0 DeviceMinor:226 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-346 DeviceMajor:0 DeviceMinor:346 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3b6432a8-b0ce-4954-8e3d-8eb8782557eb/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:432 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/119ed3531b52ab511c43385dc22917968f984e5c99e56eb48bdcb85ee43db915/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e0370d945d4e54d879de7ca1d43138dcadfb5673f42e318252a19874123f5c6c/userdata/shm DeviceMajor:0 DeviceMinor:214 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/12b6cc1c-537d-4125-92ff-fe5f6927c53e/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:217 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a2ef8269-c442-42a8-8041-6c1b6aaf0746/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:47 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/f14e8781-bf82-4810-9909-c126ccdc1ce1/volumes/kubernetes.io~projected/kube-api-access-v7xzh DeviceMajor:0 DeviceMinor:57 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/fa9c5a91-d480-4d7b-9044-be6437a76a02/volumes/kubernetes.io~projected/registry-tls DeviceMajor:0 DeviceMinor:339 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-361 DeviceMajor:0 DeviceMinor:361 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b65bb247-83f7-4749-9aa1-9b2677dd1b3d/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:435 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fadf7f58c12e185657a35443e5641a2c59dc461cb89e0db18ce1be9b812892b7/userdata/shm DeviceMajor:0 DeviceMinor:445 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-467 DeviceMajor:0 DeviceMinor:467 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/84ef672c27542c8ee2cd9aa8315ffdac87f5adea3ca97472f1d9343ae269bc89/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/45e5ce56-5cab-4546-8b31-02fc06cfa0cc/volumes/kubernetes.io~projected/kube-api-access-xvpvr DeviceMajor:0 DeviceMinor:208 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/755856f4f662e5878f1383579803c0114371ba096156e09fb4df933877698348/userdata/shm DeviceMajor:0 DeviceMinor:221 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-486 DeviceMajor:0 DeviceMinor:486 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-266 DeviceMajor:0 DeviceMinor:266 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/81b393f2-42de-4a99-8114-1445de4d6dc8/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:48 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/81b393f2-42de-4a99-8114-1445de4d6dc8/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:49 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/3eec1082-df0b-420b-b4ff-5a1c0fab1d1d/volumes/kubernetes.io~projected/kube-api-access-k8vwr DeviceMajor:0 DeviceMinor:51 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/f0531b4d-c954-4830-85f6-fe876c76437a/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:204 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-322 DeviceMajor:0 DeviceMinor:322 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-450 DeviceMajor:0 DeviceMinor:450 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7e7bb32b8719598da92f32821b935863ee8427e701ef1c413fe40777eed58690/userdata/shm DeviceMajor:0 DeviceMinor:463 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c1ac0d913b943b018616c75971989a95da11db3c09c0128e4db983e8b0d0eaa3/userdata/shm DeviceMajor:0 DeviceMinor:62 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f0531b4d-c954-4830-85f6-fe876c76437a/volumes/kubernetes.io~projected/kube-api-access-m89g8 DeviceMajor:0 DeviceMinor:210 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/3fcb7621-86c8-448f-b4fa-b5bd76be7356/volumes/kubernetes.io~projected/kube-api-access-mwh5f DeviceMajor:0 DeviceMinor:211 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/cc33996a-0d4c-49c4-ba67-283bd6a2de14/volumes/kubernetes.io~projected/kube-api-access-rd8hq DeviceMajor:0 DeviceMinor:227 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/4ba9b2b6-4be9-4303-8eab-cd464c5a32c7/volumes/kubernetes.io~projected/kube-api-access-dv9xj DeviceMajor:0 DeviceMinor:230 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-497 DeviceMajor:0 DeviceMinor:497 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/53aa1e5a-c35f-4aa6-9f23-cc61ccd331fc/volumes/kubernetes.io~secret/networking-console-plugin-cert DeviceMajor:0 DeviceMinor:356 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-372 DeviceMajor:0 DeviceMinor:372 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/05a094dc530991421555eea671fb2cf05126ac45b8261be9db29c7b4f57ad03b/userdata/shm DeviceMajor:0 DeviceMinor:437 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/3fcb7621-86c8-448f-b4fa-b5bd76be7356/volumes/kubernetes.io~secret/cluster-monitoring-operator-tls DeviceMajor:0 DeviceMinor:447 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-66 DeviceMajor:0 DeviceMinor:66 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fa9c5a91-d480-4d7b-9044-be6437a76a02/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:202 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/17dee72b-2f47-4016-8ffe-3a89934e66ee/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:218 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-419 DeviceMajor:0 DeviceMinor:419 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8e8f2e60-de1b-4c8b-9285-56568c150d92/volumes/kubernetes.io~projected/kube-api-access-zrh7x DeviceMajor:0 DeviceMinor:213 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/12b6cc1c-537d-4125-92ff-fe5f6927c53e/volumes/kubernetes.io~projected/kube-api-access-bmsjq DeviceMajor:0 DeviceMinor:233 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-286 DeviceMajor:0 DeviceMinor:286 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ac6faf80-42b2-4c7a-8e55-abf6435b3d1e/volumes/kubernetes.io~projected/kube-api-access-pgkrw DeviceMajor:0 DeviceMinor:329 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/45e5ce56-5cab-4546-8b31-02fc06cfa0cc/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:348 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a471a58ea4a732d84e89fb09f437def55b26113096801bc4a4e213c7894fbe5b/userdata/shm DeviceMajor:0 DeviceMinor:349 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/cc33996a-0d4c-49c4-ba67-283bd6a2de14/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:219 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/a4319db2-eda9-47a0-b742-985c8bdd7ea2/volumes/kubernetes.io~projected/kube-api-access-qwdcw DeviceMajor:0 DeviceMinor:317 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/12a8615ab34e8cb0bbe17c3b60704970d0bcd860a9a74b7984b41986462505ea/userdata/shm DeviceMajor:0 DeviceMinor:353 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-385 DeviceMajor:0 DeviceMinor:385 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/418136c5b736666c736e9cbf8be032be97dc9918c1dd664aae798cd2baf3649b/userdata/shm DeviceMajor:0 DeviceMinor:428 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-430 DeviceMajor:0 DeviceMinor:430 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-271 DeviceMajor:0 DeviceMinor:271 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1e1cdd6d7235fef42913f68e6790d9df4741bd50deb1e6f2c87c1978b3d17f19/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/a2ef8269-c442-42a8-8041-6c1b6aaf0746/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/dee3b502-2562-4b21-bc46-2031580c4b0a/volumes/kubernetes.io~projected/kube-api-access-txffk DeviceMajor:0 DeviceMinor:207 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/20cb7690-c976-471c-b459-a9ab15cc51bf/volumes/kubernetes.io~projected/kube-api-access-5989k DeviceMajor:0 DeviceMinor:225 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/8b491a46-9f53-4a4f-97ed-d8b32e3a03fa/volumes/kubernetes.io~projected/kube-api-access-dpt4c DeviceMajor:0 DeviceMinor:300 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-411 DeviceMajor:0 DeviceMinor:411 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/45e5ce56-5cab-4546-8b31-02fc06cfa0cc/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:201 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ab0ebd2db011ef10e8f93786016b5cc45593e7dc0f17ead3905307dd1d9fcd2d/userdata/shm DeviceMajor:0 DeviceMinor:236 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-320 DeviceMajor:0 DeviceMinor:320 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/23685e5c401a72636d0a3f2994a72afdb729f870e680b5dbd7d42a469c46b89e/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c53b73150c37aad5859309630eab56b7a553276fb27542d80ef212490baaeb30/userdata/shm DeviceMajor:0 DeviceMinor:238 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-282 DeviceMajor:0 DeviceMinor:282 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-288 DeviceMajor:0 DeviceMinor:288 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4b603936d475f6f974e687dc0306c917a65be18ac6c747bade66b4f9b09d5a3e/userdata/shm DeviceMajor:0 DeviceMinor:330 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-366 DeviceMajor:0 DeviceMinor:366 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/81b393f2-42de-4a99-8114-1445de4d6dc8/volumes/kubernetes.io~projected/kube-api-access-hrq9j DeviceMajor:0 DeviceMinor:58 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/17dee72b-2f47-4016-8ffe-3a89934e66ee/volumes/kubernetes.io~projected/kube-api-access-9tz6t DeviceMajor:0 DeviceMinor:228 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d1b287d3147a3d6fbc93b84f0c5883cfd49ac31f846e5d01f59df87df61be411/userdata/shm DeviceMajor:0 DeviceMinor:234 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-409 DeviceMajor:0 DeviceMinor:409 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a2ef8269-c442-42a8-8041-6c1b6aaf0746/volumes/kubernetes.io~projected/kube-api-access-qzxsj DeviceMajor:0 DeviceMinor:52 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f0531b4d-c954-4830-85f6-fe876c76437a/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:206 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-413 DeviceMajor:0 DeviceMinor:413 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:05a094dc5309914 MacAddress:d6:02:32:46:ca:22 Speed:10000 Mtu:8901} {Name:12a8615ab34e8cb MacAddress:b2:f2:7f:98:40:29 Speed:10000 Mtu:8901} {Name:1309dfc627fd0ed MacAddress:a2:39:bf:77:8e:05 Speed:10000 Mtu:8901} {Name:216fc5ad7a00312 MacAddress:a6:aa:fb:50:70:96 Speed:10000 Mtu:8901} {Name:418136c5b736666 MacAddress:12:dc:db:fb:29:3f Speed:10000 Mtu:8901} {Name:4963902e93a3be8 MacAddress:f2:c4:13:a9:2e:8a Speed:10000 Mtu:8901} {Name:4b603936d475f6f MacAddress:6e:7e:d4:fd:77:c2 Speed:10000 Mtu:8901} {Name:5165263c016e508 MacAddress:36:d0:7c:b5:46:f1 Speed:10000 Mtu:8901} {Name:60673025a4b89a0 MacAddress:9e:0c:70:93:23:61 Speed:10000 Mtu:8901} {Name:62d6ac995313485 MacAddress:2e:b4:4e:0e:cb:8a Speed:10000 Mtu:8901} {Name:6f2af3023df9565 MacAddress:6e:dd:31:e7:24:b9 Speed:10000 Mtu:8901} {Name:755856f4f662e58 MacAddress:0e:eb:86:78:6d:7a Speed:10000 Mtu:8901} {Name:7e7bb32b8719598 MacAddress:3e:32:0c:da:9b:a6 Speed:10000 Mtu:8901} {Name:860a45d95193ea5 MacAddress:be:bd:ac:5e:e2:c6 Speed:10000 Mtu:8901} {Name:8b4aca9cba52d27 MacAddress:b2:5a:50:60:bc:0e Speed:10000 Mtu:8901} {Name:91abcd1f2335c52 MacAddress:1a:8d:d6:68:be:0f Speed:10000 Mtu:8901} {Name:a471a58ea4a732d MacAddress:c2:84:14:1f:91:f2 Speed:10000 Mtu:8901} {Name:ab0ebd2db011ef1 MacAddress:ca:9f:8d:d7:1d:b7 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:06:ff:ca:67:ab:dd Speed:0 Mtu:9001} {Name:br-int MacAddress:9a:de:50:a0:58:c8 Speed:0 Mtu:8901} {Name:c53b73150c37aad MacAddress:06:59:dc:9a:a4:a8 Speed:10000 Mtu:8901} {Name:d1b287d3147a3d6 MacAddress:06:b9:57:31:bc:15 Speed:10000 Mtu:8901} {Name:d319bf628cd7bcb MacAddress:76:dd:26:7d:ca:ac Speed:10000 Mtu:8901} {Name:e0370d945d4e54d MacAddress:da:0c:f2:b8:24:76 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:06:ff:ca:67:ab:dd Speed:0 Mtu:9001} {Name:fadf7f58c12e185 MacAddress:9a:ec:9f:d5:c7:ad Speed:10000 Mtu:8901} {Name:fdbc98481353628 MacAddress:6e:d5:ed:11:60:10 Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:4a:57:ad:a4:79:a3 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:7a:78:2b:b9:3d:6b Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 02 06:16:51.790698 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.790442 5916 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 02 06:16:51.790698 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.790491 5916 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 02 06:16:51.790832 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.790808 5916 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 02 06:16:51.790984 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.790833 5916 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-93.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 02 06:16:51.791028 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.790986 5916 topology_manager.go:143] "Creating topology manager with none policy" Nov 02 06:16:51.791028 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.790994 5916 container_manager_linux.go:311] "Creating device plugin manager" Nov 02 06:16:51.791028 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.791004 5916 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 06:16:51.791028 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.791025 5916 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 06:16:51.791137 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.791129 5916 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 02 06:16:51.791309 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.791299 5916 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 02 06:16:51.791422 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.791413 5916 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 02 06:16:51.791459 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.791449 5916 kubelet.go:493] "Attempting to sync node with API server" Nov 02 06:16:51.791487 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.791460 5916 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 02 06:16:51.791487 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.791470 5916 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 02 06:16:51.791487 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.791480 5916 kubelet.go:405] "Adding apiserver pod source" Nov 02 06:16:51.791585 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.791488 5916 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 02 06:16:51.792184 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.792171 5916 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 02 06:16:51.792184 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.792184 5916 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 06:16:51.792724 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.792702 5916 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 02 06:16:51.796367 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.795733 5916 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 02 06:16:51.796367 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.795755 5916 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 06:16:51.796367 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.795890 5916 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Nov 02 06:16:51.796367 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.796216 5916 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 02 06:16:51.796367 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.796260 5916 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 02 06:16:51.796676 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.796408 5916 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 02 06:16:51.796676 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.796456 5916 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 02 06:16:51.796676 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.796663 5916 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 02 06:16:51.796676 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.796671 5916 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 02 06:16:51.796676 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.796677 5916 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 02 06:16:51.796896 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.796684 5916 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 02 06:16:51.796896 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.796690 5916 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 02 06:16:51.796896 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.796695 5916 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 02 06:16:51.796896 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.796703 5916 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 02 06:16:51.796896 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.796708 5916 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 02 06:16:51.796896 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.796718 5916 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 02 06:16:51.796896 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.796729 5916 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 02 06:16:51.796896 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.796753 5916 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 02 06:16:51.796896 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.796761 5916 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 02 06:16:51.800359 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.800346 5916 server.go:1263] "Started kubelet" Nov 02 06:16:51.800446 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.800416 5916 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 02 06:16:51.800862 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.800465 5916 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 02 06:16:51.800862 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.800560 5916 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 02 06:16:51.801024 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.801004 5916 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 02 06:16:51.800930 ip-10-0-130-93 systemd[1]: Started Kubernetes Kubelet. Nov 02 06:16:51.801778 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.801762 5916 server.go:317] "Adding debug handlers to kubelet server" Nov 02 06:16:51.805843 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.805821 5916 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 02 06:16:51.809956 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.807190 5916 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 02 06:16:51.809956 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:16:51.807232 5916 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-93.us-west-2.compute.internal\" not found" Nov 02 06:16:51.809956 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.807257 5916 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 06:10:39 +0000 UTC" deadline="2028-05-14 13:41:36.658123635 +0000 UTC" Nov 02 06:16:51.809956 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.807291 5916 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13423h24m44.850836287s" Nov 02 06:16:51.809956 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.807299 5916 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 02 06:16:51.809956 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.807307 5916 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 02 06:16:51.809956 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.807399 5916 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 02 06:16:51.809956 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.809426 5916 factory.go:55] Registering systemd factory Nov 02 06:16:51.809956 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.809443 5916 factory.go:223] Registration of the systemd container factory successfully Nov 02 06:16:51.809956 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.809757 5916 factory.go:153] Registering CRI-O factory Nov 02 06:16:51.809956 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.809783 5916 factory.go:223] Registration of the crio container factory successfully Nov 02 06:16:51.809956 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.809851 5916 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 02 06:16:51.809956 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.809872 5916 factory.go:103] Registering Raw factory Nov 02 06:16:51.809956 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.809889 5916 manager.go:1199] Started watching for new ooms in manager Nov 02 06:16:51.812094 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.812044 5916 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3b6432a8-b0ce-4954-8e3d-8eb8782557eb" volumeName="kubernetes.io/secret/3b6432a8-b0ce-4954-8e3d-8eb8782557eb-metrics-certs" seLinuxMountContext="" Nov 02 06:16:51.812186 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.812094 5916 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="45e5ce56-5cab-4546-8b31-02fc06cfa0cc" volumeName="kubernetes.io/projected/45e5ce56-5cab-4546-8b31-02fc06cfa0cc-kube-api-access-xvpvr" seLinuxMountContext="" Nov 02 06:16:51.812186 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.812110 5916 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="53aa1e5a-c35f-4aa6-9f23-cc61ccd331fc" volumeName="kubernetes.io/configmap/53aa1e5a-c35f-4aa6-9f23-cc61ccd331fc-nginx-conf" seLinuxMountContext="" Nov 02 06:16:51.812186 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.812123 5916 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="81b393f2-42de-4a99-8114-1445de4d6dc8" volumeName="kubernetes.io/empty-dir/81b393f2-42de-4a99-8114-1445de4d6dc8-tmp" seLinuxMountContext="" Nov 02 06:16:51.812186 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.812135 5916 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a2ef8269-c442-42a8-8041-6c1b6aaf0746" volumeName="kubernetes.io/secret/a2ef8269-c442-42a8-8041-6c1b6aaf0746-ovn-node-metrics-cert" seLinuxMountContext="" Nov 02 06:16:51.812186 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.812146 5916 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a4319db2-eda9-47a0-b742-985c8bdd7ea2" volumeName="kubernetes.io/projected/a4319db2-eda9-47a0-b742-985c8bdd7ea2-kube-api-access-qwdcw" seLinuxMountContext="" Nov 02 06:16:51.812186 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.812157 5916 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ac6faf80-42b2-4c7a-8e55-abf6435b3d1e" volumeName="kubernetes.io/secret/ac6faf80-42b2-4c7a-8e55-abf6435b3d1e-signing-key" seLinuxMountContext="" Nov 02 06:16:51.812186 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.812168 5916 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="37982aa4-bdbd-40e8-a2dd-5094209f3c96" volumeName="kubernetes.io/projected/37982aa4-bdbd-40e8-a2dd-5094209f3c96-kube-api-access-h7l5t" seLinuxMountContext="" Nov 02 06:16:51.812186 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.812181 5916 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a2ef8269-c442-42a8-8041-6c1b6aaf0746" volumeName="kubernetes.io/configmap/a2ef8269-c442-42a8-8041-6c1b6aaf0746-env-overrides" seLinuxMountContext="" Nov 02 06:16:51.812471 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.812193 5916 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a2ef8269-c442-42a8-8041-6c1b6aaf0746" volumeName="kubernetes.io/projected/a2ef8269-c442-42a8-8041-6c1b6aaf0746-kube-api-access-qzxsj" seLinuxMountContext="" Nov 02 06:16:51.812471 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.812204 5916 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a4319db2-eda9-47a0-b742-985c8bdd7ea2" volumeName="kubernetes.io/configmap/a4319db2-eda9-47a0-b742-985c8bdd7ea2-service-ca-bundle" seLinuxMountContext="" Nov 02 06:16:51.812471 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.812224 5916 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fa9c5a91-d480-4d7b-9044-be6437a76a02" volumeName="kubernetes.io/projected/fa9c5a91-d480-4d7b-9044-be6437a76a02-bound-sa-token" seLinuxMountContext="" Nov 02 06:16:51.812471 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.812236 5916 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4ba9b2b6-4be9-4303-8eab-cd464c5a32c7" volumeName="kubernetes.io/projected/4ba9b2b6-4be9-4303-8eab-cd464c5a32c7-kube-api-access-dv9xj" seLinuxMountContext="" Nov 02 06:16:51.812471 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.812255 5916 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="53aa1e5a-c35f-4aa6-9f23-cc61ccd331fc" volumeName="kubernetes.io/secret/53aa1e5a-c35f-4aa6-9f23-cc61ccd331fc-networking-console-plugin-cert" seLinuxMountContext="" Nov 02 06:16:51.812471 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.812269 5916 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a2ef8269-c442-42a8-8041-6c1b6aaf0746" volumeName="kubernetes.io/configmap/a2ef8269-c442-42a8-8041-6c1b6aaf0746-ovnkube-script-lib" seLinuxMountContext="" Nov 02 06:16:51.812471 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.812282 5916 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f0531b4d-c954-4830-85f6-fe876c76437a" volumeName="kubernetes.io/projected/f0531b4d-c954-4830-85f6-fe876c76437a-kube-api-access-m89g8" seLinuxMountContext="" Nov 02 06:16:51.812471 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.812294 5916 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f0531b4d-c954-4830-85f6-fe876c76437a" volumeName="kubernetes.io/secret/f0531b4d-c954-4830-85f6-fe876c76437a-installation-pull-secrets" seLinuxMountContext="" Nov 02 06:16:51.812471 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.812306 5916 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3b6432a8-b0ce-4954-8e3d-8eb8782557eb" volumeName="kubernetes.io/projected/3b6432a8-b0ce-4954-8e3d-8eb8782557eb-kube-api-access-pvdp9" seLinuxMountContext="" Nov 02 06:16:51.812471 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.812317 5916 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3eec1082-df0b-420b-b4ff-5a1c0fab1d1d" volumeName="kubernetes.io/projected/3eec1082-df0b-420b-b4ff-5a1c0fab1d1d-kube-api-access-k8vwr" seLinuxMountContext="" Nov 02 06:16:51.812471 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.812329 5916 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3fcb7621-86c8-448f-b4fa-b5bd76be7356" volumeName="kubernetes.io/secret/3fcb7621-86c8-448f-b4fa-b5bd76be7356-cluster-monitoring-operator-tls" seLinuxMountContext="" Nov 02 06:16:51.812471 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.812341 5916 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="12b6cc1c-537d-4125-92ff-fe5f6927c53e" volumeName="kubernetes.io/configmap/12b6cc1c-537d-4125-92ff-fe5f6927c53e-config" seLinuxMountContext="" Nov 02 06:16:51.812471 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.812357 5916 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="37982aa4-bdbd-40e8-a2dd-5094209f3c96" volumeName="kubernetes.io/configmap/37982aa4-bdbd-40e8-a2dd-5094209f3c96-service-ca-bundle" seLinuxMountContext="" Nov 02 06:16:51.812471 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.812371 5916 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="37982aa4-bdbd-40e8-a2dd-5094209f3c96" volumeName="kubernetes.io/empty-dir/37982aa4-bdbd-40e8-a2dd-5094209f3c96-tmp" seLinuxMountContext="" Nov 02 06:16:51.812471 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.812385 5916 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="49f1bd67-26ad-4fc9-b3d5-7bf7f6404aaa" volumeName="kubernetes.io/projected/49f1bd67-26ad-4fc9-b3d5-7bf7f6404aaa-kube-api-access-v7jwb" seLinuxMountContext="" Nov 02 06:16:51.812471 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.812398 5916 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8e8f2e60-de1b-4c8b-9285-56568c150d92" volumeName="kubernetes.io/projected/8e8f2e60-de1b-4c8b-9285-56568c150d92-kube-api-access-zrh7x" seLinuxMountContext="" Nov 02 06:16:51.812471 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.812420 5916 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="17dee72b-2f47-4016-8ffe-3a89934e66ee" volumeName="kubernetes.io/secret/17dee72b-2f47-4016-8ffe-3a89934e66ee-serving-cert" seLinuxMountContext="" Nov 02 06:16:51.812471 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.812433 5916 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8b491a46-9f53-4a4f-97ed-d8b32e3a03fa" volumeName="kubernetes.io/empty-dir/8b491a46-9f53-4a4f-97ed-d8b32e3a03fa-data-volume" seLinuxMountContext="" Nov 02 06:16:51.812471 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.812446 5916 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ac6faf80-42b2-4c7a-8e55-abf6435b3d1e" volumeName="kubernetes.io/configmap/ac6faf80-42b2-4c7a-8e55-abf6435b3d1e-signing-cabundle" seLinuxMountContext="" Nov 02 06:16:51.812471 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.812462 5916 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b65bb247-83f7-4749-9aa1-9b2677dd1b3d" volumeName="kubernetes.io/secret/b65bb247-83f7-4749-9aa1-9b2677dd1b3d-original-pull-secret" seLinuxMountContext="" Nov 02 06:16:51.812471 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.812473 5916 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f14e8781-bf82-4810-9909-c126ccdc1ce1" volumeName="kubernetes.io/configmap/f14e8781-bf82-4810-9909-c126ccdc1ce1-cni-sysctl-allowlist" seLinuxMountContext="" Nov 02 06:16:51.816803 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.812484 5916 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fa9c5a91-d480-4d7b-9044-be6437a76a02" volumeName="kubernetes.io/configmap/fa9c5a91-d480-4d7b-9044-be6437a76a02-registry-certificates" seLinuxMountContext="" Nov 02 06:16:51.816803 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.812495 5916 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fa9c5a91-d480-4d7b-9044-be6437a76a02" volumeName="kubernetes.io/projected/fa9c5a91-d480-4d7b-9044-be6437a76a02-kube-api-access-cctkp" seLinuxMountContext="" Nov 02 06:16:51.816803 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.812519 5916 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fa9c5a91-d480-4d7b-9044-be6437a76a02" volumeName="kubernetes.io/projected/fa9c5a91-d480-4d7b-9044-be6437a76a02-registry-tls" seLinuxMountContext="" Nov 02 06:16:51.816803 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.812531 5916 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0d2b53c4-9d39-47fc-9367-c9a8e7f5da81" volumeName="kubernetes.io/projected/0d2b53c4-9d39-47fc-9367-c9a8e7f5da81-kube-api-access-8mplc" seLinuxMountContext="" Nov 02 06:16:51.816803 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.812543 5916 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="12b6cc1c-537d-4125-92ff-fe5f6927c53e" volumeName="kubernetes.io/secret/12b6cc1c-537d-4125-92ff-fe5f6927c53e-serving-cert" seLinuxMountContext="" Nov 02 06:16:51.816803 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.812554 5916 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="37982aa4-bdbd-40e8-a2dd-5094209f3c96" volumeName="kubernetes.io/configmap/37982aa4-bdbd-40e8-a2dd-5094209f3c96-trusted-ca-bundle" seLinuxMountContext="" Nov 02 06:16:51.816803 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.812566 5916 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3fcb7621-86c8-448f-b4fa-b5bd76be7356" volumeName="kubernetes.io/configmap/3fcb7621-86c8-448f-b4fa-b5bd76be7356-telemetry-config" seLinuxMountContext="" Nov 02 06:16:51.816803 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.812577 5916 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a4319db2-eda9-47a0-b742-985c8bdd7ea2" volumeName="kubernetes.io/empty-dir/a4319db2-eda9-47a0-b742-985c8bdd7ea2-archives-path" seLinuxMountContext="" Nov 02 06:16:51.816803 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.812588 5916 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f0531b4d-c954-4830-85f6-fe876c76437a" volumeName="kubernetes.io/configmap/f0531b4d-c954-4830-85f6-fe876c76437a-registry-certificates" seLinuxMountContext="" Nov 02 06:16:51.816803 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.812601 5916 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8b491a46-9f53-4a4f-97ed-d8b32e3a03fa" volumeName="kubernetes.io/configmap/8b491a46-9f53-4a4f-97ed-d8b32e3a03fa-kube-rbac-proxy-cm" seLinuxMountContext="" Nov 02 06:16:51.816803 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.812612 5916 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8b491a46-9f53-4a4f-97ed-d8b32e3a03fa" volumeName="kubernetes.io/secret/8b491a46-9f53-4a4f-97ed-d8b32e3a03fa-insights-runtime-extractor-tls" seLinuxMountContext="" Nov 02 06:16:51.816803 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.812624 5916 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ac6faf80-42b2-4c7a-8e55-abf6435b3d1e" volumeName="kubernetes.io/projected/ac6faf80-42b2-4c7a-8e55-abf6435b3d1e-kube-api-access-pgkrw" seLinuxMountContext="" Nov 02 06:16:51.816803 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.812637 5916 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cf8e8e65-8525-484b-adfe-f56517ccdc1b" volumeName="kubernetes.io/secret/cf8e8e65-8525-484b-adfe-f56517ccdc1b-samples-operator-tls" seLinuxMountContext="" Nov 02 06:16:51.816803 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.812649 5916 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dee3b502-2562-4b21-bc46-2031580c4b0a" volumeName="kubernetes.io/projected/dee3b502-2562-4b21-bc46-2031580c4b0a-kube-api-access-txffk" seLinuxMountContext="" Nov 02 06:16:51.816803 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.812661 5916 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="45e5ce56-5cab-4546-8b31-02fc06cfa0cc" volumeName="kubernetes.io/secret/45e5ce56-5cab-4546-8b31-02fc06cfa0cc-stats-auth" seLinuxMountContext="" Nov 02 06:16:51.816803 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.812674 5916 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7434256f-45c3-47ff-9111-352114b0f6d8" volumeName="kubernetes.io/projected/7434256f-45c3-47ff-9111-352114b0f6d8-kube-api-access-l5q7t" seLinuxMountContext="" Nov 02 06:16:51.816803 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.812686 5916 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cf8e8e65-8525-484b-adfe-f56517ccdc1b" volumeName="kubernetes.io/projected/cf8e8e65-8525-484b-adfe-f56517ccdc1b-kube-api-access-vwqbt" seLinuxMountContext="" Nov 02 06:16:51.816803 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.812698 5916 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d8d87317-3f41-46d4-a58e-3c7601e0d445" volumeName="kubernetes.io/projected/d8d87317-3f41-46d4-a58e-3c7601e0d445-kube-api-access-5kcrm" seLinuxMountContext="" Nov 02 06:16:51.816803 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.812710 5916 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f0531b4d-c954-4830-85f6-fe876c76437a" volumeName="kubernetes.io/secret/f0531b4d-c954-4830-85f6-fe876c76437a-image-registry-private-configuration" seLinuxMountContext="" Nov 02 06:16:51.816803 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.812722 5916 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f14e8781-bf82-4810-9909-c126ccdc1ce1" volumeName="kubernetes.io/configmap/f14e8781-bf82-4810-9909-c126ccdc1ce1-cni-binary-copy" seLinuxMountContext="" Nov 02 06:16:51.816803 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.812733 5916 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fa9c5a91-d480-4d7b-9044-be6437a76a02" volumeName="kubernetes.io/secret/fa9c5a91-d480-4d7b-9044-be6437a76a02-installation-pull-secrets" seLinuxMountContext="" Nov 02 06:16:51.816803 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.812745 5916 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="37982aa4-bdbd-40e8-a2dd-5094209f3c96" volumeName="kubernetes.io/empty-dir/37982aa4-bdbd-40e8-a2dd-5094209f3c96-snapshots" seLinuxMountContext="" Nov 02 06:16:51.816803 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.812757 5916 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3eec1082-df0b-420b-b4ff-5a1c0fab1d1d" volumeName="kubernetes.io/configmap/3eec1082-df0b-420b-b4ff-5a1c0fab1d1d-serviceca" seLinuxMountContext="" Nov 02 06:16:51.816803 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.812769 5916 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d4615a9e-da82-4f8b-8225-d60836b1ba80" volumeName="kubernetes.io/configmap/d4615a9e-da82-4f8b-8225-d60836b1ba80-konnectivity-ca" seLinuxMountContext="" Nov 02 06:16:51.816803 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.812781 5916 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f0531b4d-c954-4830-85f6-fe876c76437a" volumeName="kubernetes.io/configmap/f0531b4d-c954-4830-85f6-fe876c76437a-trusted-ca" seLinuxMountContext="" Nov 02 06:16:51.816803 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.812793 5916 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4ba9b2b6-4be9-4303-8eab-cd464c5a32c7" volumeName="kubernetes.io/empty-dir/4ba9b2b6-4be9-4303-8eab-cd464c5a32c7-tmp-dir" seLinuxMountContext="" Nov 02 06:16:51.816803 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.812814 5916 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="87e797a0-1da6-41ca-af8d-7382afaf3987" volumeName="kubernetes.io/configmap/87e797a0-1da6-41ca-af8d-7382afaf3987-multus-daemon-config" seLinuxMountContext="" Nov 02 06:16:51.816803 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.812828 5916 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cf8e8e65-8525-484b-adfe-f56517ccdc1b" volumeName="kubernetes.io/configmap/cf8e8e65-8525-484b-adfe-f56517ccdc1b-samples-operator-config" seLinuxMountContext="" Nov 02 06:16:51.816803 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.812838 5916 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f0531b4d-c954-4830-85f6-fe876c76437a" volumeName="kubernetes.io/projected/f0531b4d-c954-4830-85f6-fe876c76437a-registry-tls" seLinuxMountContext="" Nov 02 06:16:51.816803 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.812846 5916 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="17dee72b-2f47-4016-8ffe-3a89934e66ee" volumeName="kubernetes.io/projected/17dee72b-2f47-4016-8ffe-3a89934e66ee-kube-api-access-9tz6t" seLinuxMountContext="" Nov 02 06:16:51.816803 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.812853 5916 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="37982aa4-bdbd-40e8-a2dd-5094209f3c96" volumeName="kubernetes.io/secret/37982aa4-bdbd-40e8-a2dd-5094209f3c96-serving-cert" seLinuxMountContext="" Nov 02 06:16:51.816803 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.812860 5916 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="45e5ce56-5cab-4546-8b31-02fc06cfa0cc" volumeName="kubernetes.io/secret/45e5ce56-5cab-4546-8b31-02fc06cfa0cc-default-certificate" seLinuxMountContext="" Nov 02 06:16:51.816803 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.812868 5916 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4ba9b2b6-4be9-4303-8eab-cd464c5a32c7" volumeName="kubernetes.io/configmap/4ba9b2b6-4be9-4303-8eab-cd464c5a32c7-config-volume" seLinuxMountContext="" Nov 02 06:16:51.816803 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.812879 5916 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="87e797a0-1da6-41ca-af8d-7382afaf3987" volumeName="kubernetes.io/projected/87e797a0-1da6-41ca-af8d-7382afaf3987-kube-api-access-l6vx8" seLinuxMountContext="" Nov 02 06:16:51.816803 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.812887 5916 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d8d87317-3f41-46d4-a58e-3c7601e0d445" volumeName="kubernetes.io/empty-dir/d8d87317-3f41-46d4-a58e-3c7601e0d445-tmp-dir" seLinuxMountContext="" Nov 02 06:16:51.816803 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.812894 5916 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fa9c5a91-d480-4d7b-9044-be6437a76a02" volumeName="kubernetes.io/configmap/fa9c5a91-d480-4d7b-9044-be6437a76a02-trusted-ca" seLinuxMountContext="" Nov 02 06:16:51.816803 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.812923 5916 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="12b6cc1c-537d-4125-92ff-fe5f6927c53e" volumeName="kubernetes.io/projected/12b6cc1c-537d-4125-92ff-fe5f6927c53e-kube-api-access-bmsjq" seLinuxMountContext="" Nov 02 06:16:51.816803 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.812937 5916 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="17dee72b-2f47-4016-8ffe-3a89934e66ee" volumeName="kubernetes.io/configmap/17dee72b-2f47-4016-8ffe-3a89934e66ee-config" seLinuxMountContext="" Nov 02 06:16:51.816803 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.812945 5916 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="81b393f2-42de-4a99-8114-1445de4d6dc8" volumeName="kubernetes.io/empty-dir/81b393f2-42de-4a99-8114-1445de4d6dc8-etc-tuned" seLinuxMountContext="" Nov 02 06:16:51.816803 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.812954 5916 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8b491a46-9f53-4a4f-97ed-d8b32e3a03fa" volumeName="kubernetes.io/projected/8b491a46-9f53-4a4f-97ed-d8b32e3a03fa-kube-api-access-dpt4c" seLinuxMountContext="" Nov 02 06:16:51.816803 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.812961 5916 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a4319db2-eda9-47a0-b742-985c8bdd7ea2" volumeName="kubernetes.io/configmap/a4319db2-eda9-47a0-b742-985c8bdd7ea2-trusted-ca-bundle" seLinuxMountContext="" Nov 02 06:16:51.816803 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.812980 5916 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f14e8781-bf82-4810-9909-c126ccdc1ce1" volumeName="kubernetes.io/configmap/f14e8781-bf82-4810-9909-c126ccdc1ce1-whereabouts-flatfile-configmap" seLinuxMountContext="" Nov 02 06:16:51.816803 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.812992 5916 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f14e8781-bf82-4810-9909-c126ccdc1ce1" volumeName="kubernetes.io/projected/f14e8781-bf82-4810-9909-c126ccdc1ce1-kube-api-access-v7xzh" seLinuxMountContext="" Nov 02 06:16:51.816803 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.813003 5916 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="45e5ce56-5cab-4546-8b31-02fc06cfa0cc" volumeName="kubernetes.io/secret/45e5ce56-5cab-4546-8b31-02fc06cfa0cc-metrics-certs" seLinuxMountContext="" Nov 02 06:16:51.816803 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.813016 5916 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="81b393f2-42de-4a99-8114-1445de4d6dc8" volumeName="kubernetes.io/projected/81b393f2-42de-4a99-8114-1445de4d6dc8-kube-api-access-hrq9j" seLinuxMountContext="" Nov 02 06:16:51.816803 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.813030 5916 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="87e797a0-1da6-41ca-af8d-7382afaf3987" volumeName="kubernetes.io/configmap/87e797a0-1da6-41ca-af8d-7382afaf3987-cni-binary-copy" seLinuxMountContext="" Nov 02 06:16:51.816803 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.813041 5916 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="20cb7690-c976-471c-b459-a9ab15cc51bf" volumeName="kubernetes.io/projected/20cb7690-c976-471c-b459-a9ab15cc51bf-kube-api-access-5989k" seLinuxMountContext="" Nov 02 06:16:51.816803 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.813051 5916 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0d2b53c4-9d39-47fc-9367-c9a8e7f5da81" volumeName="kubernetes.io/secret/0d2b53c4-9d39-47fc-9367-c9a8e7f5da81-cert" seLinuxMountContext="" Nov 02 06:16:51.816803 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.813062 5916 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="17dee72b-2f47-4016-8ffe-3a89934e66ee" volumeName="kubernetes.io/configmap/17dee72b-2f47-4016-8ffe-3a89934e66ee-trusted-ca" seLinuxMountContext="" Nov 02 06:16:51.816803 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.813070 5916 manager.go:319] Starting recovery of all containers Nov 02 06:16:51.816803 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.816107 5916 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3ced47b0-9cd0-4710-8f75-2a333437ed01" volumeName="kubernetes.io/projected/3ced47b0-9cd0-4710-8f75-2a333437ed01-kube-api-access-gnq6k" seLinuxMountContext="" Nov 02 06:16:51.816803 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.816140 5916 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3fcb7621-86c8-448f-b4fa-b5bd76be7356" volumeName="kubernetes.io/projected/3fcb7621-86c8-448f-b4fa-b5bd76be7356-kube-api-access-mwh5f" seLinuxMountContext="" Nov 02 06:16:51.816803 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.816154 5916 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="45e5ce56-5cab-4546-8b31-02fc06cfa0cc" volumeName="kubernetes.io/configmap/45e5ce56-5cab-4546-8b31-02fc06cfa0cc-service-ca-bundle" seLinuxMountContext="" Nov 02 06:16:51.816803 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.816165 5916 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a2ef8269-c442-42a8-8041-6c1b6aaf0746" volumeName="kubernetes.io/configmap/a2ef8269-c442-42a8-8041-6c1b6aaf0746-ovnkube-config" seLinuxMountContext="" Nov 02 06:16:51.816803 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.816176 5916 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cc33996a-0d4c-49c4-ba67-283bd6a2de14" volumeName="kubernetes.io/projected/cc33996a-0d4c-49c4-ba67-283bd6a2de14-kube-api-access-rd8hq" seLinuxMountContext="" Nov 02 06:16:51.816803 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.816190 5916 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3419bd74-22aa-4841-87d2-c4340e01dae9" volumeName="kubernetes.io/configmap/3419bd74-22aa-4841-87d2-c4340e01dae9-iptables-alerter-script" seLinuxMountContext="" Nov 02 06:16:51.816803 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.816201 5916 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cc33996a-0d4c-49c4-ba67-283bd6a2de14" volumeName="kubernetes.io/secret/cc33996a-0d4c-49c4-ba67-283bd6a2de14-serving-cert" seLinuxMountContext="" Nov 02 06:16:51.816803 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.816212 5916 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f0531b4d-c954-4830-85f6-fe876c76437a" volumeName="kubernetes.io/empty-dir/f0531b4d-c954-4830-85f6-fe876c76437a-ca-trust-extracted" seLinuxMountContext="" Nov 02 06:16:51.816803 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.816224 5916 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f0531b4d-c954-4830-85f6-fe876c76437a" volumeName="kubernetes.io/projected/f0531b4d-c954-4830-85f6-fe876c76437a-bound-sa-token" seLinuxMountContext="" Nov 02 06:16:51.816803 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.816235 5916 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fa9c5a91-d480-4d7b-9044-be6437a76a02" volumeName="kubernetes.io/secret/fa9c5a91-d480-4d7b-9044-be6437a76a02-image-registry-private-configuration" seLinuxMountContext="" Nov 02 06:16:51.816803 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.816247 5916 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3419bd74-22aa-4841-87d2-c4340e01dae9" volumeName="kubernetes.io/projected/3419bd74-22aa-4841-87d2-c4340e01dae9-kube-api-access-vz9mf" seLinuxMountContext="" Nov 02 06:16:51.816803 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.816258 5916 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3500e682-2f15-45a2-92d6-bbc35dae3231" volumeName="kubernetes.io/projected/3500e682-2f15-45a2-92d6-bbc35dae3231-kube-api-access-v8lqt" seLinuxMountContext="" Nov 02 06:16:51.816803 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.816269 5916 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="121e38c0-6fa3-46aa-9191-5c6f91c8764c" volumeName="kubernetes.io/projected/121e38c0-6fa3-46aa-9191-5c6f91c8764c-kube-api-access-fnw94" seLinuxMountContext="" Nov 02 06:16:51.816803 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.816281 5916 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4ba9b2b6-4be9-4303-8eab-cd464c5a32c7" volumeName="kubernetes.io/secret/4ba9b2b6-4be9-4303-8eab-cd464c5a32c7-metrics-tls" seLinuxMountContext="" Nov 02 06:16:51.816803 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.816294 5916 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cc33996a-0d4c-49c4-ba67-283bd6a2de14" volumeName="kubernetes.io/configmap/cc33996a-0d4c-49c4-ba67-283bd6a2de14-config" seLinuxMountContext="" Nov 02 06:16:51.816803 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.816306 5916 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d4615a9e-da82-4f8b-8225-d60836b1ba80" volumeName="kubernetes.io/secret/d4615a9e-da82-4f8b-8225-d60836b1ba80-agent-certs" seLinuxMountContext="" Nov 02 06:16:51.816803 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.816317 5916 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fa9c5a91-d480-4d7b-9044-be6437a76a02" volumeName="kubernetes.io/empty-dir/fa9c5a91-d480-4d7b-9044-be6437a76a02-ca-trust-extracted" seLinuxMountContext="" Nov 02 06:16:51.816803 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.816329 5916 reconstruct.go:97] "Volume reconstruction finished" Nov 02 06:16:51.816803 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.816336 5916 reconciler.go:29] "Reconciler: start to sync state" Nov 02 06:16:51.829960 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.829940 5916 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-csfhg_17dee72b-2f47-4016-8ffe-3a89934e66ee/console-operator/2.log" Nov 02 06:16:51.830958 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.830936 5916 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-csfhg_17dee72b-2f47-4016-8ffe-3a89934e66ee/console-operator/2.log" Nov 02 06:16:51.832042 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.831998 5916 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 02 06:16:51.834237 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.834214 5916 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 02 06:16:51.834324 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.834258 5916 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 02 06:16:51.834324 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.834282 5916 kubelet.go:2521] "Starting kubelet main sync loop" Nov 02 06:16:51.835700 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:16:51.834766 5916 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 02 06:16:51.859931 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.856244 5916 generic.go:356] "Generic (PLEG): container finished" podID="12b6cc1c-537d-4125-92ff-fe5f6927c53e" containerID="4191b5d75eba96c8e2cd049f0f075d84f7480825b2c51619e1348610d941d6e9" exitCode=0 Nov 02 06:16:51.859931 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.859847 5916 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-csfhg_17dee72b-2f47-4016-8ffe-3a89934e66ee/console-operator/2.log" Nov 02 06:16:51.859931 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.859869 5916 generic.go:356] "Generic (PLEG): container finished" podID="17dee72b-2f47-4016-8ffe-3a89934e66ee" containerID="8bd3233465a5560a8e5e0dc6a2c9a27ed2a1f235d342c92357cf3a9a272a46f8" exitCode=255 Nov 02 06:16:51.886935 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.886886 5916 generic.go:356] "Generic (PLEG): container finished" podID="f14e8781-bf82-4810-9909-c126ccdc1ce1" containerID="6126d2d3da241baf441593f8786423fe729eaafe1d673f2edba5283c8315c686" exitCode=0 Nov 02 06:16:51.886935 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.886933 5916 generic.go:356] "Generic (PLEG): container finished" podID="f14e8781-bf82-4810-9909-c126ccdc1ce1" containerID="2590abd77229ac30d7d9c7b2730e33a159aae8da0294b29c2cb2179fce248d00" exitCode=0 Nov 02 06:16:51.887127 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.886945 5916 generic.go:356] "Generic (PLEG): container finished" podID="f14e8781-bf82-4810-9909-c126ccdc1ce1" containerID="461ad93213230147cb160bb730bef28178a765173b74c29daa8aa46e000b25c2" exitCode=0 Nov 02 06:16:51.887127 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.886959 5916 generic.go:356] "Generic (PLEG): container finished" podID="f14e8781-bf82-4810-9909-c126ccdc1ce1" containerID="6319d210ebfcd8b05a29408bdb389e63c5195f58158050e90c9ce7b60421c445" exitCode=0 Nov 02 06:16:51.887127 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.886968 5916 generic.go:356] "Generic (PLEG): container finished" podID="f14e8781-bf82-4810-9909-c126ccdc1ce1" containerID="94631bd8da05ef22b70e792603345d4a8c23600a4c60a85272ac7327b8c9e6de" exitCode=0 Nov 02 06:16:51.887127 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.886976 5916 generic.go:356] "Generic (PLEG): container finished" podID="f14e8781-bf82-4810-9909-c126ccdc1ce1" containerID="76edac44fe3d033b7481267e1b7ca70bc402fc94925f72e65777944577000b23" exitCode=0 Nov 02 06:16:51.907596 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:16:51.907574 5916 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-93.us-west-2.compute.internal\" not found" Nov 02 06:16:51.908524 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.908496 5916 generic.go:356] "Generic (PLEG): container finished" podID="71bd052da0df40177024b26f7f0347a4" containerID="06999e0d9cadcd372d30037d1ddca8b04b836b2fc7c351f5fd08606d319a06e3" exitCode=0 Nov 02 06:16:51.935501 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:16:51.935441 5916 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Nov 02 06:16:51.938841 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.938820 5916 manager.go:324] Recovery completed Nov 02 06:16:51.963650 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.963628 5916 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 06:16:51.965545 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.965529 5916 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-93.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 06:16:51.965606 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.965559 5916 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-93.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 06:16:51.965606 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.965569 5916 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-93.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 06:16:51.968726 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.968701 5916 cpu_manager.go:226] "Starting" policy="none" Nov 02 06:16:51.968726 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.968712 5916 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 06:16:51.968868 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.968717 5916 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 02 06:16:51.968868 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.968786 5916 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 02 06:16:51.968945 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.968926 5916 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Nov 02 06:16:51.968974 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.968938 5916 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Nov 02 06:16:51.968974 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.968969 5916 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Nov 02 06:16:51.968974 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.968974 5916 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Nov 02 06:16:51.969096 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.968980 5916 policy_none.go:50] "Start" Nov 02 06:16:51.969096 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.968991 5916 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 02 06:16:51.969096 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.969001 5916 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 02 06:16:51.969622 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.969603 5916 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-93.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 06:16:51.969673 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.969641 5916 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-93.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 06:16:51.969673 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.969658 5916 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-93.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 06:16:51.985240 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.985220 5916 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Nov 02 06:16:51.985240 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.985239 5916 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Nov 02 06:16:51.985324 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.985247 5916 policy_none.go:44] "Start" Nov 02 06:16:51.991327 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.989968 5916 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Nov 02 06:16:51.991327 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.990158 5916 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 06:16:51.991327 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.990259 5916 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 06:16:51.991327 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.991238 5916 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 02 06:16:51.997049 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.997034 5916 manager.go:341] "Starting Device Plugin manager" Nov 02 06:16:51.997144 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:16:51.997060 5916 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 02 06:16:51.997144 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.997070 5916 server.go:87] "Starting device plugin registration server" Nov 02 06:16:51.998095 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.998080 5916 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 02 06:16:51.998146 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.998096 5916 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 02 06:16:51.998234 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.998214 5916 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 02 06:16:51.998366 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.998330 5916 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 02 06:16:51.998442 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.998374 5916 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 02 06:16:51.998442 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.998381 5916 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 02 06:16:51.998510 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.998430 5916 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-02T06:16:51.998362932Z","UUID":"9f069678-7b8f-44c4-bb42-a593ab42f31f","Handler":null,"Name":"","Endpoint":""} Nov 02 06:16:51.999727 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.999702 5916 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 02 06:16:51.999831 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:51.999749 5916 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 02 06:16:51.999983 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:16:51.999965 5916 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 02 06:16:52.000029 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:16:52.000006 5916 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-130-93.us-west-2.compute.internal\" not found" Nov 02 06:16:52.034509 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.034486 5916 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 06:16:52.062262 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.062240 5916 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 06:16:52.098619 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.098596 5916 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 06:16:52.099590 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.099573 5916 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-93.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 06:16:52.099679 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.099603 5916 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-93.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 06:16:52.099679 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.099619 5916 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-93.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 06:16:52.099679 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.099665 5916 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-130-93.us-west-2.compute.internal" Nov 02 06:16:52.135711 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.135653 5916 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-93.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-130-93.us-west-2.compute.internal"] Nov 02 06:16:52.136356 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.136340 5916 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-93.us-west-2.compute.internal" Nov 02 06:16:52.136419 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.136346 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-93.us-west-2.compute.internal" event={"ID":"df7856ce915c3ef5bc71a7a412d369e5","Type":"ContainerStarted","Data":"7cf04f4ab88fe8af710fd703df392232e8353cd4be1bbe5ab7df145c65ad7082"} Nov 02 06:16:52.136419 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.136385 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-93.us-west-2.compute.internal" event={"ID":"df7856ce915c3ef5bc71a7a412d369e5","Type":"ContainerStarted","Data":"b5d368d976619eaa837c56d4e22406db6bf7495a6d9575c227809c8db3f208ac"} Nov 02 06:16:52.136489 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.136438 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-93.us-west-2.compute.internal" event={"ID":"71bd052da0df40177024b26f7f0347a4","Type":"ContainerStarted","Data":"2d7ef57bbc7970aa1520b98dd62ad041d6041a58224f00535e5e79b211c56a00"} Nov 02 06:16:52.136489 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.136457 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-93.us-west-2.compute.internal" event={"ID":"71bd052da0df40177024b26f7f0347a4","Type":"ContainerDied","Data":"06999e0d9cadcd372d30037d1ddca8b04b836b2fc7c351f5fd08606d319a06e3"} Nov 02 06:16:52.136489 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.136475 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-93.us-west-2.compute.internal" event={"ID":"71bd052da0df40177024b26f7f0347a4","Type":"ContainerStarted","Data":"c1ac0d913b943b018616c75971989a95da11db3c09c0128e4db983e8b0d0eaa3"} Nov 02 06:16:52.136593 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.136502 5916 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-93.us-west-2.compute.internal" Nov 02 06:16:52.197077 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.197017 5916 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 06:16:52.197077 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:16:52.197058 5916 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-130-93.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-93.us-west-2.compute.internal" Nov 02 06:16:52.197582 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.197568 5916 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 06:16:52.197617 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:16:52.197604 5916 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-130-93.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-93.us-west-2.compute.internal" Nov 02 06:16:52.202513 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.202498 5916 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-130-93.us-west-2.compute.internal" Nov 02 06:16:52.202616 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.202551 5916 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-130-93.us-west-2.compute.internal" Nov 02 06:16:52.291758 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.291728 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/71bd052da0df40177024b26f7f0347a4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-93.us-west-2.compute.internal\" (UID: \"71bd052da0df40177024b26f7f0347a4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-93.us-west-2.compute.internal" Nov 02 06:16:52.291933 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.291765 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/71bd052da0df40177024b26f7f0347a4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-93.us-west-2.compute.internal\" (UID: \"71bd052da0df40177024b26f7f0347a4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-93.us-west-2.compute.internal" Nov 02 06:16:52.291933 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.291792 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/df7856ce915c3ef5bc71a7a412d369e5-config\") pod \"kube-apiserver-proxy-ip-10-0-130-93.us-west-2.compute.internal\" (UID: \"df7856ce915c3ef5bc71a7a412d369e5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-93.us-west-2.compute.internal" Nov 02 06:16:52.392147 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.392115 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/df7856ce915c3ef5bc71a7a412d369e5-config\") pod \"kube-apiserver-proxy-ip-10-0-130-93.us-west-2.compute.internal\" (UID: \"df7856ce915c3ef5bc71a7a412d369e5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-93.us-west-2.compute.internal" Nov 02 06:16:52.392147 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.392148 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/71bd052da0df40177024b26f7f0347a4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-93.us-west-2.compute.internal\" (UID: \"71bd052da0df40177024b26f7f0347a4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-93.us-west-2.compute.internal" Nov 02 06:16:52.392345 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.392166 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/71bd052da0df40177024b26f7f0347a4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-93.us-west-2.compute.internal\" (UID: \"71bd052da0df40177024b26f7f0347a4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-93.us-west-2.compute.internal" Nov 02 06:16:52.392345 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.392216 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/71bd052da0df40177024b26f7f0347a4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-93.us-west-2.compute.internal\" (UID: \"71bd052da0df40177024b26f7f0347a4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-93.us-west-2.compute.internal" Nov 02 06:16:52.392345 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.392223 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/df7856ce915c3ef5bc71a7a412d369e5-config\") pod \"kube-apiserver-proxy-ip-10-0-130-93.us-west-2.compute.internal\" (UID: \"df7856ce915c3ef5bc71a7a412d369e5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-93.us-west-2.compute.internal" Nov 02 06:16:52.392345 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.392268 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/71bd052da0df40177024b26f7f0347a4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-93.us-west-2.compute.internal\" (UID: \"71bd052da0df40177024b26f7f0347a4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-93.us-west-2.compute.internal" Nov 02 06:16:52.792550 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.792519 5916 apiserver.go:52] "Watching apiserver" Nov 02 06:16:52.889656 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.889620 5916 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 02 06:16:52.891333 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.891285 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-5f5c58667c-29wnm","openshift-network-operator/iptables-alerter-6fnx8","openshift-cluster-node-tuning-operator/tuned-6nk8l","openshift-network-console/networking-console-plugin-86c478bb88-fh4sz","openshift-insights/insights-runtime-extractor-p98cq","openshift-image-registry/node-ca-qm6wn","kube-system/konnectivity-agent-hwc99","openshift-ingress-canary/ingress-canary-6tt27","kube-system/kas-connection-checker-685895c7c9-jrtnt","openshift-multus/multus-rf7qp","kube-system/global-pull-secret-syncer-ggpsr","openshift-service-ca-operator/service-ca-operator-544d749c8c-wcrtl","kube-system/kas-connection-checker-685895c7c9-kd68m","openshift-console-operator/console-operator-7cd56b8dbc-csfhg","openshift-dns/node-resolver-2p95f","kube-system/kube-apiserver-proxy-ip-10-0-130-93.us-west-2.compute.internal","kube-system/kas-connection-checker-685895c7c9-kb9wk","openshift-multus/multus-additional-cni-plugins-ckddw","openshift-image-registry/image-registry-746899d75-dtjhs","openshift-insights/insights-operator-7c4986d449-c8dcq","openshift-service-ca/service-ca-5fffc97bdf-96jpc","openshift-image-registry/image-registry-6689c6d46b-vw558","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-sktt2","openshift-ovn-kubernetes/ovnkube-node-7wb4x","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-8fbcn","openshift-insights/periodic-gathering-rrgb2-xp5b2","openshift-network-diagnostics/network-check-source-6f78d59f99-k9hcs","openshift-dns/dns-default-gf726","openshift-monitoring/cluster-monitoring-operator-884576c57-2rs6c","openshift-network-diagnostics/network-check-target-ft7gs","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-93.us-west-2.compute.internal","openshift-multus/network-metrics-daemon-sb4nt","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-d6bc4","openshift-kube-storage-version-migrator/migrator-66df5d496b-mddfd","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cbzvd"] Nov 02 06:16:52.915524 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.915496 5916 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-93.us-west-2.compute.internal" Nov 02 06:16:52.916139 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.916120 5916 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-93.us-west-2.compute.internal" Nov 02 06:16:52.943475 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.943443 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Nov 02 06:16:52.943475 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.943443 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Nov 02 06:16:52.943700 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.943446 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Nov 02 06:16:52.943700 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.943500 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Nov 02 06:16:52.943700 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.943554 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Nov 02 06:16:52.943700 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.943516 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Nov 02 06:16:52.943841 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.943800 5916 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Nov 02 06:16:52.943897 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.943882 5916 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 02 06:16:52.943972 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.943960 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Nov 02 06:16:52.944012 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.943967 5916 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 02 06:16:52.944167 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.944150 5916 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 02 06:16:52.944167 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.944162 5916 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Nov 02 06:16:52.944314 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.944300 5916 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 02 06:16:52.944360 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.944312 5916 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Nov 02 06:16:52.944408 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.944394 5916 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 02 06:16:52.944443 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.944417 5916 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 02 06:16:52.944491 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.944478 5916 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 02 06:16:52.944536 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.944523 5916 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Nov 02 06:16:52.944581 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.944534 5916 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Nov 02 06:16:52.944581 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.944557 5916 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Nov 02 06:16:52.944651 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.944611 5916 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 02 06:16:52.944651 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.944618 5916 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Nov 02 06:16:52.944651 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.944649 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Nov 02 06:16:52.944780 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.944676 5916 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Nov 02 06:16:52.944780 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.944678 5916 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Nov 02 06:16:52.944780 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.944711 5916 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Nov 02 06:16:52.945171 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.945152 5916 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Nov 02 06:16:52.945450 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.945437 5916 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Nov 02 06:16:52.982721 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.982694 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-9bfjz\"" Nov 02 06:16:52.982946 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.982927 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-d57td\"" Nov 02 06:16:52.983156 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.983139 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Nov 02 06:16:52.983446 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.983417 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 02 06:16:52.983446 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.983422 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 02 06:16:52.983446 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.983436 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 02 06:16:52.983655 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.983463 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 02 06:16:52.983655 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.983472 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-flhk8\"" Nov 02 06:16:52.983655 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.983480 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-bpw78\"" Nov 02 06:16:52.983655 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.983495 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 02 06:16:52.983655 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.983575 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-zth7z\"" Nov 02 06:16:52.983655 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.983646 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-rwgrc\"" Nov 02 06:16:52.988139 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.988124 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-4lqj8\"" Nov 02 06:16:52.988204 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.988151 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-cdk7k\"" Nov 02 06:16:52.992780 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.992757 5916 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 06:16:52.992868 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:16:52.992807 5916 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-130-93.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-93.us-west-2.compute.internal" Nov 02 06:16:52.993337 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.993321 5916 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 06:16:52.993392 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:16:52.993361 5916 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-130-93.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-93.us-west-2.compute.internal" Nov 02 06:16:52.995549 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.995531 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f0531b4d-c954-4830-85f6-fe876c76437a-ca-trust-extracted\") pod \"image-registry-6689c6d46b-vw558\" (UID: \"f0531b4d-c954-4830-85f6-fe876c76437a\") " pod="openshift-image-registry/image-registry-6689c6d46b-vw558" Nov 02 06:16:52.995590 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.995559 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/37982aa4-bdbd-40e8-a2dd-5094209f3c96-tmp\") pod \"insights-operator-7c4986d449-c8dcq\" (UID: \"37982aa4-bdbd-40e8-a2dd-5094209f3c96\") " pod="openshift-insights/insights-operator-7c4986d449-c8dcq" Nov 02 06:16:52.995590 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.995577 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/12b6cc1c-537d-4125-92ff-fe5f6927c53e-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-sktt2\" (UID: \"12b6cc1c-537d-4125-92ff-fe5f6927c53e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-sktt2" Nov 02 06:16:52.995693 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.995603 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8lqt\" (UniqueName: \"kubernetes.io/projected/3500e682-2f15-45a2-92d6-bbc35dae3231-kube-api-access-v8lqt\") pod \"aws-ebs-csi-driver-node-cbzvd\" (UID: \"3500e682-2f15-45a2-92d6-bbc35dae3231\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cbzvd" Nov 02 06:16:52.995693 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.995656 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/87e797a0-1da6-41ca-af8d-7382afaf3987-multus-cni-dir\") pod \"multus-rf7qp\" (UID: \"87e797a0-1da6-41ca-af8d-7382afaf3987\") " pod="openshift-multus/multus-rf7qp" Nov 02 06:16:52.995693 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.995671 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f0531b4d-c954-4830-85f6-fe876c76437a-ca-trust-extracted\") pod \"image-registry-6689c6d46b-vw558\" (UID: \"f0531b4d-c954-4830-85f6-fe876c76437a\") " pod="openshift-image-registry/image-registry-6689c6d46b-vw558" Nov 02 06:16:52.995789 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.995691 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/87e797a0-1da6-41ca-af8d-7382afaf3987-host-run-netns\") pod \"multus-rf7qp\" (UID: \"87e797a0-1da6-41ca-af8d-7382afaf3987\") " pod="openshift-multus/multus-rf7qp" Nov 02 06:16:52.995789 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.995738 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a2ef8269-c442-42a8-8041-6c1b6aaf0746-log-socket\") pod \"ovnkube-node-7wb4x\" (UID: \"a2ef8269-c442-42a8-8041-6c1b6aaf0746\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wb4x" Nov 02 06:16:52.995789 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.995754 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/37982aa4-bdbd-40e8-a2dd-5094209f3c96-tmp\") pod \"insights-operator-7c4986d449-c8dcq\" (UID: \"37982aa4-bdbd-40e8-a2dd-5094209f3c96\") " pod="openshift-insights/insights-operator-7c4986d449-c8dcq" Nov 02 06:16:52.995789 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.995759 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fa9c5a91-d480-4d7b-9044-be6437a76a02-trusted-ca\") pod \"image-registry-746899d75-dtjhs\" (UID: \"fa9c5a91-d480-4d7b-9044-be6437a76a02\") " pod="openshift-image-registry/image-registry-746899d75-dtjhs" Nov 02 06:16:52.995990 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.995800 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/12b6cc1c-537d-4125-92ff-fe5f6927c53e-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-sktt2\" (UID: \"12b6cc1c-537d-4125-92ff-fe5f6927c53e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-sktt2" Nov 02 06:16:52.995990 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.995802 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mplc\" (UniqueName: \"kubernetes.io/projected/0d2b53c4-9d39-47fc-9367-c9a8e7f5da81-kube-api-access-8mplc\") pod \"ingress-canary-6tt27\" (UID: \"0d2b53c4-9d39-47fc-9367-c9a8e7f5da81\") " pod="openshift-ingress-canary/ingress-canary-6tt27" Nov 02 06:16:52.995990 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.995873 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/81b393f2-42de-4a99-8114-1445de4d6dc8-etc-sysconfig\") pod \"tuned-6nk8l\" (UID: \"81b393f2-42de-4a99-8114-1445de4d6dc8\") " pod="openshift-cluster-node-tuning-operator/tuned-6nk8l" Nov 02 06:16:52.995990 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.995930 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/87e797a0-1da6-41ca-af8d-7382afaf3987-multus-daemon-config\") pod \"multus-rf7qp\" (UID: \"87e797a0-1da6-41ca-af8d-7382afaf3987\") " pod="openshift-multus/multus-rf7qp" Nov 02 06:16:52.995990 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.995957 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a2ef8269-c442-42a8-8041-6c1b6aaf0746-run-systemd\") pod \"ovnkube-node-7wb4x\" (UID: \"a2ef8269-c442-42a8-8041-6c1b6aaf0746\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wb4x" Nov 02 06:16:52.995990 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.995982 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3500e682-2f15-45a2-92d6-bbc35dae3231-kubelet-dir\") pod \"aws-ebs-csi-driver-node-cbzvd\" (UID: \"3500e682-2f15-45a2-92d6-bbc35dae3231\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cbzvd" Nov 02 06:16:52.996207 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.996009 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/53aa1e5a-c35f-4aa6-9f23-cc61ccd331fc-nginx-conf\") pod \"networking-console-plugin-86c478bb88-fh4sz\" (UID: \"53aa1e5a-c35f-4aa6-9f23-cc61ccd331fc\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-fh4sz" Nov 02 06:16:52.996207 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.996036 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7xzh\" (UniqueName: \"kubernetes.io/projected/f14e8781-bf82-4810-9909-c126ccdc1ce1-kube-api-access-v7xzh\") pod \"multus-additional-cni-plugins-ckddw\" (UID: \"f14e8781-bf82-4810-9909-c126ccdc1ce1\") " pod="openshift-multus/multus-additional-cni-plugins-ckddw" Nov 02 06:16:52.996207 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.996060 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4ba9b2b6-4be9-4303-8eab-cd464c5a32c7-config-volume\") pod \"dns-default-gf726\" (UID: \"4ba9b2b6-4be9-4303-8eab-cd464c5a32c7\") " pod="openshift-dns/dns-default-gf726" Nov 02 06:16:52.996207 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.996082 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45e5ce56-5cab-4546-8b31-02fc06cfa0cc-metrics-certs\") pod \"router-default-5f5c58667c-29wnm\" (UID: \"45e5ce56-5cab-4546-8b31-02fc06cfa0cc\") " pod="openshift-ingress/router-default-5f5c58667c-29wnm" Nov 02 06:16:52.996207 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.996099 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b65bb247-83f7-4749-9aa1-9b2677dd1b3d-dbus\") pod \"global-pull-secret-syncer-ggpsr\" (UID: \"b65bb247-83f7-4749-9aa1-9b2677dd1b3d\") " pod="kube-system/global-pull-secret-syncer-ggpsr" Nov 02 06:16:52.996207 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.996118 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/a4319db2-eda9-47a0-b742-985c8bdd7ea2-archives-path\") pod \"periodic-gathering-rrgb2-xp5b2\" (UID: \"a4319db2-eda9-47a0-b742-985c8bdd7ea2\") " pod="openshift-insights/periodic-gathering-rrgb2-xp5b2" Nov 02 06:16:52.996207 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.996157 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5989k\" (UniqueName: \"kubernetes.io/projected/20cb7690-c976-471c-b459-a9ab15cc51bf-kube-api-access-5989k\") pod \"network-check-source-6f78d59f99-k9hcs\" (UID: \"20cb7690-c976-471c-b459-a9ab15cc51bf\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-k9hcs" Nov 02 06:16:52.996511 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.996241 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/87e797a0-1da6-41ca-af8d-7382afaf3987-host-run-k8s-cni-cncf-io\") pod \"multus-rf7qp\" (UID: \"87e797a0-1da6-41ca-af8d-7382afaf3987\") " pod="openshift-multus/multus-rf7qp" Nov 02 06:16:52.996511 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.996273 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/81b393f2-42de-4a99-8114-1445de4d6dc8-var-lib-kubelet\") pod \"tuned-6nk8l\" (UID: \"81b393f2-42de-4a99-8114-1445de4d6dc8\") " pod="openshift-cluster-node-tuning-operator/tuned-6nk8l" Nov 02 06:16:52.996511 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.996277 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/a4319db2-eda9-47a0-b742-985c8bdd7ea2-archives-path\") pod \"periodic-gathering-rrgb2-xp5b2\" (UID: \"a4319db2-eda9-47a0-b742-985c8bdd7ea2\") " pod="openshift-insights/periodic-gathering-rrgb2-xp5b2" Nov 02 06:16:52.996511 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.996311 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/81b393f2-42de-4a99-8114-1445de4d6dc8-host\") pod \"tuned-6nk8l\" (UID: \"81b393f2-42de-4a99-8114-1445de4d6dc8\") " pod="openshift-cluster-node-tuning-operator/tuned-6nk8l" Nov 02 06:16:52.996511 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.996334 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a2ef8269-c442-42a8-8041-6c1b6aaf0746-systemd-units\") pod \"ovnkube-node-7wb4x\" (UID: \"a2ef8269-c442-42a8-8041-6c1b6aaf0746\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wb4x" Nov 02 06:16:52.996511 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.996352 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/87e797a0-1da6-41ca-af8d-7382afaf3987-multus-socket-dir-parent\") pod \"multus-rf7qp\" (UID: \"87e797a0-1da6-41ca-af8d-7382afaf3987\") " pod="openshift-multus/multus-rf7qp" Nov 02 06:16:52.996511 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.996373 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvdp9\" (UniqueName: \"kubernetes.io/projected/3b6432a8-b0ce-4954-8e3d-8eb8782557eb-kube-api-access-pvdp9\") pod \"network-metrics-daemon-sb4nt\" (UID: \"3b6432a8-b0ce-4954-8e3d-8eb8782557eb\") " pod="openshift-multus/network-metrics-daemon-sb4nt" Nov 02 06:16:52.996511 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.996370 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45e5ce56-5cab-4546-8b31-02fc06cfa0cc-metrics-certs\") pod \"router-default-5f5c58667c-29wnm\" (UID: \"45e5ce56-5cab-4546-8b31-02fc06cfa0cc\") " pod="openshift-ingress/router-default-5f5c58667c-29wnm" Nov 02 06:16:52.996511 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.996403 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8vwr\" (UniqueName: \"kubernetes.io/projected/3eec1082-df0b-420b-b4ff-5a1c0fab1d1d-kube-api-access-k8vwr\") pod \"node-ca-qm6wn\" (UID: \"3eec1082-df0b-420b-b4ff-5a1c0fab1d1d\") " pod="openshift-image-registry/node-ca-qm6wn" Nov 02 06:16:52.996511 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.996464 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/3fcb7621-86c8-448f-b4fa-b5bd76be7356-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-2rs6c\" (UID: \"3fcb7621-86c8-448f-b4fa-b5bd76be7356\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2rs6c" Nov 02 06:16:52.996922 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.996529 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8d87317-3f41-46d4-a58e-3c7601e0d445-tmp-dir\") pod \"node-resolver-2p95f\" (UID: \"d8d87317-3f41-46d4-a58e-3c7601e0d445\") " pod="openshift-dns/node-resolver-2p95f" Nov 02 06:16:52.996922 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.996546 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/81b393f2-42de-4a99-8114-1445de4d6dc8-run\") pod \"tuned-6nk8l\" (UID: \"81b393f2-42de-4a99-8114-1445de4d6dc8\") " pod="openshift-cluster-node-tuning-operator/tuned-6nk8l" Nov 02 06:16:52.996922 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.996589 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/81b393f2-42de-4a99-8114-1445de4d6dc8-sys\") pod \"tuned-6nk8l\" (UID: \"81b393f2-42de-4a99-8114-1445de4d6dc8\") " pod="openshift-cluster-node-tuning-operator/tuned-6nk8l" Nov 02 06:16:52.996922 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.996649 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a2ef8269-c442-42a8-8041-6c1b6aaf0746-env-overrides\") pod \"ovnkube-node-7wb4x\" (UID: \"a2ef8269-c442-42a8-8041-6c1b6aaf0746\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wb4x" Nov 02 06:16:52.996922 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.996672 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f0531b4d-c954-4830-85f6-fe876c76437a-registry-tls\") pod \"image-registry-6689c6d46b-vw558\" (UID: \"f0531b4d-c954-4830-85f6-fe876c76437a\") " pod="openshift-image-registry/image-registry-6689c6d46b-vw558" Nov 02 06:16:52.996922 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.996730 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8d87317-3f41-46d4-a58e-3c7601e0d445-tmp-dir\") pod \"node-resolver-2p95f\" (UID: \"d8d87317-3f41-46d4-a58e-3c7601e0d445\") " pod="openshift-dns/node-resolver-2p95f" Nov 02 06:16:52.996922 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.996736 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4ba9b2b6-4be9-4303-8eab-cd464c5a32c7-metrics-tls\") pod \"dns-default-gf726\" (UID: \"4ba9b2b6-4be9-4303-8eab-cd464c5a32c7\") " pod="openshift-dns/dns-default-gf726" Nov 02 06:16:52.996922 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.996757 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/3fcb7621-86c8-448f-b4fa-b5bd76be7356-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-2rs6c\" (UID: \"3fcb7621-86c8-448f-b4fa-b5bd76be7356\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2rs6c" Nov 02 06:16:52.996922 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.996820 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ba9b2b6-4be9-4303-8eab-cd464c5a32c7-tmp-dir\") pod \"dns-default-gf726\" (UID: \"4ba9b2b6-4be9-4303-8eab-cd464c5a32c7\") " pod="openshift-dns/dns-default-gf726" Nov 02 06:16:52.996922 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.996897 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ba9b2b6-4be9-4303-8eab-cd464c5a32c7-tmp-dir\") pod \"dns-default-gf726\" (UID: \"4ba9b2b6-4be9-4303-8eab-cd464c5a32c7\") " pod="openshift-dns/dns-default-gf726" Nov 02 06:16:52.997339 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.996898 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnq6k\" (UniqueName: \"kubernetes.io/projected/3ced47b0-9cd0-4710-8f75-2a333437ed01-kube-api-access-gnq6k\") pod \"network-check-target-ft7gs\" (UID: \"3ced47b0-9cd0-4710-8f75-2a333437ed01\") " pod="openshift-network-diagnostics/network-check-target-ft7gs" Nov 02 06:16:52.997339 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.996952 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fa9c5a91-d480-4d7b-9044-be6437a76a02-ca-trust-extracted\") pod \"image-registry-746899d75-dtjhs\" (UID: \"fa9c5a91-d480-4d7b-9044-be6437a76a02\") " pod="openshift-image-registry/image-registry-746899d75-dtjhs" Nov 02 06:16:52.997339 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.996972 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a2ef8269-c442-42a8-8041-6c1b6aaf0746-run-ovn\") pod \"ovnkube-node-7wb4x\" (UID: \"a2ef8269-c442-42a8-8041-6c1b6aaf0746\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wb4x" Nov 02 06:16:52.997339 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.996998 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cc33996a-0d4c-49c4-ba67-283bd6a2de14-serving-cert\") pod \"service-ca-operator-544d749c8c-wcrtl\" (UID: \"cc33996a-0d4c-49c4-ba67-283bd6a2de14\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wcrtl" Nov 02 06:16:52.997339 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.997024 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a4319db2-eda9-47a0-b742-985c8bdd7ea2-trusted-ca-bundle\") pod \"periodic-gathering-rrgb2-xp5b2\" (UID: \"a4319db2-eda9-47a0-b742-985c8bdd7ea2\") " pod="openshift-insights/periodic-gathering-rrgb2-xp5b2" Nov 02 06:16:52.997339 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.997038 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fa9c5a91-d480-4d7b-9044-be6437a76a02-ca-trust-extracted\") pod \"image-registry-746899d75-dtjhs\" (UID: \"fa9c5a91-d480-4d7b-9044-be6437a76a02\") " pod="openshift-image-registry/image-registry-746899d75-dtjhs" Nov 02 06:16:52.997339 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.997051 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/17dee72b-2f47-4016-8ffe-3a89934e66ee-trusted-ca\") pod \"console-operator-7cd56b8dbc-csfhg\" (UID: \"17dee72b-2f47-4016-8ffe-3a89934e66ee\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-csfhg" Nov 02 06:16:52.997339 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.997108 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnw94\" (UniqueName: \"kubernetes.io/projected/121e38c0-6fa3-46aa-9191-5c6f91c8764c-kube-api-access-fnw94\") pod \"volume-data-source-validator-58c9f5f46d-8fbcn\" (UID: \"121e38c0-6fa3-46aa-9191-5c6f91c8764c\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-8fbcn" Nov 02 06:16:52.997339 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.997132 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3fcb7621-86c8-448f-b4fa-b5bd76be7356-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-2rs6c\" (UID: \"3fcb7621-86c8-448f-b4fa-b5bd76be7356\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2rs6c" Nov 02 06:16:52.997339 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.997292 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwh5f\" (UniqueName: \"kubernetes.io/projected/3fcb7621-86c8-448f-b4fa-b5bd76be7356-kube-api-access-mwh5f\") pod \"cluster-monitoring-operator-884576c57-2rs6c\" (UID: \"3fcb7621-86c8-448f-b4fa-b5bd76be7356\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2rs6c" Nov 02 06:16:52.997339 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.997332 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmsjq\" (UniqueName: \"kubernetes.io/projected/12b6cc1c-537d-4125-92ff-fe5f6927c53e-kube-api-access-bmsjq\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-sktt2\" (UID: \"12b6cc1c-537d-4125-92ff-fe5f6927c53e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-sktt2" Nov 02 06:16:52.997755 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.997368 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f14e8781-bf82-4810-9909-c126ccdc1ce1-system-cni-dir\") pod \"multus-additional-cni-plugins-ckddw\" (UID: \"f14e8781-bf82-4810-9909-c126ccdc1ce1\") " pod="openshift-multus/multus-additional-cni-plugins-ckddw" Nov 02 06:16:52.997755 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.997394 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f14e8781-bf82-4810-9909-c126ccdc1ce1-cnibin\") pod \"multus-additional-cni-plugins-ckddw\" (UID: \"f14e8781-bf82-4810-9909-c126ccdc1ce1\") " pod="openshift-multus/multus-additional-cni-plugins-ckddw" Nov 02 06:16:52.997755 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.997401 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cc33996a-0d4c-49c4-ba67-283bd6a2de14-serving-cert\") pod \"service-ca-operator-544d749c8c-wcrtl\" (UID: \"cc33996a-0d4c-49c4-ba67-283bd6a2de14\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wcrtl" Nov 02 06:16:52.997755 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.997408 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3fcb7621-86c8-448f-b4fa-b5bd76be7356-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-2rs6c\" (UID: \"3fcb7621-86c8-448f-b4fa-b5bd76be7356\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2rs6c" Nov 02 06:16:52.997755 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.997423 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f0531b4d-c954-4830-85f6-fe876c76437a-trusted-ca\") pod \"image-registry-6689c6d46b-vw558\" (UID: \"f0531b4d-c954-4830-85f6-fe876c76437a\") " pod="openshift-image-registry/image-registry-6689c6d46b-vw558" Nov 02 06:16:52.997755 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.997467 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a4319db2-eda9-47a0-b742-985c8bdd7ea2-trusted-ca-bundle\") pod \"periodic-gathering-rrgb2-xp5b2\" (UID: \"a4319db2-eda9-47a0-b742-985c8bdd7ea2\") " pod="openshift-insights/periodic-gathering-rrgb2-xp5b2" Nov 02 06:16:52.997755 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.997480 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cctkp\" (UniqueName: \"kubernetes.io/projected/fa9c5a91-d480-4d7b-9044-be6437a76a02-kube-api-access-cctkp\") pod \"image-registry-746899d75-dtjhs\" (UID: \"fa9c5a91-d480-4d7b-9044-be6437a76a02\") " pod="openshift-image-registry/image-registry-746899d75-dtjhs" Nov 02 06:16:52.997755 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.997525 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/87e797a0-1da6-41ca-af8d-7382afaf3987-host-run-multus-certs\") pod \"multus-rf7qp\" (UID: \"87e797a0-1da6-41ca-af8d-7382afaf3987\") " pod="openshift-multus/multus-rf7qp" Nov 02 06:16:52.997755 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.997553 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3eec1082-df0b-420b-b4ff-5a1c0fab1d1d-serviceca\") pod \"node-ca-qm6wn\" (UID: \"3eec1082-df0b-420b-b4ff-5a1c0fab1d1d\") " pod="openshift-image-registry/node-ca-qm6wn" Nov 02 06:16:52.997755 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.997589 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvpvr\" (UniqueName: \"kubernetes.io/projected/45e5ce56-5cab-4546-8b31-02fc06cfa0cc-kube-api-access-xvpvr\") pod \"router-default-5f5c58667c-29wnm\" (UID: \"45e5ce56-5cab-4546-8b31-02fc06cfa0cc\") " pod="openshift-ingress/router-default-5f5c58667c-29wnm" Nov 02 06:16:52.997755 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.997660 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fa9c5a91-d480-4d7b-9044-be6437a76a02-image-registry-private-configuration\") pod \"image-registry-746899d75-dtjhs\" (UID: \"fa9c5a91-d480-4d7b-9044-be6437a76a02\") " pod="openshift-image-registry/image-registry-746899d75-dtjhs" Nov 02 06:16:52.997755 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.997687 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/12b6cc1c-537d-4125-92ff-fe5f6927c53e-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-sktt2\" (UID: \"12b6cc1c-537d-4125-92ff-fe5f6927c53e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-sktt2" Nov 02 06:16:52.997755 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.997705 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/87e797a0-1da6-41ca-af8d-7382afaf3987-multus-conf-dir\") pod \"multus-rf7qp\" (UID: \"87e797a0-1da6-41ca-af8d-7382afaf3987\") " pod="openshift-multus/multus-rf7qp" Nov 02 06:16:52.997755 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.997734 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6vx8\" (UniqueName: \"kubernetes.io/projected/87e797a0-1da6-41ca-af8d-7382afaf3987-kube-api-access-l6vx8\") pod \"multus-rf7qp\" (UID: \"87e797a0-1da6-41ca-af8d-7382afaf3987\") " pod="openshift-multus/multus-rf7qp" Nov 02 06:16:52.997755 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.997762 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/53aa1e5a-c35f-4aa6-9f23-cc61ccd331fc-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-fh4sz\" (UID: \"53aa1e5a-c35f-4aa6-9f23-cc61ccd331fc\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-fh4sz" Nov 02 06:16:52.998396 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.997789 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a2ef8269-c442-42a8-8041-6c1b6aaf0746-node-log\") pod \"ovnkube-node-7wb4x\" (UID: \"a2ef8269-c442-42a8-8041-6c1b6aaf0746\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wb4x" Nov 02 06:16:52.998396 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.997817 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a2ef8269-c442-42a8-8041-6c1b6aaf0746-ovnkube-config\") pod \"ovnkube-node-7wb4x\" (UID: \"a2ef8269-c442-42a8-8041-6c1b6aaf0746\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wb4x" Nov 02 06:16:52.998396 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.997835 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a2ef8269-c442-42a8-8041-6c1b6aaf0746-ovnkube-script-lib\") pod \"ovnkube-node-7wb4x\" (UID: \"a2ef8269-c442-42a8-8041-6c1b6aaf0746\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wb4x" Nov 02 06:16:52.998396 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.997855 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f14e8781-bf82-4810-9909-c126ccdc1ce1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ckddw\" (UID: \"f14e8781-bf82-4810-9909-c126ccdc1ce1\") " pod="openshift-multus/multus-additional-cni-plugins-ckddw" Nov 02 06:16:52.998396 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.997879 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3eec1082-df0b-420b-b4ff-5a1c0fab1d1d-host\") pod \"node-ca-qm6wn\" (UID: \"3eec1082-df0b-420b-b4ff-5a1c0fab1d1d\") " pod="openshift-image-registry/node-ca-qm6wn" Nov 02 06:16:52.998396 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.997936 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrh7x\" (UniqueName: \"kubernetes.io/projected/8e8f2e60-de1b-4c8b-9285-56568c150d92-kube-api-access-zrh7x\") pod \"kas-connection-checker-685895c7c9-kd68m\" (UID: \"8e8f2e60-de1b-4c8b-9285-56568c150d92\") " pod="kube-system/kas-connection-checker-685895c7c9-kd68m" Nov 02 06:16:52.998396 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.997980 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/12b6cc1c-537d-4125-92ff-fe5f6927c53e-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-sktt2\" (UID: \"12b6cc1c-537d-4125-92ff-fe5f6927c53e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-sktt2" Nov 02 06:16:52.998396 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.998003 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/53aa1e5a-c35f-4aa6-9f23-cc61ccd331fc-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-fh4sz\" (UID: \"53aa1e5a-c35f-4aa6-9f23-cc61ccd331fc\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-fh4sz" Nov 02 06:16:52.998396 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.998027 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a2ef8269-c442-42a8-8041-6c1b6aaf0746-run-openvswitch\") pod \"ovnkube-node-7wb4x\" (UID: \"a2ef8269-c442-42a8-8041-6c1b6aaf0746\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wb4x" Nov 02 06:16:52.998396 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.998057 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f14e8781-bf82-4810-9909-c126ccdc1ce1-cni-binary-copy\") pod \"multus-additional-cni-plugins-ckddw\" (UID: \"f14e8781-bf82-4810-9909-c126ccdc1ce1\") " pod="openshift-multus/multus-additional-cni-plugins-ckddw" Nov 02 06:16:52.998396 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.998087 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5q7t\" (UniqueName: \"kubernetes.io/projected/7434256f-45c3-47ff-9111-352114b0f6d8-kube-api-access-l5q7t\") pod \"kas-connection-checker-685895c7c9-kb9wk\" (UID: \"7434256f-45c3-47ff-9111-352114b0f6d8\") " pod="kube-system/kas-connection-checker-685895c7c9-kb9wk" Nov 02 06:16:52.998396 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.998119 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/37982aa4-bdbd-40e8-a2dd-5094209f3c96-snapshots\") pod \"insights-operator-7c4986d449-c8dcq\" (UID: \"37982aa4-bdbd-40e8-a2dd-5094209f3c96\") " pod="openshift-insights/insights-operator-7c4986d449-c8dcq" Nov 02 06:16:52.998396 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.998147 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/37982aa4-bdbd-40e8-a2dd-5094209f3c96-service-ca-bundle\") pod \"insights-operator-7c4986d449-c8dcq\" (UID: \"37982aa4-bdbd-40e8-a2dd-5094209f3c96\") " pod="openshift-insights/insights-operator-7c4986d449-c8dcq" Nov 02 06:16:52.998396 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.998179 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3500e682-2f15-45a2-92d6-bbc35dae3231-etc-selinux\") pod \"aws-ebs-csi-driver-node-cbzvd\" (UID: \"3500e682-2f15-45a2-92d6-bbc35dae3231\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cbzvd" Nov 02 06:16:52.998396 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.998225 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/87e797a0-1da6-41ca-af8d-7382afaf3987-etc-kubernetes\") pod \"multus-rf7qp\" (UID: \"87e797a0-1da6-41ca-af8d-7382afaf3987\") " pod="openshift-multus/multus-rf7qp" Nov 02 06:16:52.998396 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.998257 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a2ef8269-c442-42a8-8041-6c1b6aaf0746-host-run-netns\") pod \"ovnkube-node-7wb4x\" (UID: \"a2ef8269-c442-42a8-8041-6c1b6aaf0746\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wb4x" Nov 02 06:16:52.998396 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.998310 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a2ef8269-c442-42a8-8041-6c1b6aaf0746-host-cni-bin\") pod \"ovnkube-node-7wb4x\" (UID: \"a2ef8269-c442-42a8-8041-6c1b6aaf0746\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wb4x" Nov 02 06:16:52.998396 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.998353 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b65bb247-83f7-4749-9aa1-9b2677dd1b3d-kubelet-config\") pod \"global-pull-secret-syncer-ggpsr\" (UID: \"b65bb247-83f7-4749-9aa1-9b2677dd1b3d\") " pod="kube-system/global-pull-secret-syncer-ggpsr" Nov 02 06:16:52.998396 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.998365 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/37982aa4-bdbd-40e8-a2dd-5094209f3c96-snapshots\") pod \"insights-operator-7c4986d449-c8dcq\" (UID: \"37982aa4-bdbd-40e8-a2dd-5094209f3c96\") " pod="openshift-insights/insights-operator-7c4986d449-c8dcq" Nov 02 06:16:52.998396 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.998377 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fa9c5a91-d480-4d7b-9044-be6437a76a02-bound-sa-token\") pod \"image-registry-746899d75-dtjhs\" (UID: \"fa9c5a91-d480-4d7b-9044-be6437a76a02\") " pod="openshift-image-registry/image-registry-746899d75-dtjhs" Nov 02 06:16:52.998396 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.998407 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/37982aa4-bdbd-40e8-a2dd-5094209f3c96-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-c8dcq\" (UID: \"37982aa4-bdbd-40e8-a2dd-5094209f3c96\") " pod="openshift-insights/insights-operator-7c4986d449-c8dcq" Nov 02 06:16:52.999223 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.998510 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/81b393f2-42de-4a99-8114-1445de4d6dc8-etc-modprobe-d\") pod \"tuned-6nk8l\" (UID: \"81b393f2-42de-4a99-8114-1445de4d6dc8\") " pod="openshift-cluster-node-tuning-operator/tuned-6nk8l" Nov 02 06:16:52.999223 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.998621 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a2ef8269-c442-42a8-8041-6c1b6aaf0746-host-kubelet\") pod \"ovnkube-node-7wb4x\" (UID: \"a2ef8269-c442-42a8-8041-6c1b6aaf0746\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wb4x" Nov 02 06:16:52.999223 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.998679 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b65bb247-83f7-4749-9aa1-9b2677dd1b3d-original-pull-secret\") pod \"global-pull-secret-syncer-ggpsr\" (UID: \"b65bb247-83f7-4749-9aa1-9b2677dd1b3d\") " pod="kube-system/global-pull-secret-syncer-ggpsr" Nov 02 06:16:52.999223 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.998713 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/87e797a0-1da6-41ca-af8d-7382afaf3987-host-var-lib-cni-multus\") pod \"multus-rf7qp\" (UID: \"87e797a0-1da6-41ca-af8d-7382afaf3987\") " pod="openshift-multus/multus-rf7qp" Nov 02 06:16:52.999223 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.998681 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/37982aa4-bdbd-40e8-a2dd-5094209f3c96-service-ca-bundle\") pod \"insights-operator-7c4986d449-c8dcq\" (UID: \"37982aa4-bdbd-40e8-a2dd-5094209f3c96\") " pod="openshift-insights/insights-operator-7c4986d449-c8dcq" Nov 02 06:16:52.999223 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.998747 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45e5ce56-5cab-4546-8b31-02fc06cfa0cc-service-ca-bundle\") pod \"router-default-5f5c58667c-29wnm\" (UID: \"45e5ce56-5cab-4546-8b31-02fc06cfa0cc\") " pod="openshift-ingress/router-default-5f5c58667c-29wnm" Nov 02 06:16:52.999223 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.998779 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/81b393f2-42de-4a99-8114-1445de4d6dc8-lib-modules\") pod \"tuned-6nk8l\" (UID: \"81b393f2-42de-4a99-8114-1445de4d6dc8\") " pod="openshift-cluster-node-tuning-operator/tuned-6nk8l" Nov 02 06:16:52.999223 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.998806 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/81b393f2-42de-4a99-8114-1445de4d6dc8-tmp\") pod \"tuned-6nk8l\" (UID: \"81b393f2-42de-4a99-8114-1445de4d6dc8\") " pod="openshift-cluster-node-tuning-operator/tuned-6nk8l" Nov 02 06:16:52.999223 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.998815 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/37982aa4-bdbd-40e8-a2dd-5094209f3c96-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-c8dcq\" (UID: \"37982aa4-bdbd-40e8-a2dd-5094209f3c96\") " pod="openshift-insights/insights-operator-7c4986d449-c8dcq" Nov 02 06:16:52.999223 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.998848 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fa9c5a91-d480-4d7b-9044-be6437a76a02-registry-tls\") pod \"image-registry-746899d75-dtjhs\" (UID: \"fa9c5a91-d480-4d7b-9044-be6437a76a02\") " pod="openshift-image-registry/image-registry-746899d75-dtjhs" Nov 02 06:16:52.999223 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.998872 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fa9c5a91-d480-4d7b-9044-be6437a76a02-installation-pull-secrets\") pod \"image-registry-746899d75-dtjhs\" (UID: \"fa9c5a91-d480-4d7b-9044-be6437a76a02\") " pod="openshift-image-registry/image-registry-746899d75-dtjhs" Nov 02 06:16:52.999223 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.998895 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3500e682-2f15-45a2-92d6-bbc35dae3231-device-dir\") pod \"aws-ebs-csi-driver-node-cbzvd\" (UID: \"3500e682-2f15-45a2-92d6-bbc35dae3231\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cbzvd" Nov 02 06:16:52.999223 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.998895 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/81b393f2-42de-4a99-8114-1445de4d6dc8-tmp\") pod \"tuned-6nk8l\" (UID: \"81b393f2-42de-4a99-8114-1445de4d6dc8\") " pod="openshift-cluster-node-tuning-operator/tuned-6nk8l" Nov 02 06:16:52.999223 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.998930 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45e5ce56-5cab-4546-8b31-02fc06cfa0cc-service-ca-bundle\") pod \"router-default-5f5c58667c-29wnm\" (UID: \"45e5ce56-5cab-4546-8b31-02fc06cfa0cc\") " pod="openshift-ingress/router-default-5f5c58667c-29wnm" Nov 02 06:16:52.999223 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.998941 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/87e797a0-1da6-41ca-af8d-7382afaf3987-system-cni-dir\") pod \"multus-rf7qp\" (UID: \"87e797a0-1da6-41ca-af8d-7382afaf3987\") " pod="openshift-multus/multus-rf7qp" Nov 02 06:16:52.999223 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.998979 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vz9mf\" (UniqueName: \"kubernetes.io/projected/3419bd74-22aa-4841-87d2-c4340e01dae9-kube-api-access-vz9mf\") pod \"iptables-alerter-6fnx8\" (UID: \"3419bd74-22aa-4841-87d2-c4340e01dae9\") " pod="openshift-network-operator/iptables-alerter-6fnx8" Nov 02 06:16:52.999223 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.998995 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b65bb247-83f7-4749-9aa1-9b2677dd1b3d-original-pull-secret\") pod \"global-pull-secret-syncer-ggpsr\" (UID: \"b65bb247-83f7-4749-9aa1-9b2677dd1b3d\") " pod="kube-system/global-pull-secret-syncer-ggpsr" Nov 02 06:16:52.999223 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.999000 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a2ef8269-c442-42a8-8041-6c1b6aaf0746-host-cni-netd\") pod \"ovnkube-node-7wb4x\" (UID: \"a2ef8269-c442-42a8-8041-6c1b6aaf0746\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wb4x" Nov 02 06:16:52.999223 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.999039 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3500e682-2f15-45a2-92d6-bbc35dae3231-registration-dir\") pod \"aws-ebs-csi-driver-node-cbzvd\" (UID: \"3500e682-2f15-45a2-92d6-bbc35dae3231\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cbzvd" Nov 02 06:16:52.999223 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.999090 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f0531b4d-c954-4830-85f6-fe876c76437a-registry-certificates\") pod \"image-registry-6689c6d46b-vw558\" (UID: \"f0531b4d-c954-4830-85f6-fe876c76437a\") " pod="openshift-image-registry/image-registry-6689c6d46b-vw558" Nov 02 06:16:52.999223 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.999116 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/81b393f2-42de-4a99-8114-1445de4d6dc8-etc-kubernetes\") pod \"tuned-6nk8l\" (UID: \"81b393f2-42de-4a99-8114-1445de4d6dc8\") " pod="openshift-cluster-node-tuning-operator/tuned-6nk8l" Nov 02 06:16:52.999223 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.999150 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/81b393f2-42de-4a99-8114-1445de4d6dc8-etc-sysctl-d\") pod \"tuned-6nk8l\" (UID: \"81b393f2-42de-4a99-8114-1445de4d6dc8\") " pod="openshift-cluster-node-tuning-operator/tuned-6nk8l" Nov 02 06:16:52.999223 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.999181 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/37982aa4-bdbd-40e8-a2dd-5094209f3c96-serving-cert\") pod \"insights-operator-7c4986d449-c8dcq\" (UID: \"37982aa4-bdbd-40e8-a2dd-5094209f3c96\") " pod="openshift-insights/insights-operator-7c4986d449-c8dcq" Nov 02 06:16:52.999223 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.999210 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0d2b53c4-9d39-47fc-9367-c9a8e7f5da81-cert\") pod \"ingress-canary-6tt27\" (UID: \"0d2b53c4-9d39-47fc-9367-c9a8e7f5da81\") " pod="openshift-ingress-canary/ingress-canary-6tt27" Nov 02 06:16:53.000164 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.999296 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tz6t\" (UniqueName: \"kubernetes.io/projected/17dee72b-2f47-4016-8ffe-3a89934e66ee-kube-api-access-9tz6t\") pod \"console-operator-7cd56b8dbc-csfhg\" (UID: \"17dee72b-2f47-4016-8ffe-3a89934e66ee\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-csfhg" Nov 02 06:16:53.000164 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.999316 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3500e682-2f15-45a2-92d6-bbc35dae3231-sys-fs\") pod \"aws-ebs-csi-driver-node-cbzvd\" (UID: \"3500e682-2f15-45a2-92d6-bbc35dae3231\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cbzvd" Nov 02 06:16:53.000164 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.999332 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a2ef8269-c442-42a8-8041-6c1b6aaf0746-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7wb4x\" (UID: \"a2ef8269-c442-42a8-8041-6c1b6aaf0746\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wb4x" Nov 02 06:16:53.000164 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.999351 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a2ef8269-c442-42a8-8041-6c1b6aaf0746-ovn-node-metrics-cert\") pod \"ovnkube-node-7wb4x\" (UID: \"a2ef8269-c442-42a8-8041-6c1b6aaf0746\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wb4x" Nov 02 06:16:53.000164 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.999368 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a2ef8269-c442-42a8-8041-6c1b6aaf0746-host-slash\") pod \"ovnkube-node-7wb4x\" (UID: \"a2ef8269-c442-42a8-8041-6c1b6aaf0746\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wb4x" Nov 02 06:16:53.000164 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.999409 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rd8hq\" (UniqueName: \"kubernetes.io/projected/cc33996a-0d4c-49c4-ba67-283bd6a2de14-kube-api-access-rd8hq\") pod \"service-ca-operator-544d749c8c-wcrtl\" (UID: \"cc33996a-0d4c-49c4-ba67-283bd6a2de14\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wcrtl" Nov 02 06:16:53.000164 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.999445 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a4319db2-eda9-47a0-b742-985c8bdd7ea2-service-ca-bundle\") pod \"periodic-gathering-rrgb2-xp5b2\" (UID: \"a4319db2-eda9-47a0-b742-985c8bdd7ea2\") " pod="openshift-insights/periodic-gathering-rrgb2-xp5b2" Nov 02 06:16:53.000164 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.999554 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/17dee72b-2f47-4016-8ffe-3a89934e66ee-serving-cert\") pod \"console-operator-7cd56b8dbc-csfhg\" (UID: \"17dee72b-2f47-4016-8ffe-3a89934e66ee\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-csfhg" Nov 02 06:16:53.000164 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.999565 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/37982aa4-bdbd-40e8-a2dd-5094209f3c96-serving-cert\") pod \"insights-operator-7c4986d449-c8dcq\" (UID: \"37982aa4-bdbd-40e8-a2dd-5094209f3c96\") " pod="openshift-insights/insights-operator-7c4986d449-c8dcq" Nov 02 06:16:53.000164 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.999614 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f0531b4d-c954-4830-85f6-fe876c76437a-image-registry-private-configuration\") pod \"image-registry-6689c6d46b-vw558\" (UID: \"f0531b4d-c954-4830-85f6-fe876c76437a\") " pod="openshift-image-registry/image-registry-6689c6d46b-vw558" Nov 02 06:16:53.000164 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.999651 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m89g8\" (UniqueName: \"kubernetes.io/projected/f0531b4d-c954-4830-85f6-fe876c76437a-kube-api-access-m89g8\") pod \"image-registry-6689c6d46b-vw558\" (UID: \"f0531b4d-c954-4830-85f6-fe876c76437a\") " pod="openshift-image-registry/image-registry-6689c6d46b-vw558" Nov 02 06:16:53.000164 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.999679 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dv9xj\" (UniqueName: \"kubernetes.io/projected/4ba9b2b6-4be9-4303-8eab-cd464c5a32c7-kube-api-access-dv9xj\") pod \"dns-default-gf726\" (UID: \"4ba9b2b6-4be9-4303-8eab-cd464c5a32c7\") " pod="openshift-dns/dns-default-gf726" Nov 02 06:16:53.000164 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.999690 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a2ef8269-c442-42a8-8041-6c1b6aaf0746-ovn-node-metrics-cert\") pod \"ovnkube-node-7wb4x\" (UID: \"a2ef8269-c442-42a8-8041-6c1b6aaf0746\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wb4x" Nov 02 06:16:53.000164 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.999711 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/87e797a0-1da6-41ca-af8d-7382afaf3987-hostroot\") pod \"multus-rf7qp\" (UID: \"87e797a0-1da6-41ca-af8d-7382afaf3987\") " pod="openshift-multus/multus-rf7qp" Nov 02 06:16:53.000164 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.999737 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/81b393f2-42de-4a99-8114-1445de4d6dc8-etc-sysctl-conf\") pod \"tuned-6nk8l\" (UID: \"81b393f2-42de-4a99-8114-1445de4d6dc8\") " pod="openshift-cluster-node-tuning-operator/tuned-6nk8l" Nov 02 06:16:53.000164 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.999761 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/81b393f2-42de-4a99-8114-1445de4d6dc8-etc-systemd\") pod \"tuned-6nk8l\" (UID: \"81b393f2-42de-4a99-8114-1445de4d6dc8\") " pod="openshift-cluster-node-tuning-operator/tuned-6nk8l" Nov 02 06:16:53.000164 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.999816 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/81b393f2-42de-4a99-8114-1445de4d6dc8-etc-tuned\") pod \"tuned-6nk8l\" (UID: \"81b393f2-42de-4a99-8114-1445de4d6dc8\") " pod="openshift-cluster-node-tuning-operator/tuned-6nk8l" Nov 02 06:16:53.000164 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.999860 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3419bd74-22aa-4841-87d2-c4340e01dae9-host-slash\") pod \"iptables-alerter-6fnx8\" (UID: \"3419bd74-22aa-4841-87d2-c4340e01dae9\") " pod="openshift-network-operator/iptables-alerter-6fnx8" Nov 02 06:16:53.000164 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.999866 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/81b393f2-42de-4a99-8114-1445de4d6dc8-etc-tuned\") pod \"tuned-6nk8l\" (UID: \"81b393f2-42de-4a99-8114-1445de4d6dc8\") " pod="openshift-cluster-node-tuning-operator/tuned-6nk8l" Nov 02 06:16:53.000164 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.999937 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a4319db2-eda9-47a0-b742-985c8bdd7ea2-service-ca-bundle\") pod \"periodic-gathering-rrgb2-xp5b2\" (UID: \"a4319db2-eda9-47a0-b742-985c8bdd7ea2\") " pod="openshift-insights/periodic-gathering-rrgb2-xp5b2" Nov 02 06:16:53.000164 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.999937 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a2ef8269-c442-42a8-8041-6c1b6aaf0746-host-run-ovn-kubernetes\") pod \"ovnkube-node-7wb4x\" (UID: \"a2ef8269-c442-42a8-8041-6c1b6aaf0746\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wb4x" Nov 02 06:16:53.000164 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:52.999992 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fa9c5a91-d480-4d7b-9044-be6437a76a02-registry-certificates\") pod \"image-registry-746899d75-dtjhs\" (UID: \"fa9c5a91-d480-4d7b-9044-be6437a76a02\") " pod="openshift-image-registry/image-registry-746899d75-dtjhs" Nov 02 06:16:53.000164 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.000025 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f0531b4d-c954-4830-85f6-fe876c76437a-bound-sa-token\") pod \"image-registry-6689c6d46b-vw558\" (UID: \"f0531b4d-c954-4830-85f6-fe876c76437a\") " pod="openshift-image-registry/image-registry-6689c6d46b-vw558" Nov 02 06:16:53.000164 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.000055 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d4615a9e-da82-4f8b-8225-d60836b1ba80-agent-certs\") pod \"konnectivity-agent-hwc99\" (UID: \"d4615a9e-da82-4f8b-8225-d60836b1ba80\") " pod="kube-system/konnectivity-agent-hwc99" Nov 02 06:16:53.000164 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.000083 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a2ef8269-c442-42a8-8041-6c1b6aaf0746-var-lib-openvswitch\") pod \"ovnkube-node-7wb4x\" (UID: \"a2ef8269-c442-42a8-8041-6c1b6aaf0746\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wb4x" Nov 02 06:16:53.000164 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.000112 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzxsj\" (UniqueName: \"kubernetes.io/projected/a2ef8269-c442-42a8-8041-6c1b6aaf0746-kube-api-access-qzxsj\") pod \"ovnkube-node-7wb4x\" (UID: \"a2ef8269-c442-42a8-8041-6c1b6aaf0746\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wb4x" Nov 02 06:16:53.000164 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.000153 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/87e797a0-1da6-41ca-af8d-7382afaf3987-host-var-lib-kubelet\") pod \"multus-rf7qp\" (UID: \"87e797a0-1da6-41ca-af8d-7382afaf3987\") " pod="openshift-multus/multus-rf7qp" Nov 02 06:16:53.001121 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.000205 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrq9j\" (UniqueName: \"kubernetes.io/projected/81b393f2-42de-4a99-8114-1445de4d6dc8-kube-api-access-hrq9j\") pod \"tuned-6nk8l\" (UID: \"81b393f2-42de-4a99-8114-1445de4d6dc8\") " pod="openshift-cluster-node-tuning-operator/tuned-6nk8l" Nov 02 06:16:53.001121 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.000238 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3419bd74-22aa-4841-87d2-c4340e01dae9-iptables-alerter-script\") pod \"iptables-alerter-6fnx8\" (UID: \"3419bd74-22aa-4841-87d2-c4340e01dae9\") " pod="openshift-network-operator/iptables-alerter-6fnx8" Nov 02 06:16:53.001121 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.000265 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txffk\" (UniqueName: \"kubernetes.io/projected/dee3b502-2562-4b21-bc46-2031580c4b0a-kube-api-access-txffk\") pod \"kas-connection-checker-685895c7c9-jrtnt\" (UID: \"dee3b502-2562-4b21-bc46-2031580c4b0a\") " pod="kube-system/kas-connection-checker-685895c7c9-jrtnt" Nov 02 06:16:53.001121 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.000308 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d4615a9e-da82-4f8b-8225-d60836b1ba80-agent-certs\") pod \"konnectivity-agent-hwc99\" (UID: \"d4615a9e-da82-4f8b-8225-d60836b1ba80\") " pod="kube-system/konnectivity-agent-hwc99" Nov 02 06:16:53.001121 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.000325 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/cf8e8e65-8525-484b-adfe-f56517ccdc1b-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-d6bc4\" (UID: \"cf8e8e65-8525-484b-adfe-f56517ccdc1b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-d6bc4" Nov 02 06:16:53.001121 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.000357 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwqbt\" (UniqueName: \"kubernetes.io/projected/cf8e8e65-8525-484b-adfe-f56517ccdc1b-kube-api-access-vwqbt\") pod \"cluster-samples-operator-6ddb6c8b94-d6bc4\" (UID: \"cf8e8e65-8525-484b-adfe-f56517ccdc1b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-d6bc4" Nov 02 06:16:53.001121 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.000386 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3500e682-2f15-45a2-92d6-bbc35dae3231-socket-dir\") pod \"aws-ebs-csi-driver-node-cbzvd\" (UID: \"3500e682-2f15-45a2-92d6-bbc35dae3231\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cbzvd" Nov 02 06:16:53.001121 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.000439 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d8d87317-3f41-46d4-a58e-3c7601e0d445-hosts-file\") pod \"node-resolver-2p95f\" (UID: \"d8d87317-3f41-46d4-a58e-3c7601e0d445\") " pod="openshift-dns/node-resolver-2p95f" Nov 02 06:16:53.001121 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.000472 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f0531b4d-c954-4830-85f6-fe876c76437a-installation-pull-secrets\") pod \"image-registry-6689c6d46b-vw558\" (UID: \"f0531b4d-c954-4830-85f6-fe876c76437a\") " pod="openshift-image-registry/image-registry-6689c6d46b-vw558" Nov 02 06:16:53.001121 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.000510 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/45e5ce56-5cab-4546-8b31-02fc06cfa0cc-default-certificate\") pod \"router-default-5f5c58667c-29wnm\" (UID: \"45e5ce56-5cab-4546-8b31-02fc06cfa0cc\") " pod="openshift-ingress/router-default-5f5c58667c-29wnm" Nov 02 06:16:53.001121 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.000537 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d4615a9e-da82-4f8b-8225-d60836b1ba80-konnectivity-ca\") pod \"konnectivity-agent-hwc99\" (UID: \"d4615a9e-da82-4f8b-8225-d60836b1ba80\") " pod="kube-system/konnectivity-agent-hwc99" Nov 02 06:16:53.001121 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.000555 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwdcw\" (UniqueName: \"kubernetes.io/projected/a4319db2-eda9-47a0-b742-985c8bdd7ea2-kube-api-access-qwdcw\") pod \"periodic-gathering-rrgb2-xp5b2\" (UID: \"a4319db2-eda9-47a0-b742-985c8bdd7ea2\") " pod="openshift-insights/periodic-gathering-rrgb2-xp5b2" Nov 02 06:16:53.001121 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.000571 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/17dee72b-2f47-4016-8ffe-3a89934e66ee-config\") pod \"console-operator-7cd56b8dbc-csfhg\" (UID: \"17dee72b-2f47-4016-8ffe-3a89934e66ee\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-csfhg" Nov 02 06:16:53.001121 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.000587 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/87e797a0-1da6-41ca-af8d-7382afaf3987-cni-binary-copy\") pod \"multus-rf7qp\" (UID: \"87e797a0-1da6-41ca-af8d-7382afaf3987\") " pod="openshift-multus/multus-rf7qp" Nov 02 06:16:53.001121 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.000607 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b6432a8-b0ce-4954-8e3d-8eb8782557eb-metrics-certs\") pod \"network-metrics-daemon-sb4nt\" (UID: \"3b6432a8-b0ce-4954-8e3d-8eb8782557eb\") " pod="openshift-multus/network-metrics-daemon-sb4nt" Nov 02 06:16:53.001121 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.000624 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a2ef8269-c442-42a8-8041-6c1b6aaf0746-etc-openvswitch\") pod \"ovnkube-node-7wb4x\" (UID: \"a2ef8269-c442-42a8-8041-6c1b6aaf0746\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wb4x" Nov 02 06:16:53.001121 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.000642 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f14e8781-bf82-4810-9909-c126ccdc1ce1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ckddw\" (UID: \"f14e8781-bf82-4810-9909-c126ccdc1ce1\") " pod="openshift-multus/multus-additional-cni-plugins-ckddw" Nov 02 06:16:53.001121 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.000666 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f14e8781-bf82-4810-9909-c126ccdc1ce1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ckddw\" (UID: \"f14e8781-bf82-4810-9909-c126ccdc1ce1\") " pod="openshift-multus/multus-additional-cni-plugins-ckddw" Nov 02 06:16:53.001121 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.000685 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kcrm\" (UniqueName: \"kubernetes.io/projected/d8d87317-3f41-46d4-a58e-3c7601e0d445-kube-api-access-5kcrm\") pod \"node-resolver-2p95f\" (UID: \"d8d87317-3f41-46d4-a58e-3c7601e0d445\") " pod="openshift-dns/node-resolver-2p95f" Nov 02 06:16:53.001121 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.000701 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/87e797a0-1da6-41ca-af8d-7382afaf3987-host-var-lib-cni-bin\") pod \"multus-rf7qp\" (UID: \"87e797a0-1da6-41ca-af8d-7382afaf3987\") " pod="openshift-multus/multus-rf7qp" Nov 02 06:16:53.001121 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.000723 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/45e5ce56-5cab-4546-8b31-02fc06cfa0cc-stats-auth\") pod \"router-default-5f5c58667c-29wnm\" (UID: \"45e5ce56-5cab-4546-8b31-02fc06cfa0cc\") " pod="openshift-ingress/router-default-5f5c58667c-29wnm" Nov 02 06:16:53.001121 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.000778 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7l5t\" (UniqueName: \"kubernetes.io/projected/37982aa4-bdbd-40e8-a2dd-5094209f3c96-kube-api-access-h7l5t\") pod \"insights-operator-7c4986d449-c8dcq\" (UID: \"37982aa4-bdbd-40e8-a2dd-5094209f3c96\") " pod="openshift-insights/insights-operator-7c4986d449-c8dcq" Nov 02 06:16:53.001121 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.000781 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/45e5ce56-5cab-4546-8b31-02fc06cfa0cc-default-certificate\") pod \"router-default-5f5c58667c-29wnm\" (UID: \"45e5ce56-5cab-4546-8b31-02fc06cfa0cc\") " pod="openshift-ingress/router-default-5f5c58667c-29wnm" Nov 02 06:16:53.001121 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.000799 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/cc33996a-0d4c-49c4-ba67-283bd6a2de14-config\") pod \"service-ca-operator-544d749c8c-wcrtl\" (UID: \"cc33996a-0d4c-49c4-ba67-283bd6a2de14\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wcrtl" Nov 02 06:16:53.001121 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.000865 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b6432a8-b0ce-4954-8e3d-8eb8782557eb-metrics-certs\") pod \"network-metrics-daemon-sb4nt\" (UID: \"3b6432a8-b0ce-4954-8e3d-8eb8782557eb\") " pod="openshift-multus/network-metrics-daemon-sb4nt" Nov 02 06:16:53.001121 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.000957 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/cf8e8e65-8525-484b-adfe-f56517ccdc1b-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-d6bc4\" (UID: \"cf8e8e65-8525-484b-adfe-f56517ccdc1b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-d6bc4" Nov 02 06:16:53.001121 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.000985 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/87e797a0-1da6-41ca-af8d-7382afaf3987-cnibin\") pod \"multus-rf7qp\" (UID: \"87e797a0-1da6-41ca-af8d-7382afaf3987\") " pod="openshift-multus/multus-rf7qp" Nov 02 06:16:53.001121 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.000986 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/45e5ce56-5cab-4546-8b31-02fc06cfa0cc-stats-auth\") pod \"router-default-5f5c58667c-29wnm\" (UID: \"45e5ce56-5cab-4546-8b31-02fc06cfa0cc\") " pod="openshift-ingress/router-default-5f5c58667c-29wnm" Nov 02 06:16:53.001121 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.001076 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/cc33996a-0d4c-49c4-ba67-283bd6a2de14-config\") pod \"service-ca-operator-544d749c8c-wcrtl\" (UID: \"cc33996a-0d4c-49c4-ba67-283bd6a2de14\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wcrtl" Nov 02 06:16:53.001930 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.001196 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/cf8e8e65-8525-484b-adfe-f56517ccdc1b-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-d6bc4\" (UID: \"cf8e8e65-8525-484b-adfe-f56517ccdc1b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-d6bc4" Nov 02 06:16:53.008291 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.008274 5916 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 02 06:16:53.034100 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.034073 5916 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 02 06:16:53.034197 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.034103 5916 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 02 06:16:53.034197 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.034187 5916 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 02 06:16:53.034342 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.034328 5916 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Nov 02 06:16:53.034508 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.034493 5916 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 02 06:16:53.034548 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.034520 5916 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 02 06:16:53.034659 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.034638 5916 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 02 06:16:53.034944 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.034929 5916 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 02 06:16:53.035000 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.034962 5916 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 02 06:16:53.035000 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.034987 5916 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Nov 02 06:16:53.035107 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.035045 5916 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 02 06:16:53.035153 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.035135 5916 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Nov 02 06:16:53.035370 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.035356 5916 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Nov 02 06:16:53.035413 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.035394 5916 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 02 06:16:53.035511 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.035497 5916 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 02 06:16:53.035568 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.035501 5916 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 02 06:16:53.035603 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.035568 5916 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Nov 02 06:16:53.035709 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.035693 5916 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Nov 02 06:16:53.035760 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.035695 5916 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Nov 02 06:16:53.035760 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.035727 5916 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 02 06:16:53.035760 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.035739 5916 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 02 06:16:53.035890 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.035783 5916 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 02 06:16:53.035890 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.035857 5916 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 02 06:16:53.035992 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.035938 5916 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 02 06:16:53.036036 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.035991 5916 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 02 06:16:53.036077 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.036033 5916 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Nov 02 06:16:53.036077 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.036055 5916 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 02 06:16:53.036077 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.036056 5916 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 02 06:16:53.036266 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.036117 5916 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Nov 02 06:16:53.036266 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.036123 5916 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 02 06:16:53.036266 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.036136 5916 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 02 06:16:53.036266 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.036191 5916 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Nov 02 06:16:53.036266 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.036202 5916 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 02 06:16:53.036266 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.036224 5916 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Nov 02 06:16:53.036580 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.036314 5916 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Nov 02 06:16:53.036580 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.036318 5916 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Nov 02 06:16:53.036580 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.036339 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/87e797a0-1da6-41ca-af8d-7382afaf3987-multus-daemon-config\") pod \"multus-rf7qp\" (UID: \"87e797a0-1da6-41ca-af8d-7382afaf3987\") " pod="openshift-multus/multus-rf7qp" Nov 02 06:16:53.036580 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.036372 5916 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 02 06:16:53.036580 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.036382 5916 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 02 06:16:53.036580 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.036387 5916 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Nov 02 06:16:53.036580 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.036467 5916 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 02 06:16:53.036580 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.036540 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4ba9b2b6-4be9-4303-8eab-cd464c5a32c7-config-volume\") pod \"dns-default-gf726\" (UID: \"4ba9b2b6-4be9-4303-8eab-cd464c5a32c7\") " pod="openshift-dns/dns-default-gf726" Nov 02 06:16:53.036794 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.036602 5916 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 02 06:16:53.036794 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.036692 5916 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 02 06:16:53.036973 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.036956 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a2ef8269-c442-42a8-8041-6c1b6aaf0746-env-overrides\") pod \"ovnkube-node-7wb4x\" (UID: \"a2ef8269-c442-42a8-8041-6c1b6aaf0746\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wb4x" Nov 02 06:16:53.037574 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.037559 5916 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Nov 02 06:16:53.037738 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.037721 5916 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Nov 02 06:16:53.037945 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.037929 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3eec1082-df0b-420b-b4ff-5a1c0fab1d1d-serviceca\") pod \"node-ca-qm6wn\" (UID: \"3eec1082-df0b-420b-b4ff-5a1c0fab1d1d\") " pod="openshift-image-registry/node-ca-qm6wn" Nov 02 06:16:53.038113 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.038098 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a2ef8269-c442-42a8-8041-6c1b6aaf0746-ovnkube-script-lib\") pod \"ovnkube-node-7wb4x\" (UID: \"a2ef8269-c442-42a8-8041-6c1b6aaf0746\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wb4x" Nov 02 06:16:53.038168 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.038151 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a2ef8269-c442-42a8-8041-6c1b6aaf0746-ovnkube-config\") pod \"ovnkube-node-7wb4x\" (UID: \"a2ef8269-c442-42a8-8041-6c1b6aaf0746\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wb4x" Nov 02 06:16:53.038203 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.038195 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f14e8781-bf82-4810-9909-c126ccdc1ce1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ckddw\" (UID: \"f14e8781-bf82-4810-9909-c126ccdc1ce1\") " pod="openshift-multus/multus-additional-cni-plugins-ckddw" Nov 02 06:16:53.038246 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.038239 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f14e8781-bf82-4810-9909-c126ccdc1ce1-cni-binary-copy\") pod \"multus-additional-cni-plugins-ckddw\" (UID: \"f14e8781-bf82-4810-9909-c126ccdc1ce1\") " pod="openshift-multus/multus-additional-cni-plugins-ckddw" Nov 02 06:16:53.039440 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.039425 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f0531b4d-c954-4830-85f6-fe876c76437a-registry-certificates\") pod \"image-registry-6689c6d46b-vw558\" (UID: \"f0531b4d-c954-4830-85f6-fe876c76437a\") " pod="openshift-image-registry/image-registry-6689c6d46b-vw558" Nov 02 06:16:53.040520 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.040503 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/cf8e8e65-8525-484b-adfe-f56517ccdc1b-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-d6bc4\" (UID: \"cf8e8e65-8525-484b-adfe-f56517ccdc1b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-d6bc4" Nov 02 06:16:53.040568 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.040512 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fa9c5a91-d480-4d7b-9044-be6437a76a02-registry-certificates\") pod \"image-registry-746899d75-dtjhs\" (UID: \"fa9c5a91-d480-4d7b-9044-be6437a76a02\") " pod="openshift-image-registry/image-registry-746899d75-dtjhs" Nov 02 06:16:53.040568 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.040558 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3419bd74-22aa-4841-87d2-c4340e01dae9-iptables-alerter-script\") pod \"iptables-alerter-6fnx8\" (UID: \"3419bd74-22aa-4841-87d2-c4340e01dae9\") " pod="openshift-network-operator/iptables-alerter-6fnx8" Nov 02 06:16:53.041668 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.041648 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f14e8781-bf82-4810-9909-c126ccdc1ce1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ckddw\" (UID: \"f14e8781-bf82-4810-9909-c126ccdc1ce1\") " pod="openshift-multus/multus-additional-cni-plugins-ckddw" Nov 02 06:16:53.041728 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.041661 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d4615a9e-da82-4f8b-8225-d60836b1ba80-konnectivity-ca\") pod \"konnectivity-agent-hwc99\" (UID: \"d4615a9e-da82-4f8b-8225-d60836b1ba80\") " pod="kube-system/konnectivity-agent-hwc99" Nov 02 06:16:53.041728 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.041703 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/17dee72b-2f47-4016-8ffe-3a89934e66ee-config\") pod \"console-operator-7cd56b8dbc-csfhg\" (UID: \"17dee72b-2f47-4016-8ffe-3a89934e66ee\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-csfhg" Nov 02 06:16:53.041795 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.041703 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/87e797a0-1da6-41ca-af8d-7382afaf3987-cni-binary-copy\") pod \"multus-rf7qp\" (UID: \"87e797a0-1da6-41ca-af8d-7382afaf3987\") " pod="openshift-multus/multus-rf7qp" Nov 02 06:16:53.046579 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.046540 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/53aa1e5a-c35f-4aa6-9f23-cc61ccd331fc-nginx-conf\") pod \"networking-console-plugin-86c478bb88-fh4sz\" (UID: \"53aa1e5a-c35f-4aa6-9f23-cc61ccd331fc\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-fh4sz" Nov 02 06:16:53.046652 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.046632 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fa9c5a91-d480-4d7b-9044-be6437a76a02-trusted-ca\") pod \"image-registry-746899d75-dtjhs\" (UID: \"fa9c5a91-d480-4d7b-9044-be6437a76a02\") " pod="openshift-image-registry/image-registry-746899d75-dtjhs" Nov 02 06:16:53.048099 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.048076 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f0531b4d-c954-4830-85f6-fe876c76437a-trusted-ca\") pod \"image-registry-6689c6d46b-vw558\" (UID: \"f0531b4d-c954-4830-85f6-fe876c76437a\") " pod="openshift-image-registry/image-registry-6689c6d46b-vw558" Nov 02 06:16:53.052202 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.048713 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/17dee72b-2f47-4016-8ffe-3a89934e66ee-trusted-ca\") pod \"console-operator-7cd56b8dbc-csfhg\" (UID: \"17dee72b-2f47-4016-8ffe-3a89934e66ee\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-csfhg" Nov 02 06:16:53.053168 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.053147 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnq6k\" (UniqueName: \"kubernetes.io/projected/3ced47b0-9cd0-4710-8f75-2a333437ed01-kube-api-access-gnq6k\") pod \"network-check-target-ft7gs\" (UID: \"3ced47b0-9cd0-4710-8f75-2a333437ed01\") " pod="openshift-network-diagnostics/network-check-target-ft7gs" Nov 02 06:16:53.053259 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.053192 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mplc\" (UniqueName: \"kubernetes.io/projected/0d2b53c4-9d39-47fc-9367-c9a8e7f5da81-kube-api-access-8mplc\") pod \"ingress-canary-6tt27\" (UID: \"0d2b53c4-9d39-47fc-9367-c9a8e7f5da81\") " pod="openshift-ingress-canary/ingress-canary-6tt27" Nov 02 06:16:53.054566 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.054547 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8lqt\" (UniqueName: \"kubernetes.io/projected/3500e682-2f15-45a2-92d6-bbc35dae3231-kube-api-access-v8lqt\") pod \"aws-ebs-csi-driver-node-cbzvd\" (UID: \"3500e682-2f15-45a2-92d6-bbc35dae3231\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cbzvd" Nov 02 06:16:53.054642 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.054587 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7xzh\" (UniqueName: \"kubernetes.io/projected/f14e8781-bf82-4810-9909-c126ccdc1ce1-kube-api-access-v7xzh\") pod \"multus-additional-cni-plugins-ckddw\" (UID: \"f14e8781-bf82-4810-9909-c126ccdc1ce1\") " pod="openshift-multus/multus-additional-cni-plugins-ckddw" Nov 02 06:16:53.054806 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.054788 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5989k\" (UniqueName: \"kubernetes.io/projected/20cb7690-c976-471c-b459-a9ab15cc51bf-kube-api-access-5989k\") pod \"network-check-source-6f78d59f99-k9hcs\" (UID: \"20cb7690-c976-471c-b459-a9ab15cc51bf\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-k9hcs" Nov 02 06:16:53.054894 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.054876 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnw94\" (UniqueName: \"kubernetes.io/projected/121e38c0-6fa3-46aa-9191-5c6f91c8764c-kube-api-access-fnw94\") pod \"volume-data-source-validator-58c9f5f46d-8fbcn\" (UID: \"121e38c0-6fa3-46aa-9191-5c6f91c8764c\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-8fbcn" Nov 02 06:16:53.055346 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.055330 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 02 06:16:53.055565 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.055521 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-d5zsv\"" Nov 02 06:16:53.055565 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.055559 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-nxd68\"" Nov 02 06:16:53.055663 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.055561 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Nov 02 06:16:53.055663 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.055580 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 02 06:16:53.055663 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.055527 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-2646z\"" Nov 02 06:16:53.055663 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.055586 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-xffwp\"" Nov 02 06:16:53.055826 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.055684 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-77l82\"" Nov 02 06:16:53.055826 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.055694 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Nov 02 06:16:53.055826 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.055714 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-mf7sr\"" Nov 02 06:16:53.055826 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.055751 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Nov 02 06:16:53.055826 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.055757 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Nov 02 06:16:53.055826 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.055761 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-75dck\"" Nov 02 06:16:53.055826 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.055784 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-kqjnz\"" Nov 02 06:16:53.056046 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.055828 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-rkkc6\"" Nov 02 06:16:53.056046 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.055841 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-rpvjg\"" Nov 02 06:16:53.056046 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.055861 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 02 06:16:53.056046 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.055872 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-v49gc\"" Nov 02 06:16:53.056046 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.055899 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Nov 02 06:16:53.057119 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.057105 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f0531b4d-c954-4830-85f6-fe876c76437a-registry-tls\") pod \"image-registry-6689c6d46b-vw558\" (UID: \"f0531b4d-c954-4830-85f6-fe876c76437a\") " pod="openshift-image-registry/image-registry-6689c6d46b-vw558" Nov 02 06:16:53.057164 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.057131 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4ba9b2b6-4be9-4303-8eab-cd464c5a32c7-metrics-tls\") pod \"dns-default-gf726\" (UID: \"4ba9b2b6-4be9-4303-8eab-cd464c5a32c7\") " pod="openshift-dns/dns-default-gf726" Nov 02 06:16:53.057929 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.057890 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fa9c5a91-d480-4d7b-9044-be6437a76a02-image-registry-private-configuration\") pod \"image-registry-746899d75-dtjhs\" (UID: \"fa9c5a91-d480-4d7b-9044-be6437a76a02\") " pod="openshift-image-registry/image-registry-746899d75-dtjhs" Nov 02 06:16:53.058386 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.058361 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmsjq\" (UniqueName: \"kubernetes.io/projected/12b6cc1c-537d-4125-92ff-fe5f6927c53e-kube-api-access-bmsjq\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-sktt2\" (UID: \"12b6cc1c-537d-4125-92ff-fe5f6927c53e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-sktt2" Nov 02 06:16:53.058466 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.058408 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8vwr\" (UniqueName: \"kubernetes.io/projected/3eec1082-df0b-420b-b4ff-5a1c0fab1d1d-kube-api-access-k8vwr\") pod \"node-ca-qm6wn\" (UID: \"3eec1082-df0b-420b-b4ff-5a1c0fab1d1d\") " pod="openshift-image-registry/node-ca-qm6wn" Nov 02 06:16:53.058466 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.058448 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvdp9\" (UniqueName: \"kubernetes.io/projected/3b6432a8-b0ce-4954-8e3d-8eb8782557eb-kube-api-access-pvdp9\") pod \"network-metrics-daemon-sb4nt\" (UID: \"3b6432a8-b0ce-4954-8e3d-8eb8782557eb\") " pod="openshift-multus/network-metrics-daemon-sb4nt" Nov 02 06:16:53.058580 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.058470 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6vx8\" (UniqueName: \"kubernetes.io/projected/87e797a0-1da6-41ca-af8d-7382afaf3987-kube-api-access-l6vx8\") pod \"multus-rf7qp\" (UID: \"87e797a0-1da6-41ca-af8d-7382afaf3987\") " pod="openshift-multus/multus-rf7qp" Nov 02 06:16:53.058580 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.058495 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrh7x\" (UniqueName: \"kubernetes.io/projected/8e8f2e60-de1b-4c8b-9285-56568c150d92-kube-api-access-zrh7x\") pod \"kas-connection-checker-685895c7c9-kd68m\" (UID: \"8e8f2e60-de1b-4c8b-9285-56568c150d92\") " pod="kube-system/kas-connection-checker-685895c7c9-kd68m" Nov 02 06:16:53.058580 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.058524 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvpvr\" (UniqueName: \"kubernetes.io/projected/45e5ce56-5cab-4546-8b31-02fc06cfa0cc-kube-api-access-xvpvr\") pod \"router-default-5f5c58667c-29wnm\" (UID: \"45e5ce56-5cab-4546-8b31-02fc06cfa0cc\") " pod="openshift-ingress/router-default-5f5c58667c-29wnm" Nov 02 06:16:53.058681 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.058577 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cctkp\" (UniqueName: \"kubernetes.io/projected/fa9c5a91-d480-4d7b-9044-be6437a76a02-kube-api-access-cctkp\") pod \"image-registry-746899d75-dtjhs\" (UID: \"fa9c5a91-d480-4d7b-9044-be6437a76a02\") " pod="openshift-image-registry/image-registry-746899d75-dtjhs" Nov 02 06:16:53.059103 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.059085 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fa9c5a91-d480-4d7b-9044-be6437a76a02-registry-tls\") pod \"image-registry-746899d75-dtjhs\" (UID: \"fa9c5a91-d480-4d7b-9044-be6437a76a02\") " pod="openshift-image-registry/image-registry-746899d75-dtjhs" Nov 02 06:16:53.059219 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.059206 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fa9c5a91-d480-4d7b-9044-be6437a76a02-installation-pull-secrets\") pod \"image-registry-746899d75-dtjhs\" (UID: \"fa9c5a91-d480-4d7b-9044-be6437a76a02\") " pod="openshift-image-registry/image-registry-746899d75-dtjhs" Nov 02 06:16:53.059432 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.059420 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0d2b53c4-9d39-47fc-9367-c9a8e7f5da81-cert\") pod \"ingress-canary-6tt27\" (UID: \"0d2b53c4-9d39-47fc-9367-c9a8e7f5da81\") " pod="openshift-ingress-canary/ingress-canary-6tt27" Nov 02 06:16:53.059979 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.059896 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fa9c5a91-d480-4d7b-9044-be6437a76a02-bound-sa-token\") pod \"image-registry-746899d75-dtjhs\" (UID: \"fa9c5a91-d480-4d7b-9044-be6437a76a02\") " pod="openshift-image-registry/image-registry-746899d75-dtjhs" Nov 02 06:16:53.059979 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.059937 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/17dee72b-2f47-4016-8ffe-3a89934e66ee-serving-cert\") pod \"console-operator-7cd56b8dbc-csfhg\" (UID: \"17dee72b-2f47-4016-8ffe-3a89934e66ee\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-csfhg" Nov 02 06:16:53.059979 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.059948 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f0531b4d-c954-4830-85f6-fe876c76437a-image-registry-private-configuration\") pod \"image-registry-6689c6d46b-vw558\" (UID: \"f0531b4d-c954-4830-85f6-fe876c76437a\") " pod="openshift-image-registry/image-registry-6689c6d46b-vw558" Nov 02 06:16:53.060113 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.059976 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dv9xj\" (UniqueName: \"kubernetes.io/projected/4ba9b2b6-4be9-4303-8eab-cd464c5a32c7-kube-api-access-dv9xj\") pod \"dns-default-gf726\" (UID: \"4ba9b2b6-4be9-4303-8eab-cd464c5a32c7\") " pod="openshift-dns/dns-default-gf726" Nov 02 06:16:53.060113 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.059979 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrq9j\" (UniqueName: \"kubernetes.io/projected/81b393f2-42de-4a99-8114-1445de4d6dc8-kube-api-access-hrq9j\") pod \"tuned-6nk8l\" (UID: \"81b393f2-42de-4a99-8114-1445de4d6dc8\") " pod="openshift-cluster-node-tuning-operator/tuned-6nk8l" Nov 02 06:16:53.060198 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.060175 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7l5t\" (UniqueName: \"kubernetes.io/projected/37982aa4-bdbd-40e8-a2dd-5094209f3c96-kube-api-access-h7l5t\") pod \"insights-operator-7c4986d449-c8dcq\" (UID: \"37982aa4-bdbd-40e8-a2dd-5094209f3c96\") " pod="openshift-insights/insights-operator-7c4986d449-c8dcq" Nov 02 06:16:53.060491 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.060472 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-txffk\" (UniqueName: \"kubernetes.io/projected/dee3b502-2562-4b21-bc46-2031580c4b0a-kube-api-access-txffk\") pod \"kas-connection-checker-685895c7c9-jrtnt\" (UID: \"dee3b502-2562-4b21-bc46-2031580c4b0a\") " pod="kube-system/kas-connection-checker-685895c7c9-jrtnt" Nov 02 06:16:53.060524 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.060490 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzxsj\" (UniqueName: \"kubernetes.io/projected/a2ef8269-c442-42a8-8041-6c1b6aaf0746-kube-api-access-qzxsj\") pod \"ovnkube-node-7wb4x\" (UID: \"a2ef8269-c442-42a8-8041-6c1b6aaf0746\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wb4x" Nov 02 06:16:53.060609 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.060594 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f0531b4d-c954-4830-85f6-fe876c76437a-bound-sa-token\") pod \"image-registry-6689c6d46b-vw558\" (UID: \"f0531b4d-c954-4830-85f6-fe876c76437a\") " pod="openshift-image-registry/image-registry-6689c6d46b-vw558" Nov 02 06:16:53.060664 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.060650 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwdcw\" (UniqueName: \"kubernetes.io/projected/a4319db2-eda9-47a0-b742-985c8bdd7ea2-kube-api-access-qwdcw\") pod \"periodic-gathering-rrgb2-xp5b2\" (UID: \"a4319db2-eda9-47a0-b742-985c8bdd7ea2\") " pod="openshift-insights/periodic-gathering-rrgb2-xp5b2" Nov 02 06:16:53.060706 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.060692 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwqbt\" (UniqueName: \"kubernetes.io/projected/cf8e8e65-8525-484b-adfe-f56517ccdc1b-kube-api-access-vwqbt\") pod \"cluster-samples-operator-6ddb6c8b94-d6bc4\" (UID: \"cf8e8e65-8525-484b-adfe-f56517ccdc1b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-d6bc4" Nov 02 06:16:53.060741 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.060726 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f0531b4d-c954-4830-85f6-fe876c76437a-installation-pull-secrets\") pod \"image-registry-6689c6d46b-vw558\" (UID: \"f0531b4d-c954-4830-85f6-fe876c76437a\") " pod="openshift-image-registry/image-registry-6689c6d46b-vw558" Nov 02 06:16:53.061220 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.061202 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vz9mf\" (UniqueName: \"kubernetes.io/projected/3419bd74-22aa-4841-87d2-c4340e01dae9-kube-api-access-vz9mf\") pod \"iptables-alerter-6fnx8\" (UID: \"3419bd74-22aa-4841-87d2-c4340e01dae9\") " pod="openshift-network-operator/iptables-alerter-6fnx8" Nov 02 06:16:53.061331 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.061308 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5q7t\" (UniqueName: \"kubernetes.io/projected/7434256f-45c3-47ff-9111-352114b0f6d8-kube-api-access-l5q7t\") pod \"kas-connection-checker-685895c7c9-kb9wk\" (UID: \"7434256f-45c3-47ff-9111-352114b0f6d8\") " pod="kube-system/kas-connection-checker-685895c7c9-kb9wk" Nov 02 06:16:53.061587 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.061570 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rd8hq\" (UniqueName: \"kubernetes.io/projected/cc33996a-0d4c-49c4-ba67-283bd6a2de14-kube-api-access-rd8hq\") pod \"service-ca-operator-544d749c8c-wcrtl\" (UID: \"cc33996a-0d4c-49c4-ba67-283bd6a2de14\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wcrtl" Nov 02 06:16:53.061649 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.061630 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwh5f\" (UniqueName: \"kubernetes.io/projected/3fcb7621-86c8-448f-b4fa-b5bd76be7356-kube-api-access-mwh5f\") pod \"cluster-monitoring-operator-884576c57-2rs6c\" (UID: \"3fcb7621-86c8-448f-b4fa-b5bd76be7356\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2rs6c" Nov 02 06:16:53.061810 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.061792 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tz6t\" (UniqueName: \"kubernetes.io/projected/17dee72b-2f47-4016-8ffe-3a89934e66ee-kube-api-access-9tz6t\") pod \"console-operator-7cd56b8dbc-csfhg\" (UID: \"17dee72b-2f47-4016-8ffe-3a89934e66ee\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-csfhg" Nov 02 06:16:53.061867 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.061822 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m89g8\" (UniqueName: \"kubernetes.io/projected/f0531b4d-c954-4830-85f6-fe876c76437a-kube-api-access-m89g8\") pod \"image-registry-6689c6d46b-vw558\" (UID: \"f0531b4d-c954-4830-85f6-fe876c76437a\") " pod="openshift-image-registry/image-registry-6689c6d46b-vw558" Nov 02 06:16:53.062868 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.062852 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kcrm\" (UniqueName: \"kubernetes.io/projected/d8d87317-3f41-46d4-a58e-3c7601e0d445-kube-api-access-5kcrm\") pod \"node-resolver-2p95f\" (UID: \"d8d87317-3f41-46d4-a58e-3c7601e0d445\") " pod="openshift-dns/node-resolver-2p95f" Nov 02 06:16:53.102258 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.102225 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/87e797a0-1da6-41ca-af8d-7382afaf3987-host-var-lib-cni-multus\") pod \"multus-rf7qp\" (UID: \"87e797a0-1da6-41ca-af8d-7382afaf3987\") " pod="openshift-multus/multus-rf7qp" Nov 02 06:16:53.102450 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.102267 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/81b393f2-42de-4a99-8114-1445de4d6dc8-lib-modules\") pod \"tuned-6nk8l\" (UID: \"81b393f2-42de-4a99-8114-1445de4d6dc8\") " pod="openshift-cluster-node-tuning-operator/tuned-6nk8l" Nov 02 06:16:53.102450 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.102295 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3500e682-2f15-45a2-92d6-bbc35dae3231-device-dir\") pod \"aws-ebs-csi-driver-node-cbzvd\" (UID: \"3500e682-2f15-45a2-92d6-bbc35dae3231\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cbzvd" Nov 02 06:16:53.102450 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.102316 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/87e797a0-1da6-41ca-af8d-7382afaf3987-system-cni-dir\") pod \"multus-rf7qp\" (UID: \"87e797a0-1da6-41ca-af8d-7382afaf3987\") " pod="openshift-multus/multus-rf7qp" Nov 02 06:16:53.102450 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.102331 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/87e797a0-1da6-41ca-af8d-7382afaf3987-host-var-lib-cni-multus\") pod \"multus-rf7qp\" (UID: \"87e797a0-1da6-41ca-af8d-7382afaf3987\") " pod="openshift-multus/multus-rf7qp" Nov 02 06:16:53.102450 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.102339 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a2ef8269-c442-42a8-8041-6c1b6aaf0746-host-cni-netd\") pod \"ovnkube-node-7wb4x\" (UID: \"a2ef8269-c442-42a8-8041-6c1b6aaf0746\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wb4x" Nov 02 06:16:53.102450 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.102365 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3500e682-2f15-45a2-92d6-bbc35dae3231-device-dir\") pod \"aws-ebs-csi-driver-node-cbzvd\" (UID: \"3500e682-2f15-45a2-92d6-bbc35dae3231\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cbzvd" Nov 02 06:16:53.102450 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.102372 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a2ef8269-c442-42a8-8041-6c1b6aaf0746-host-cni-netd\") pod \"ovnkube-node-7wb4x\" (UID: \"a2ef8269-c442-42a8-8041-6c1b6aaf0746\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wb4x" Nov 02 06:16:53.102450 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.102391 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3500e682-2f15-45a2-92d6-bbc35dae3231-registration-dir\") pod \"aws-ebs-csi-driver-node-cbzvd\" (UID: \"3500e682-2f15-45a2-92d6-bbc35dae3231\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cbzvd" Nov 02 06:16:53.102450 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.102424 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/87e797a0-1da6-41ca-af8d-7382afaf3987-system-cni-dir\") pod \"multus-rf7qp\" (UID: \"87e797a0-1da6-41ca-af8d-7382afaf3987\") " pod="openshift-multus/multus-rf7qp" Nov 02 06:16:53.102450 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.102422 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/81b393f2-42de-4a99-8114-1445de4d6dc8-lib-modules\") pod \"tuned-6nk8l\" (UID: \"81b393f2-42de-4a99-8114-1445de4d6dc8\") " pod="openshift-cluster-node-tuning-operator/tuned-6nk8l" Nov 02 06:16:53.102450 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.102431 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/81b393f2-42de-4a99-8114-1445de4d6dc8-etc-kubernetes\") pod \"tuned-6nk8l\" (UID: \"81b393f2-42de-4a99-8114-1445de4d6dc8\") " pod="openshift-cluster-node-tuning-operator/tuned-6nk8l" Nov 02 06:16:53.102450 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.102457 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/81b393f2-42de-4a99-8114-1445de4d6dc8-etc-sysctl-d\") pod \"tuned-6nk8l\" (UID: \"81b393f2-42de-4a99-8114-1445de4d6dc8\") " pod="openshift-cluster-node-tuning-operator/tuned-6nk8l" Nov 02 06:16:53.102934 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.102469 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3500e682-2f15-45a2-92d6-bbc35dae3231-registration-dir\") pod \"aws-ebs-csi-driver-node-cbzvd\" (UID: \"3500e682-2f15-45a2-92d6-bbc35dae3231\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cbzvd" Nov 02 06:16:53.102934 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.102474 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3500e682-2f15-45a2-92d6-bbc35dae3231-sys-fs\") pod \"aws-ebs-csi-driver-node-cbzvd\" (UID: \"3500e682-2f15-45a2-92d6-bbc35dae3231\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cbzvd" Nov 02 06:16:53.102934 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.102496 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/81b393f2-42de-4a99-8114-1445de4d6dc8-etc-kubernetes\") pod \"tuned-6nk8l\" (UID: \"81b393f2-42de-4a99-8114-1445de4d6dc8\") " pod="openshift-cluster-node-tuning-operator/tuned-6nk8l" Nov 02 06:16:53.102934 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.102510 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3500e682-2f15-45a2-92d6-bbc35dae3231-sys-fs\") pod \"aws-ebs-csi-driver-node-cbzvd\" (UID: \"3500e682-2f15-45a2-92d6-bbc35dae3231\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cbzvd" Nov 02 06:16:53.102934 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.102521 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a2ef8269-c442-42a8-8041-6c1b6aaf0746-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7wb4x\" (UID: \"a2ef8269-c442-42a8-8041-6c1b6aaf0746\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wb4x" Nov 02 06:16:53.102934 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.102544 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a2ef8269-c442-42a8-8041-6c1b6aaf0746-host-slash\") pod \"ovnkube-node-7wb4x\" (UID: \"a2ef8269-c442-42a8-8041-6c1b6aaf0746\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wb4x" Nov 02 06:16:53.102934 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.102573 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/87e797a0-1da6-41ca-af8d-7382afaf3987-hostroot\") pod \"multus-rf7qp\" (UID: \"87e797a0-1da6-41ca-af8d-7382afaf3987\") " pod="openshift-multus/multus-rf7qp" Nov 02 06:16:53.102934 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.102590 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/81b393f2-42de-4a99-8114-1445de4d6dc8-etc-sysctl-d\") pod \"tuned-6nk8l\" (UID: \"81b393f2-42de-4a99-8114-1445de4d6dc8\") " pod="openshift-cluster-node-tuning-operator/tuned-6nk8l" Nov 02 06:16:53.102934 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.102609 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a2ef8269-c442-42a8-8041-6c1b6aaf0746-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7wb4x\" (UID: \"a2ef8269-c442-42a8-8041-6c1b6aaf0746\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wb4x" Nov 02 06:16:53.102934 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.102612 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/87e797a0-1da6-41ca-af8d-7382afaf3987-hostroot\") pod \"multus-rf7qp\" (UID: \"87e797a0-1da6-41ca-af8d-7382afaf3987\") " pod="openshift-multus/multus-rf7qp" Nov 02 06:16:53.102934 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.102636 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/81b393f2-42de-4a99-8114-1445de4d6dc8-etc-sysctl-conf\") pod \"tuned-6nk8l\" (UID: \"81b393f2-42de-4a99-8114-1445de4d6dc8\") " pod="openshift-cluster-node-tuning-operator/tuned-6nk8l" Nov 02 06:16:53.102934 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.102692 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/81b393f2-42de-4a99-8114-1445de4d6dc8-etc-systemd\") pod \"tuned-6nk8l\" (UID: \"81b393f2-42de-4a99-8114-1445de4d6dc8\") " pod="openshift-cluster-node-tuning-operator/tuned-6nk8l" Nov 02 06:16:53.102934 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.102720 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3419bd74-22aa-4841-87d2-c4340e01dae9-host-slash\") pod \"iptables-alerter-6fnx8\" (UID: \"3419bd74-22aa-4841-87d2-c4340e01dae9\") " pod="openshift-network-operator/iptables-alerter-6fnx8" Nov 02 06:16:53.102934 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.102736 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a2ef8269-c442-42a8-8041-6c1b6aaf0746-host-run-ovn-kubernetes\") pod \"ovnkube-node-7wb4x\" (UID: \"a2ef8269-c442-42a8-8041-6c1b6aaf0746\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wb4x" Nov 02 06:16:53.102934 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.102759 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/81b393f2-42de-4a99-8114-1445de4d6dc8-etc-systemd\") pod \"tuned-6nk8l\" (UID: \"81b393f2-42de-4a99-8114-1445de4d6dc8\") " pod="openshift-cluster-node-tuning-operator/tuned-6nk8l" Nov 02 06:16:53.102934 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.102762 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8b491a46-9f53-4a4f-97ed-d8b32e3a03fa-crio-socket\") pod \"insights-runtime-extractor-p98cq\" (UID: \"8b491a46-9f53-4a4f-97ed-d8b32e3a03fa\") " pod="openshift-insights/insights-runtime-extractor-p98cq" Nov 02 06:16:53.102934 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.102763 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a2ef8269-c442-42a8-8041-6c1b6aaf0746-host-slash\") pod \"ovnkube-node-7wb4x\" (UID: \"a2ef8269-c442-42a8-8041-6c1b6aaf0746\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wb4x" Nov 02 06:16:53.102934 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.102784 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/81b393f2-42de-4a99-8114-1445de4d6dc8-etc-sysctl-conf\") pod \"tuned-6nk8l\" (UID: \"81b393f2-42de-4a99-8114-1445de4d6dc8\") " pod="openshift-cluster-node-tuning-operator/tuned-6nk8l" Nov 02 06:16:53.102934 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.102810 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3419bd74-22aa-4841-87d2-c4340e01dae9-host-slash\") pod \"iptables-alerter-6fnx8\" (UID: \"3419bd74-22aa-4841-87d2-c4340e01dae9\") " pod="openshift-network-operator/iptables-alerter-6fnx8" Nov 02 06:16:53.102934 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.102815 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a2ef8269-c442-42a8-8041-6c1b6aaf0746-host-run-ovn-kubernetes\") pod \"ovnkube-node-7wb4x\" (UID: \"a2ef8269-c442-42a8-8041-6c1b6aaf0746\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wb4x" Nov 02 06:16:53.102934 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.102810 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a2ef8269-c442-42a8-8041-6c1b6aaf0746-var-lib-openvswitch\") pod \"ovnkube-node-7wb4x\" (UID: \"a2ef8269-c442-42a8-8041-6c1b6aaf0746\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wb4x" Nov 02 06:16:53.102934 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.102852 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/ac6faf80-42b2-4c7a-8e55-abf6435b3d1e-signing-key\") pod \"service-ca-5fffc97bdf-96jpc\" (UID: \"ac6faf80-42b2-4c7a-8e55-abf6435b3d1e\") " pod="openshift-service-ca/service-ca-5fffc97bdf-96jpc" Nov 02 06:16:53.102934 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.102859 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a2ef8269-c442-42a8-8041-6c1b6aaf0746-var-lib-openvswitch\") pod \"ovnkube-node-7wb4x\" (UID: \"a2ef8269-c442-42a8-8041-6c1b6aaf0746\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wb4x" Nov 02 06:16:53.102934 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.102880 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/87e797a0-1da6-41ca-af8d-7382afaf3987-host-var-lib-kubelet\") pod \"multus-rf7qp\" (UID: \"87e797a0-1da6-41ca-af8d-7382afaf3987\") " pod="openshift-multus/multus-rf7qp" Nov 02 06:16:53.102934 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.102899 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3500e682-2f15-45a2-92d6-bbc35dae3231-socket-dir\") pod \"aws-ebs-csi-driver-node-cbzvd\" (UID: \"3500e682-2f15-45a2-92d6-bbc35dae3231\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cbzvd" Nov 02 06:16:53.102934 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.102940 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d8d87317-3f41-46d4-a58e-3c7601e0d445-hosts-file\") pod \"node-resolver-2p95f\" (UID: \"d8d87317-3f41-46d4-a58e-3c7601e0d445\") " pod="openshift-dns/node-resolver-2p95f" Nov 02 06:16:53.103783 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.102942 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/87e797a0-1da6-41ca-af8d-7382afaf3987-host-var-lib-kubelet\") pod \"multus-rf7qp\" (UID: \"87e797a0-1da6-41ca-af8d-7382afaf3987\") " pod="openshift-multus/multus-rf7qp" Nov 02 06:16:53.103783 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.102962 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a2ef8269-c442-42a8-8041-6c1b6aaf0746-etc-openvswitch\") pod \"ovnkube-node-7wb4x\" (UID: \"a2ef8269-c442-42a8-8041-6c1b6aaf0746\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wb4x" Nov 02 06:16:53.103783 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.102987 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a2ef8269-c442-42a8-8041-6c1b6aaf0746-etc-openvswitch\") pod \"ovnkube-node-7wb4x\" (UID: \"a2ef8269-c442-42a8-8041-6c1b6aaf0746\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wb4x" Nov 02 06:16:53.103783 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.102992 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f14e8781-bf82-4810-9909-c126ccdc1ce1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ckddw\" (UID: \"f14e8781-bf82-4810-9909-c126ccdc1ce1\") " pod="openshift-multus/multus-additional-cni-plugins-ckddw" Nov 02 06:16:53.103783 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.103020 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/87e797a0-1da6-41ca-af8d-7382afaf3987-host-var-lib-cni-bin\") pod \"multus-rf7qp\" (UID: \"87e797a0-1da6-41ca-af8d-7382afaf3987\") " pod="openshift-multus/multus-rf7qp" Nov 02 06:16:53.103783 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.103039 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/87e797a0-1da6-41ca-af8d-7382afaf3987-cnibin\") pod \"multus-rf7qp\" (UID: \"87e797a0-1da6-41ca-af8d-7382afaf3987\") " pod="openshift-multus/multus-rf7qp" Nov 02 06:16:53.103783 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.103042 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3500e682-2f15-45a2-92d6-bbc35dae3231-socket-dir\") pod \"aws-ebs-csi-driver-node-cbzvd\" (UID: \"3500e682-2f15-45a2-92d6-bbc35dae3231\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cbzvd" Nov 02 06:16:53.103783 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.103056 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/87e797a0-1da6-41ca-af8d-7382afaf3987-multus-cni-dir\") pod \"multus-rf7qp\" (UID: \"87e797a0-1da6-41ca-af8d-7382afaf3987\") " pod="openshift-multus/multus-rf7qp" Nov 02 06:16:53.103783 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.103037 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d8d87317-3f41-46d4-a58e-3c7601e0d445-hosts-file\") pod \"node-resolver-2p95f\" (UID: \"d8d87317-3f41-46d4-a58e-3c7601e0d445\") " pod="openshift-dns/node-resolver-2p95f" Nov 02 06:16:53.103783 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.103077 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f14e8781-bf82-4810-9909-c126ccdc1ce1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ckddw\" (UID: \"f14e8781-bf82-4810-9909-c126ccdc1ce1\") " pod="openshift-multus/multus-additional-cni-plugins-ckddw" Nov 02 06:16:53.103783 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.103075 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/87e797a0-1da6-41ca-af8d-7382afaf3987-host-run-netns\") pod \"multus-rf7qp\" (UID: \"87e797a0-1da6-41ca-af8d-7382afaf3987\") " pod="openshift-multus/multus-rf7qp" Nov 02 06:16:53.103783 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.103094 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/87e797a0-1da6-41ca-af8d-7382afaf3987-cnibin\") pod \"multus-rf7qp\" (UID: \"87e797a0-1da6-41ca-af8d-7382afaf3987\") " pod="openshift-multus/multus-rf7qp" Nov 02 06:16:53.103783 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.103106 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/87e797a0-1da6-41ca-af8d-7382afaf3987-host-var-lib-cni-bin\") pod \"multus-rf7qp\" (UID: \"87e797a0-1da6-41ca-af8d-7382afaf3987\") " pod="openshift-multus/multus-rf7qp" Nov 02 06:16:53.103783 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.103122 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/ac6faf80-42b2-4c7a-8e55-abf6435b3d1e-signing-key\") pod \"service-ca-5fffc97bdf-96jpc\" (UID: \"ac6faf80-42b2-4c7a-8e55-abf6435b3d1e\") " pod="openshift-service-ca/service-ca-5fffc97bdf-96jpc" Nov 02 06:16:53.103783 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.103121 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a2ef8269-c442-42a8-8041-6c1b6aaf0746-log-socket\") pod \"ovnkube-node-7wb4x\" (UID: \"a2ef8269-c442-42a8-8041-6c1b6aaf0746\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wb4x" Nov 02 06:16:53.103783 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.103134 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/87e797a0-1da6-41ca-af8d-7382afaf3987-host-run-netns\") pod \"multus-rf7qp\" (UID: \"87e797a0-1da6-41ca-af8d-7382afaf3987\") " pod="openshift-multus/multus-rf7qp" Nov 02 06:16:53.103783 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.103149 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a2ef8269-c442-42a8-8041-6c1b6aaf0746-log-socket\") pod \"ovnkube-node-7wb4x\" (UID: \"a2ef8269-c442-42a8-8041-6c1b6aaf0746\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wb4x" Nov 02 06:16:53.103783 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.103153 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/81b393f2-42de-4a99-8114-1445de4d6dc8-etc-sysconfig\") pod \"tuned-6nk8l\" (UID: \"81b393f2-42de-4a99-8114-1445de4d6dc8\") " pod="openshift-cluster-node-tuning-operator/tuned-6nk8l" Nov 02 06:16:53.103783 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.103152 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/87e797a0-1da6-41ca-af8d-7382afaf3987-multus-cni-dir\") pod \"multus-rf7qp\" (UID: \"87e797a0-1da6-41ca-af8d-7382afaf3987\") " pod="openshift-multus/multus-rf7qp" Nov 02 06:16:53.103783 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.103172 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a2ef8269-c442-42a8-8041-6c1b6aaf0746-run-systemd\") pod \"ovnkube-node-7wb4x\" (UID: \"a2ef8269-c442-42a8-8041-6c1b6aaf0746\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wb4x" Nov 02 06:16:53.103783 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.103193 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3500e682-2f15-45a2-92d6-bbc35dae3231-kubelet-dir\") pod \"aws-ebs-csi-driver-node-cbzvd\" (UID: \"3500e682-2f15-45a2-92d6-bbc35dae3231\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cbzvd" Nov 02 06:16:53.103783 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.103188 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/81b393f2-42de-4a99-8114-1445de4d6dc8-etc-sysconfig\") pod \"tuned-6nk8l\" (UID: \"81b393f2-42de-4a99-8114-1445de4d6dc8\") " pod="openshift-cluster-node-tuning-operator/tuned-6nk8l" Nov 02 06:16:53.103783 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.103222 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b65bb247-83f7-4749-9aa1-9b2677dd1b3d-dbus\") pod \"global-pull-secret-syncer-ggpsr\" (UID: \"b65bb247-83f7-4749-9aa1-9b2677dd1b3d\") " pod="kube-system/global-pull-secret-syncer-ggpsr" Nov 02 06:16:53.103783 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.103218 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a2ef8269-c442-42a8-8041-6c1b6aaf0746-run-systemd\") pod \"ovnkube-node-7wb4x\" (UID: \"a2ef8269-c442-42a8-8041-6c1b6aaf0746\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wb4x" Nov 02 06:16:53.103783 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.103249 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpt4c\" (UniqueName: \"kubernetes.io/projected/8b491a46-9f53-4a4f-97ed-d8b32e3a03fa-kube-api-access-dpt4c\") pod \"insights-runtime-extractor-p98cq\" (UID: \"8b491a46-9f53-4a4f-97ed-d8b32e3a03fa\") " pod="openshift-insights/insights-runtime-extractor-p98cq" Nov 02 06:16:53.103783 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.103264 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3500e682-2f15-45a2-92d6-bbc35dae3231-kubelet-dir\") pod \"aws-ebs-csi-driver-node-cbzvd\" (UID: \"3500e682-2f15-45a2-92d6-bbc35dae3231\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cbzvd" Nov 02 06:16:53.103783 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.103279 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/87e797a0-1da6-41ca-af8d-7382afaf3987-host-run-k8s-cni-cncf-io\") pod \"multus-rf7qp\" (UID: \"87e797a0-1da6-41ca-af8d-7382afaf3987\") " pod="openshift-multus/multus-rf7qp" Nov 02 06:16:53.103783 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.103284 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b65bb247-83f7-4749-9aa1-9b2677dd1b3d-dbus\") pod \"global-pull-secret-syncer-ggpsr\" (UID: \"b65bb247-83f7-4749-9aa1-9b2677dd1b3d\") " pod="kube-system/global-pull-secret-syncer-ggpsr" Nov 02 06:16:53.103783 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.103315 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/87e797a0-1da6-41ca-af8d-7382afaf3987-host-run-k8s-cni-cncf-io\") pod \"multus-rf7qp\" (UID: \"87e797a0-1da6-41ca-af8d-7382afaf3987\") " pod="openshift-multus/multus-rf7qp" Nov 02 06:16:53.103783 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.103374 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/81b393f2-42de-4a99-8114-1445de4d6dc8-var-lib-kubelet\") pod \"tuned-6nk8l\" (UID: \"81b393f2-42de-4a99-8114-1445de4d6dc8\") " pod="openshift-cluster-node-tuning-operator/tuned-6nk8l" Nov 02 06:16:53.103783 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.103399 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/81b393f2-42de-4a99-8114-1445de4d6dc8-host\") pod \"tuned-6nk8l\" (UID: \"81b393f2-42de-4a99-8114-1445de4d6dc8\") " pod="openshift-cluster-node-tuning-operator/tuned-6nk8l" Nov 02 06:16:53.103783 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.103414 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a2ef8269-c442-42a8-8041-6c1b6aaf0746-systemd-units\") pod \"ovnkube-node-7wb4x\" (UID: \"a2ef8269-c442-42a8-8041-6c1b6aaf0746\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wb4x" Nov 02 06:16:53.103783 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.103429 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/87e797a0-1da6-41ca-af8d-7382afaf3987-multus-socket-dir-parent\") pod \"multus-rf7qp\" (UID: \"87e797a0-1da6-41ca-af8d-7382afaf3987\") " pod="openshift-multus/multus-rf7qp" Nov 02 06:16:53.103783 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.103446 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8b491a46-9f53-4a4f-97ed-d8b32e3a03fa-data-volume\") pod \"insights-runtime-extractor-p98cq\" (UID: \"8b491a46-9f53-4a4f-97ed-d8b32e3a03fa\") " pod="openshift-insights/insights-runtime-extractor-p98cq" Nov 02 06:16:53.103783 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.103480 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/81b393f2-42de-4a99-8114-1445de4d6dc8-host\") pod \"tuned-6nk8l\" (UID: \"81b393f2-42de-4a99-8114-1445de4d6dc8\") " pod="openshift-cluster-node-tuning-operator/tuned-6nk8l" Nov 02 06:16:53.103783 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.103486 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a2ef8269-c442-42a8-8041-6c1b6aaf0746-systemd-units\") pod \"ovnkube-node-7wb4x\" (UID: \"a2ef8269-c442-42a8-8041-6c1b6aaf0746\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wb4x" Nov 02 06:16:53.103783 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.103490 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/81b393f2-42de-4a99-8114-1445de4d6dc8-run\") pod \"tuned-6nk8l\" (UID: \"81b393f2-42de-4a99-8114-1445de4d6dc8\") " pod="openshift-cluster-node-tuning-operator/tuned-6nk8l" Nov 02 06:16:53.103783 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.103499 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/81b393f2-42de-4a99-8114-1445de4d6dc8-var-lib-kubelet\") pod \"tuned-6nk8l\" (UID: \"81b393f2-42de-4a99-8114-1445de4d6dc8\") " pod="openshift-cluster-node-tuning-operator/tuned-6nk8l" Nov 02 06:16:53.103783 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.103514 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/87e797a0-1da6-41ca-af8d-7382afaf3987-multus-socket-dir-parent\") pod \"multus-rf7qp\" (UID: \"87e797a0-1da6-41ca-af8d-7382afaf3987\") " pod="openshift-multus/multus-rf7qp" Nov 02 06:16:53.103783 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.103519 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/81b393f2-42de-4a99-8114-1445de4d6dc8-sys\") pod \"tuned-6nk8l\" (UID: \"81b393f2-42de-4a99-8114-1445de4d6dc8\") " pod="openshift-cluster-node-tuning-operator/tuned-6nk8l" Nov 02 06:16:53.103783 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.103525 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/81b393f2-42de-4a99-8114-1445de4d6dc8-run\") pod \"tuned-6nk8l\" (UID: \"81b393f2-42de-4a99-8114-1445de4d6dc8\") " pod="openshift-cluster-node-tuning-operator/tuned-6nk8l" Nov 02 06:16:53.103783 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.103550 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/81b393f2-42de-4a99-8114-1445de4d6dc8-sys\") pod \"tuned-6nk8l\" (UID: \"81b393f2-42de-4a99-8114-1445de4d6dc8\") " pod="openshift-cluster-node-tuning-operator/tuned-6nk8l" Nov 02 06:16:53.103783 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.103548 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8b491a46-9f53-4a4f-97ed-d8b32e3a03fa-data-volume\") pod \"insights-runtime-extractor-p98cq\" (UID: \"8b491a46-9f53-4a4f-97ed-d8b32e3a03fa\") " pod="openshift-insights/insights-runtime-extractor-p98cq" Nov 02 06:16:53.103783 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.103561 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a2ef8269-c442-42a8-8041-6c1b6aaf0746-run-ovn\") pod \"ovnkube-node-7wb4x\" (UID: \"a2ef8269-c442-42a8-8041-6c1b6aaf0746\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wb4x" Nov 02 06:16:53.103783 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.103586 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a2ef8269-c442-42a8-8041-6c1b6aaf0746-run-ovn\") pod \"ovnkube-node-7wb4x\" (UID: \"a2ef8269-c442-42a8-8041-6c1b6aaf0746\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wb4x" Nov 02 06:16:53.103783 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.103598 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7jwb\" (UniqueName: \"kubernetes.io/projected/49f1bd67-26ad-4fc9-b3d5-7bf7f6404aaa-kube-api-access-v7jwb\") pod \"migrator-66df5d496b-mddfd\" (UID: \"49f1bd67-26ad-4fc9-b3d5-7bf7f6404aaa\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-mddfd" Nov 02 06:16:53.103783 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.103617 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f14e8781-bf82-4810-9909-c126ccdc1ce1-system-cni-dir\") pod \"multus-additional-cni-plugins-ckddw\" (UID: \"f14e8781-bf82-4810-9909-c126ccdc1ce1\") " pod="openshift-multus/multus-additional-cni-plugins-ckddw" Nov 02 06:16:53.103783 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.103634 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f14e8781-bf82-4810-9909-c126ccdc1ce1-cnibin\") pod \"multus-additional-cni-plugins-ckddw\" (UID: \"f14e8781-bf82-4810-9909-c126ccdc1ce1\") " pod="openshift-multus/multus-additional-cni-plugins-ckddw" Nov 02 06:16:53.103783 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.103652 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8b491a46-9f53-4a4f-97ed-d8b32e3a03fa-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p98cq\" (UID: \"8b491a46-9f53-4a4f-97ed-d8b32e3a03fa\") " pod="openshift-insights/insights-runtime-extractor-p98cq" Nov 02 06:16:53.103783 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.103670 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/87e797a0-1da6-41ca-af8d-7382afaf3987-host-run-multus-certs\") pod \"multus-rf7qp\" (UID: \"87e797a0-1da6-41ca-af8d-7382afaf3987\") " pod="openshift-multus/multus-rf7qp" Nov 02 06:16:53.103783 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.103686 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/87e797a0-1da6-41ca-af8d-7382afaf3987-multus-conf-dir\") pod \"multus-rf7qp\" (UID: \"87e797a0-1da6-41ca-af8d-7382afaf3987\") " pod="openshift-multus/multus-rf7qp" Nov 02 06:16:53.103783 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.103706 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgkrw\" (UniqueName: \"kubernetes.io/projected/ac6faf80-42b2-4c7a-8e55-abf6435b3d1e-kube-api-access-pgkrw\") pod \"service-ca-5fffc97bdf-96jpc\" (UID: \"ac6faf80-42b2-4c7a-8e55-abf6435b3d1e\") " pod="openshift-service-ca/service-ca-5fffc97bdf-96jpc" Nov 02 06:16:53.103783 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.103720 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f14e8781-bf82-4810-9909-c126ccdc1ce1-system-cni-dir\") pod \"multus-additional-cni-plugins-ckddw\" (UID: \"f14e8781-bf82-4810-9909-c126ccdc1ce1\") " pod="openshift-multus/multus-additional-cni-plugins-ckddw" Nov 02 06:16:53.103783 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.103725 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f14e8781-bf82-4810-9909-c126ccdc1ce1-cnibin\") pod \"multus-additional-cni-plugins-ckddw\" (UID: \"f14e8781-bf82-4810-9909-c126ccdc1ce1\") " pod="openshift-multus/multus-additional-cni-plugins-ckddw" Nov 02 06:16:53.103783 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.103737 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/87e797a0-1da6-41ca-af8d-7382afaf3987-host-run-multus-certs\") pod \"multus-rf7qp\" (UID: \"87e797a0-1da6-41ca-af8d-7382afaf3987\") " pod="openshift-multus/multus-rf7qp" Nov 02 06:16:53.103783 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.103731 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a2ef8269-c442-42a8-8041-6c1b6aaf0746-node-log\") pod \"ovnkube-node-7wb4x\" (UID: \"a2ef8269-c442-42a8-8041-6c1b6aaf0746\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wb4x" Nov 02 06:16:53.103783 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.103750 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/87e797a0-1da6-41ca-af8d-7382afaf3987-multus-conf-dir\") pod \"multus-rf7qp\" (UID: \"87e797a0-1da6-41ca-af8d-7382afaf3987\") " pod="openshift-multus/multus-rf7qp" Nov 02 06:16:53.103783 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.103774 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a2ef8269-c442-42a8-8041-6c1b6aaf0746-node-log\") pod \"ovnkube-node-7wb4x\" (UID: \"a2ef8269-c442-42a8-8041-6c1b6aaf0746\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wb4x" Nov 02 06:16:53.105498 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.103841 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3eec1082-df0b-420b-b4ff-5a1c0fab1d1d-host\") pod \"node-ca-qm6wn\" (UID: \"3eec1082-df0b-420b-b4ff-5a1c0fab1d1d\") " pod="openshift-image-registry/node-ca-qm6wn" Nov 02 06:16:53.105498 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.103948 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a2ef8269-c442-42a8-8041-6c1b6aaf0746-run-openvswitch\") pod \"ovnkube-node-7wb4x\" (UID: \"a2ef8269-c442-42a8-8041-6c1b6aaf0746\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wb4x" Nov 02 06:16:53.105498 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.103979 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a2ef8269-c442-42a8-8041-6c1b6aaf0746-run-openvswitch\") pod \"ovnkube-node-7wb4x\" (UID: \"a2ef8269-c442-42a8-8041-6c1b6aaf0746\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wb4x" Nov 02 06:16:53.105498 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.104018 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8b491a46-9f53-4a4f-97ed-d8b32e3a03fa-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p98cq\" (UID: \"8b491a46-9f53-4a4f-97ed-d8b32e3a03fa\") " pod="openshift-insights/insights-runtime-extractor-p98cq" Nov 02 06:16:53.105498 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.104029 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3500e682-2f15-45a2-92d6-bbc35dae3231-etc-selinux\") pod \"aws-ebs-csi-driver-node-cbzvd\" (UID: \"3500e682-2f15-45a2-92d6-bbc35dae3231\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cbzvd" Nov 02 06:16:53.105498 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.104018 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3eec1082-df0b-420b-b4ff-5a1c0fab1d1d-host\") pod \"node-ca-qm6wn\" (UID: \"3eec1082-df0b-420b-b4ff-5a1c0fab1d1d\") " pod="openshift-image-registry/node-ca-qm6wn" Nov 02 06:16:53.105498 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.104079 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/87e797a0-1da6-41ca-af8d-7382afaf3987-etc-kubernetes\") pod \"multus-rf7qp\" (UID: \"87e797a0-1da6-41ca-af8d-7382afaf3987\") " pod="openshift-multus/multus-rf7qp" Nov 02 06:16:53.105498 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.104096 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/ac6faf80-42b2-4c7a-8e55-abf6435b3d1e-signing-cabundle\") pod \"service-ca-5fffc97bdf-96jpc\" (UID: \"ac6faf80-42b2-4c7a-8e55-abf6435b3d1e\") " pod="openshift-service-ca/service-ca-5fffc97bdf-96jpc" Nov 02 06:16:53.105498 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.104110 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a2ef8269-c442-42a8-8041-6c1b6aaf0746-host-run-netns\") pod \"ovnkube-node-7wb4x\" (UID: \"a2ef8269-c442-42a8-8041-6c1b6aaf0746\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wb4x" Nov 02 06:16:53.105498 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.104118 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3500e682-2f15-45a2-92d6-bbc35dae3231-etc-selinux\") pod \"aws-ebs-csi-driver-node-cbzvd\" (UID: \"3500e682-2f15-45a2-92d6-bbc35dae3231\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cbzvd" Nov 02 06:16:53.105498 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.104132 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a2ef8269-c442-42a8-8041-6c1b6aaf0746-host-cni-bin\") pod \"ovnkube-node-7wb4x\" (UID: \"a2ef8269-c442-42a8-8041-6c1b6aaf0746\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wb4x" Nov 02 06:16:53.105498 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.104147 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/87e797a0-1da6-41ca-af8d-7382afaf3987-etc-kubernetes\") pod \"multus-rf7qp\" (UID: \"87e797a0-1da6-41ca-af8d-7382afaf3987\") " pod="openshift-multus/multus-rf7qp" Nov 02 06:16:53.105498 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.104156 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a2ef8269-c442-42a8-8041-6c1b6aaf0746-host-run-netns\") pod \"ovnkube-node-7wb4x\" (UID: \"a2ef8269-c442-42a8-8041-6c1b6aaf0746\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wb4x" Nov 02 06:16:53.105498 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.104163 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b65bb247-83f7-4749-9aa1-9b2677dd1b3d-kubelet-config\") pod \"global-pull-secret-syncer-ggpsr\" (UID: \"b65bb247-83f7-4749-9aa1-9b2677dd1b3d\") " pod="kube-system/global-pull-secret-syncer-ggpsr" Nov 02 06:16:53.105498 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.104182 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a2ef8269-c442-42a8-8041-6c1b6aaf0746-host-cni-bin\") pod \"ovnkube-node-7wb4x\" (UID: \"a2ef8269-c442-42a8-8041-6c1b6aaf0746\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wb4x" Nov 02 06:16:53.105498 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.104193 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8b491a46-9f53-4a4f-97ed-d8b32e3a03fa-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p98cq\" (UID: \"8b491a46-9f53-4a4f-97ed-d8b32e3a03fa\") " pod="openshift-insights/insights-runtime-extractor-p98cq" Nov 02 06:16:53.105498 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.104224 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b65bb247-83f7-4749-9aa1-9b2677dd1b3d-kubelet-config\") pod \"global-pull-secret-syncer-ggpsr\" (UID: \"b65bb247-83f7-4749-9aa1-9b2677dd1b3d\") " pod="kube-system/global-pull-secret-syncer-ggpsr" Nov 02 06:16:53.105498 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.104218 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/81b393f2-42de-4a99-8114-1445de4d6dc8-etc-modprobe-d\") pod \"tuned-6nk8l\" (UID: \"81b393f2-42de-4a99-8114-1445de4d6dc8\") " pod="openshift-cluster-node-tuning-operator/tuned-6nk8l" Nov 02 06:16:53.105498 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.104301 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a2ef8269-c442-42a8-8041-6c1b6aaf0746-host-kubelet\") pod \"ovnkube-node-7wb4x\" (UID: \"a2ef8269-c442-42a8-8041-6c1b6aaf0746\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wb4x" Nov 02 06:16:53.105498 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.104307 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/ac6faf80-42b2-4c7a-8e55-abf6435b3d1e-signing-cabundle\") pod \"service-ca-5fffc97bdf-96jpc\" (UID: \"ac6faf80-42b2-4c7a-8e55-abf6435b3d1e\") " pod="openshift-service-ca/service-ca-5fffc97bdf-96jpc" Nov 02 06:16:53.105498 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.104332 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/81b393f2-42de-4a99-8114-1445de4d6dc8-etc-modprobe-d\") pod \"tuned-6nk8l\" (UID: \"81b393f2-42de-4a99-8114-1445de4d6dc8\") " pod="openshift-cluster-node-tuning-operator/tuned-6nk8l" Nov 02 06:16:53.105498 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.104351 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a2ef8269-c442-42a8-8041-6c1b6aaf0746-host-kubelet\") pod \"ovnkube-node-7wb4x\" (UID: \"a2ef8269-c442-42a8-8041-6c1b6aaf0746\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wb4x" Nov 02 06:16:53.105498 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.104358 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8b491a46-9f53-4a4f-97ed-d8b32e3a03fa-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p98cq\" (UID: \"8b491a46-9f53-4a4f-97ed-d8b32e3a03fa\") " pod="openshift-insights/insights-runtime-extractor-p98cq" Nov 02 06:16:53.154629 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.154584 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7jwb\" (UniqueName: \"kubernetes.io/projected/49f1bd67-26ad-4fc9-b3d5-7bf7f6404aaa-kube-api-access-v7jwb\") pod \"migrator-66df5d496b-mddfd\" (UID: \"49f1bd67-26ad-4fc9-b3d5-7bf7f6404aaa\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-mddfd" Nov 02 06:16:53.154800 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.154780 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpt4c\" (UniqueName: \"kubernetes.io/projected/8b491a46-9f53-4a4f-97ed-d8b32e3a03fa-kube-api-access-dpt4c\") pod \"insights-runtime-extractor-p98cq\" (UID: \"8b491a46-9f53-4a4f-97ed-d8b32e3a03fa\") " pod="openshift-insights/insights-runtime-extractor-p98cq" Nov 02 06:16:53.154960 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.154894 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgkrw\" (UniqueName: \"kubernetes.io/projected/ac6faf80-42b2-4c7a-8e55-abf6435b3d1e-kube-api-access-pgkrw\") pod \"service-ca-5fffc97bdf-96jpc\" (UID: \"ac6faf80-42b2-4c7a-8e55-abf6435b3d1e\") " pod="openshift-service-ca/service-ca-5fffc97bdf-96jpc" Nov 02 06:16:53.205451 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.205420 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8b491a46-9f53-4a4f-97ed-d8b32e3a03fa-crio-socket\") pod \"insights-runtime-extractor-p98cq\" (UID: \"8b491a46-9f53-4a4f-97ed-d8b32e3a03fa\") " pod="openshift-insights/insights-runtime-extractor-p98cq" Nov 02 06:16:53.205619 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.205530 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8b491a46-9f53-4a4f-97ed-d8b32e3a03fa-crio-socket\") pod \"insights-runtime-extractor-p98cq\" (UID: \"8b491a46-9f53-4a4f-97ed-d8b32e3a03fa\") " pod="openshift-insights/insights-runtime-extractor-p98cq" Nov 02 06:16:53.650734 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.650706 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7wb4x" Nov 02 06:16:53.920765 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.920686 5916 generic.go:356] "Generic (PLEG): container finished" podID="37982aa4-bdbd-40e8-a2dd-5094209f3c96" containerID="7fb19f26c0ab0cab47b874832c74cf70aafb4ff88d00575d84440952233bd3e2" exitCode=0 Nov 02 06:16:53.921172 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.920755 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-c8dcq" event={"ID":"37982aa4-bdbd-40e8-a2dd-5094209f3c96","Type":"ContainerDied","Data":"7fb19f26c0ab0cab47b874832c74cf70aafb4ff88d00575d84440952233bd3e2"} Nov 02 06:16:53.921721 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:53.921700 5916 scope.go:122] "RemoveContainer" containerID="7fb19f26c0ab0cab47b874832c74cf70aafb4ff88d00575d84440952233bd3e2" Nov 02 06:16:54.926228 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:54.926182 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-c8dcq" event={"ID":"37982aa4-bdbd-40e8-a2dd-5094209f3c96","Type":"ContainerStarted","Data":"21588e7b8a618c11bde1be600bd520a8bab2970765ed9c894a039f5424402b78"} Nov 02 06:16:55.931191 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:55.931154 5916 generic.go:356] "Generic (PLEG): container finished" podID="cc33996a-0d4c-49c4-ba67-283bd6a2de14" containerID="0fcc6142d3a9a4c78a2751ebba8f71bcd77e10e57db430c4f043e35836afcda2" exitCode=0 Nov 02 06:16:55.931582 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:55.931224 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wcrtl" event={"ID":"cc33996a-0d4c-49c4-ba67-283bd6a2de14","Type":"ContainerDied","Data":"0fcc6142d3a9a4c78a2751ebba8f71bcd77e10e57db430c4f043e35836afcda2"} Nov 02 06:16:55.931582 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:55.931525 5916 scope.go:122] "RemoveContainer" containerID="0fcc6142d3a9a4c78a2751ebba8f71bcd77e10e57db430c4f043e35836afcda2" Nov 02 06:16:55.996640 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:55.996617 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-gf726" Nov 02 06:16:56.043592 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:56.043569 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-746899d75-dtjhs" Nov 02 06:16:56.585843 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:56.585808 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-csfhg" Nov 02 06:16:56.589535 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:56.589514 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-csfhg" Nov 02 06:16:56.936261 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:56.936223 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wcrtl" event={"ID":"cc33996a-0d4c-49c4-ba67-283bd6a2de14","Type":"ContainerStarted","Data":"f80aeab852490fcaf7ba83536f50e4c915e209d4f99e0c6f9e617dd02b7fee63"} Nov 02 06:16:59.517707 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:16:59.517674 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-5f5c58667c-29wnm" Nov 02 06:17:01.971820 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:17:01.971733 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7wb4x" Nov 02 06:17:01.972289 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:17:01.972002 5916 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 06:17:02.542091 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:17:02.542051 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-86wbt"] Nov 02 06:17:02.550553 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:17:02.550516 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-86wbt" Nov 02 06:17:02.566961 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:17:02.566933 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7wb4x" Nov 02 06:17:02.578525 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:17:02.578494 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/297f9abf-ffb4-493a-8850-fc65648543a8-root\") pod \"node-exporter-86wbt\" (UID: \"297f9abf-ffb4-493a-8850-fc65648543a8\") " pod="openshift-monitoring/node-exporter-86wbt" Nov 02 06:17:02.578672 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:17:02.578546 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/297f9abf-ffb4-493a-8850-fc65648543a8-node-exporter-tls\") pod \"node-exporter-86wbt\" (UID: \"297f9abf-ffb4-493a-8850-fc65648543a8\") " pod="openshift-monitoring/node-exporter-86wbt" Nov 02 06:17:02.578672 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:17:02.578645 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/297f9abf-ffb4-493a-8850-fc65648543a8-node-exporter-textfile\") pod \"node-exporter-86wbt\" (UID: \"297f9abf-ffb4-493a-8850-fc65648543a8\") " pod="openshift-monitoring/node-exporter-86wbt" Nov 02 06:17:02.578745 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:17:02.578697 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frbg8\" (UniqueName: \"kubernetes.io/projected/297f9abf-ffb4-493a-8850-fc65648543a8-kube-api-access-frbg8\") pod \"node-exporter-86wbt\" (UID: \"297f9abf-ffb4-493a-8850-fc65648543a8\") " pod="openshift-monitoring/node-exporter-86wbt" Nov 02 06:17:02.578745 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:17:02.578740 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/297f9abf-ffb4-493a-8850-fc65648543a8-metrics-client-ca\") pod \"node-exporter-86wbt\" (UID: \"297f9abf-ffb4-493a-8850-fc65648543a8\") " pod="openshift-monitoring/node-exporter-86wbt" Nov 02 06:17:02.578813 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:17:02.578786 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/297f9abf-ffb4-493a-8850-fc65648543a8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-86wbt\" (UID: \"297f9abf-ffb4-493a-8850-fc65648543a8\") " pod="openshift-monitoring/node-exporter-86wbt" Nov 02 06:17:02.578846 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:17:02.578840 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/297f9abf-ffb4-493a-8850-fc65648543a8-sys\") pod \"node-exporter-86wbt\" (UID: \"297f9abf-ffb4-493a-8850-fc65648543a8\") " pod="openshift-monitoring/node-exporter-86wbt" Nov 02 06:17:02.578879 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:17:02.578859 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/297f9abf-ffb4-493a-8850-fc65648543a8-node-exporter-wtmp\") pod \"node-exporter-86wbt\" (UID: \"297f9abf-ffb4-493a-8850-fc65648543a8\") " pod="openshift-monitoring/node-exporter-86wbt" Nov 02 06:17:02.578930 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:17:02.578884 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/297f9abf-ffb4-493a-8850-fc65648543a8-node-exporter-accelerators-collector-config\") pod \"node-exporter-86wbt\" (UID: \"297f9abf-ffb4-493a-8850-fc65648543a8\") " pod="openshift-monitoring/node-exporter-86wbt" Nov 02 06:17:02.649223 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:17:02.649188 5916 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 02 06:17:02.649223 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:17:02.649214 5916 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 02 06:17:02.680107 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:17:02.680075 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/297f9abf-ffb4-493a-8850-fc65648543a8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-86wbt\" (UID: \"297f9abf-ffb4-493a-8850-fc65648543a8\") " pod="openshift-monitoring/node-exporter-86wbt" Nov 02 06:17:02.680266 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:17:02.680119 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/297f9abf-ffb4-493a-8850-fc65648543a8-sys\") pod \"node-exporter-86wbt\" (UID: \"297f9abf-ffb4-493a-8850-fc65648543a8\") " pod="openshift-monitoring/node-exporter-86wbt" Nov 02 06:17:02.680266 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:17:02.680140 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/297f9abf-ffb4-493a-8850-fc65648543a8-node-exporter-wtmp\") pod \"node-exporter-86wbt\" (UID: \"297f9abf-ffb4-493a-8850-fc65648543a8\") " pod="openshift-monitoring/node-exporter-86wbt" Nov 02 06:17:02.680266 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:17:02.680194 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/297f9abf-ffb4-493a-8850-fc65648543a8-node-exporter-accelerators-collector-config\") pod \"node-exporter-86wbt\" (UID: \"297f9abf-ffb4-493a-8850-fc65648543a8\") " pod="openshift-monitoring/node-exporter-86wbt" Nov 02 06:17:02.680266 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:17:02.680236 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/297f9abf-ffb4-493a-8850-fc65648543a8-root\") pod \"node-exporter-86wbt\" (UID: \"297f9abf-ffb4-493a-8850-fc65648543a8\") " pod="openshift-monitoring/node-exporter-86wbt" Nov 02 06:17:02.680423 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:17:02.680266 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/297f9abf-ffb4-493a-8850-fc65648543a8-node-exporter-tls\") pod \"node-exporter-86wbt\" (UID: \"297f9abf-ffb4-493a-8850-fc65648543a8\") " pod="openshift-monitoring/node-exporter-86wbt" Nov 02 06:17:02.680423 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:17:02.680275 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/297f9abf-ffb4-493a-8850-fc65648543a8-node-exporter-wtmp\") pod \"node-exporter-86wbt\" (UID: \"297f9abf-ffb4-493a-8850-fc65648543a8\") " pod="openshift-monitoring/node-exporter-86wbt" Nov 02 06:17:02.680423 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:17:02.680279 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/297f9abf-ffb4-493a-8850-fc65648543a8-sys\") pod \"node-exporter-86wbt\" (UID: \"297f9abf-ffb4-493a-8850-fc65648543a8\") " pod="openshift-monitoring/node-exporter-86wbt" Nov 02 06:17:02.680423 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:17:02.680325 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/297f9abf-ffb4-493a-8850-fc65648543a8-root\") pod \"node-exporter-86wbt\" (UID: \"297f9abf-ffb4-493a-8850-fc65648543a8\") " pod="openshift-monitoring/node-exporter-86wbt" Nov 02 06:17:02.680423 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:17:02.680339 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/297f9abf-ffb4-493a-8850-fc65648543a8-node-exporter-textfile\") pod \"node-exporter-86wbt\" (UID: \"297f9abf-ffb4-493a-8850-fc65648543a8\") " pod="openshift-monitoring/node-exporter-86wbt" Nov 02 06:17:02.680423 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:17:02.680364 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frbg8\" (UniqueName: \"kubernetes.io/projected/297f9abf-ffb4-493a-8850-fc65648543a8-kube-api-access-frbg8\") pod \"node-exporter-86wbt\" (UID: \"297f9abf-ffb4-493a-8850-fc65648543a8\") " pod="openshift-monitoring/node-exporter-86wbt" Nov 02 06:17:02.680423 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:17:02.680404 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/297f9abf-ffb4-493a-8850-fc65648543a8-metrics-client-ca\") pod \"node-exporter-86wbt\" (UID: \"297f9abf-ffb4-493a-8850-fc65648543a8\") " pod="openshift-monitoring/node-exporter-86wbt" Nov 02 06:17:02.680734 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:17:02.680674 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/297f9abf-ffb4-493a-8850-fc65648543a8-node-exporter-textfile\") pod \"node-exporter-86wbt\" (UID: \"297f9abf-ffb4-493a-8850-fc65648543a8\") " pod="openshift-monitoring/node-exporter-86wbt" Nov 02 06:17:02.680874 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:17:02.680854 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/297f9abf-ffb4-493a-8850-fc65648543a8-node-exporter-accelerators-collector-config\") pod \"node-exporter-86wbt\" (UID: \"297f9abf-ffb4-493a-8850-fc65648543a8\") " pod="openshift-monitoring/node-exporter-86wbt" Nov 02 06:17:02.680929 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:17:02.680882 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/297f9abf-ffb4-493a-8850-fc65648543a8-metrics-client-ca\") pod \"node-exporter-86wbt\" (UID: \"297f9abf-ffb4-493a-8850-fc65648543a8\") " pod="openshift-monitoring/node-exporter-86wbt" Nov 02 06:17:02.689459 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:17:02.689436 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-ft7gs" Nov 02 06:17:02.695309 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:17:02.695262 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 02 06:17:02.695604 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:17:02.695575 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 02 06:17:02.695732 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:17:02.695710 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-xbcc6\"" Nov 02 06:17:02.701294 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:17:02.701270 5916 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 02 06:17:02.703418 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:17:02.703398 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/297f9abf-ffb4-493a-8850-fc65648543a8-node-exporter-tls\") pod \"node-exporter-86wbt\" (UID: \"297f9abf-ffb4-493a-8850-fc65648543a8\") " pod="openshift-monitoring/node-exporter-86wbt" Nov 02 06:17:02.703528 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:17:02.703510 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/297f9abf-ffb4-493a-8850-fc65648543a8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-86wbt\" (UID: \"297f9abf-ffb4-493a-8850-fc65648543a8\") " pod="openshift-monitoring/node-exporter-86wbt" Nov 02 06:17:02.737800 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:17:02.737777 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frbg8\" (UniqueName: \"kubernetes.io/projected/297f9abf-ffb4-493a-8850-fc65648543a8-kube-api-access-frbg8\") pod \"node-exporter-86wbt\" (UID: \"297f9abf-ffb4-493a-8850-fc65648543a8\") " pod="openshift-monitoring/node-exporter-86wbt" Nov 02 06:17:02.870064 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:17:02.869979 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-86wbt" Nov 02 06:17:02.883607 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:17:02.883565 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod297f9abf_ffb4_493a_8850_fc65648543a8.slice/crio-586d7dd12b46c271a92cb332efc2d4f2ad85ac1c97932c7e7133ee0e4a491149 WatchSource:0}: Error finding container 586d7dd12b46c271a92cb332efc2d4f2ad85ac1c97932c7e7133ee0e4a491149: Status 404 returned error can't find the container with id 586d7dd12b46c271a92cb332efc2d4f2ad85ac1c97932c7e7133ee0e4a491149 Nov 02 06:17:02.885447 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:17:02.885429 5916 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 06:17:02.956500 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:17:02.956463 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-86wbt" event={"ID":"297f9abf-ffb4-493a-8850-fc65648543a8","Type":"ContainerStarted","Data":"586d7dd12b46c271a92cb332efc2d4f2ad85ac1c97932c7e7133ee0e4a491149"} Nov 02 06:17:03.125974 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:17:03.125877 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-hwc99" Nov 02 06:17:04.965773 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:17:04.965737 5916 generic.go:356] "Generic (PLEG): container finished" podID="297f9abf-ffb4-493a-8850-fc65648543a8" containerID="2e822dc2b814a7e1b98ad437ca2b1c45e20beb362fb8cdc83555fa2885830ce5" exitCode=0 Nov 02 06:17:04.966196 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:17:04.965845 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-86wbt" event={"ID":"297f9abf-ffb4-493a-8850-fc65648543a8","Type":"ContainerDied","Data":"2e822dc2b814a7e1b98ad437ca2b1c45e20beb362fb8cdc83555fa2885830ce5"} Nov 02 06:17:05.972091 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:17:05.972051 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-86wbt" event={"ID":"297f9abf-ffb4-493a-8850-fc65648543a8","Type":"ContainerStarted","Data":"e22106b8c61af828feed190d0f717743c066b4943dd20bd458636d000d03c665"} Nov 02 06:17:05.972594 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:17:05.972102 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-86wbt" event={"ID":"297f9abf-ffb4-493a-8850-fc65648543a8","Type":"ContainerStarted","Data":"62f8ff8248b294b0d6bbefd75ac4f2f90631d2bfc886d59819f265c57312b593"} Nov 02 06:17:06.127987 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:17:06.127931 5916 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-86wbt" podStartSLOduration=2.325148243 podStartE2EDuration="4.127898857s" totalImagesPullingTime="1.7334126s" totalInitContainerRuntime="69.338014ms" isStatefulPod=true podCreationTimestamp="2026-11-02 06:17:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:17:06.127753437 +0000 UTC m=+14.399573831" watchObservedRunningTime="2026-11-02 06:17:06.127898857 +0000 UTC m=+14.399719252" Nov 02 06:17:09.773453 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:17:09.773418 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-746899d75-dtjhs"] Nov 02 06:17:13.480016 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:17:13.479981 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-rrgb2-xp5b2"] Nov 02 06:17:13.480490 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:17:13.480181 5916 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-rrgb2-xp5b2" podUID="a4319db2-eda9-47a0-b742-985c8bdd7ea2" containerName="insights-gathering" containerID="cri-o://5cc58db3667246c8ed607f7c7c5dd57762103332123a56f39b752d4af4280288" gracePeriod=30 Nov 02 06:17:13.739871 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:17:13.739812 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-rrgb2-xp5b2" Nov 02 06:17:13.881322 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:17:13.881286 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4319db2-eda9-47a0-b742-985c8bdd7ea2-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a4319db2-eda9-47a0-b742-985c8bdd7ea2-trusted-ca-bundle\") pod \"a4319db2-eda9-47a0-b742-985c8bdd7ea2\" (UID: \"a4319db2-eda9-47a0-b742-985c8bdd7ea2\") " Nov 02 06:17:13.881492 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:17:13.881348 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4319db2-eda9-47a0-b742-985c8bdd7ea2-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/a4319db2-eda9-47a0-b742-985c8bdd7ea2-archives-path\") pod \"a4319db2-eda9-47a0-b742-985c8bdd7ea2\" (UID: \"a4319db2-eda9-47a0-b742-985c8bdd7ea2\") " Nov 02 06:17:13.881492 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:17:13.881374 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4319db2-eda9-47a0-b742-985c8bdd7ea2-kube-api-access-qwdcw\" (UniqueName: \"kubernetes.io/projected/a4319db2-eda9-47a0-b742-985c8bdd7ea2-kube-api-access-qwdcw\") pod \"a4319db2-eda9-47a0-b742-985c8bdd7ea2\" (UID: \"a4319db2-eda9-47a0-b742-985c8bdd7ea2\") " Nov 02 06:17:13.881492 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:17:13.881426 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4319db2-eda9-47a0-b742-985c8bdd7ea2-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a4319db2-eda9-47a0-b742-985c8bdd7ea2-service-ca-bundle\") pod \"a4319db2-eda9-47a0-b742-985c8bdd7ea2\" (UID: \"a4319db2-eda9-47a0-b742-985c8bdd7ea2\") " Nov 02 06:17:13.881752 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:17:13.881728 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4319db2-eda9-47a0-b742-985c8bdd7ea2-archives-path" pod "a4319db2-eda9-47a0-b742-985c8bdd7ea2" (UID: "a4319db2-eda9-47a0-b742-985c8bdd7ea2"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:17:13.881826 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:17:13.881743 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4319db2-eda9-47a0-b742-985c8bdd7ea2-trusted-ca-bundle" pod "a4319db2-eda9-47a0-b742-985c8bdd7ea2" (UID: "a4319db2-eda9-47a0-b742-985c8bdd7ea2"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:17:13.881880 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:17:13.881832 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4319db2-eda9-47a0-b742-985c8bdd7ea2-service-ca-bundle" pod "a4319db2-eda9-47a0-b742-985c8bdd7ea2" (UID: "a4319db2-eda9-47a0-b742-985c8bdd7ea2"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:17:13.883675 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:17:13.883653 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4319db2-eda9-47a0-b742-985c8bdd7ea2-kube-api-access-qwdcw" pod "a4319db2-eda9-47a0-b742-985c8bdd7ea2" (UID: "a4319db2-eda9-47a0-b742-985c8bdd7ea2"). InnerVolumeSpecName "kube-api-access-qwdcw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:17:13.982769 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:17:13.982725 5916 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a4319db2-eda9-47a0-b742-985c8bdd7ea2-trusted-ca-bundle\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:17:13.982769 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:17:13.982760 5916 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/a4319db2-eda9-47a0-b742-985c8bdd7ea2-archives-path\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:17:13.982769 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:17:13.982770 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwdcw\" (UniqueName: \"kubernetes.io/projected/a4319db2-eda9-47a0-b742-985c8bdd7ea2-kube-api-access-qwdcw\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:17:13.982769 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:17:13.982781 5916 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a4319db2-eda9-47a0-b742-985c8bdd7ea2-service-ca-bundle\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:17:14.001436 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:17:14.001355 5916 generic.go:356] "Generic (PLEG): container finished" podID="a4319db2-eda9-47a0-b742-985c8bdd7ea2" containerID="5cc58db3667246c8ed607f7c7c5dd57762103332123a56f39b752d4af4280288" exitCode=2 Nov 02 06:17:14.001436 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:17:14.001389 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-rrgb2-xp5b2" event={"ID":"a4319db2-eda9-47a0-b742-985c8bdd7ea2","Type":"ContainerDied","Data":"5cc58db3667246c8ed607f7c7c5dd57762103332123a56f39b752d4af4280288"} Nov 02 06:17:14.001436 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:17:14.001406 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-rrgb2-xp5b2" Nov 02 06:17:14.001687 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:17:14.001446 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-rrgb2-xp5b2" event={"ID":"a4319db2-eda9-47a0-b742-985c8bdd7ea2","Type":"ContainerDied","Data":"6f2af3023df9565bf072c1569930184b0aa5c7af28b5d5da6f6d32982f397446"} Nov 02 06:17:14.001687 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:17:14.001470 5916 scope.go:122] "RemoveContainer" containerID="5cc58db3667246c8ed607f7c7c5dd57762103332123a56f39b752d4af4280288" Nov 02 06:17:14.010405 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:17:14.010389 5916 scope.go:122] "RemoveContainer" containerID="5cc58db3667246c8ed607f7c7c5dd57762103332123a56f39b752d4af4280288" Nov 02 06:17:14.010740 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:17:14.010717 5916 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5cc58db3667246c8ed607f7c7c5dd57762103332123a56f39b752d4af4280288\": container with ID starting with 5cc58db3667246c8ed607f7c7c5dd57762103332123a56f39b752d4af4280288 not found: ID does not exist" containerID="5cc58db3667246c8ed607f7c7c5dd57762103332123a56f39b752d4af4280288" Nov 02 06:17:14.010816 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:17:14.010753 5916 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cc58db3667246c8ed607f7c7c5dd57762103332123a56f39b752d4af4280288"} err="failed to get container status \"5cc58db3667246c8ed607f7c7c5dd57762103332123a56f39b752d4af4280288\": rpc error: code = NotFound desc = could not find container \"5cc58db3667246c8ed607f7c7c5dd57762103332123a56f39b752d4af4280288\": container with ID starting with 5cc58db3667246c8ed607f7c7c5dd57762103332123a56f39b752d4af4280288 not found: ID does not exist" Nov 02 06:17:14.151980 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:17:14.151950 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-rrgb2-xp5b2"] Nov 02 06:17:14.157539 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:17:14.157515 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-rrgb2-xp5b2"] Nov 02 06:17:15.840246 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:17:15.840168 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4319db2-eda9-47a0-b742-985c8bdd7ea2" path="/var/lib/kubelet/pods/a4319db2-eda9-47a0-b742-985c8bdd7ea2/volumes" Nov 02 06:17:17.921190 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:17:17.921150 5916 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-6689c6d46b-vw558" podUID="f0531b4d-c954-4830-85f6-fe876c76437a" containerName="registry" containerID="cri-o://8d5f9daaf11f23bb864f8bc6039339327cfe071ed1a68abec1f2b1a7919f8a3d" gracePeriod=30 Nov 02 06:17:18.192683 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:17:18.192621 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6689c6d46b-vw558" Nov 02 06:17:18.322964 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:17:18.322936 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0531b4d-c954-4830-85f6-fe876c76437a-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f0531b4d-c954-4830-85f6-fe876c76437a-bound-sa-token\") pod \"f0531b4d-c954-4830-85f6-fe876c76437a\" (UID: \"f0531b4d-c954-4830-85f6-fe876c76437a\") " Nov 02 06:17:18.323160 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:17:18.322974 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f0531b4d-c954-4830-85f6-fe876c76437a-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f0531b4d-c954-4830-85f6-fe876c76437a-installation-pull-secrets\") pod \"f0531b4d-c954-4830-85f6-fe876c76437a\" (UID: \"f0531b4d-c954-4830-85f6-fe876c76437a\") " Nov 02 06:17:18.323160 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:17:18.323002 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0531b4d-c954-4830-85f6-fe876c76437a-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f0531b4d-c954-4830-85f6-fe876c76437a-trusted-ca\") pod \"f0531b4d-c954-4830-85f6-fe876c76437a\" (UID: \"f0531b4d-c954-4830-85f6-fe876c76437a\") " Nov 02 06:17:18.323160 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:17:18.323024 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0531b4d-c954-4830-85f6-fe876c76437a-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f0531b4d-c954-4830-85f6-fe876c76437a-ca-trust-extracted\") pod \"f0531b4d-c954-4830-85f6-fe876c76437a\" (UID: \"f0531b4d-c954-4830-85f6-fe876c76437a\") " Nov 02 06:17:18.323160 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:17:18.323057 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0531b4d-c954-4830-85f6-fe876c76437a-registry-tls\" (UniqueName: \"kubernetes.io/projected/f0531b4d-c954-4830-85f6-fe876c76437a-registry-tls\") pod \"f0531b4d-c954-4830-85f6-fe876c76437a\" (UID: \"f0531b4d-c954-4830-85f6-fe876c76437a\") " Nov 02 06:17:18.323160 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:17:18.323096 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0531b4d-c954-4830-85f6-fe876c76437a-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f0531b4d-c954-4830-85f6-fe876c76437a-registry-certificates\") pod \"f0531b4d-c954-4830-85f6-fe876c76437a\" (UID: \"f0531b4d-c954-4830-85f6-fe876c76437a\") " Nov 02 06:17:18.323160 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:17:18.323132 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f0531b4d-c954-4830-85f6-fe876c76437a-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f0531b4d-c954-4830-85f6-fe876c76437a-image-registry-private-configuration\") pod \"f0531b4d-c954-4830-85f6-fe876c76437a\" (UID: \"f0531b4d-c954-4830-85f6-fe876c76437a\") " Nov 02 06:17:18.323486 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:17:18.323169 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0531b4d-c954-4830-85f6-fe876c76437a-kube-api-access-m89g8\" (UniqueName: \"kubernetes.io/projected/f0531b4d-c954-4830-85f6-fe876c76437a-kube-api-access-m89g8\") pod \"f0531b4d-c954-4830-85f6-fe876c76437a\" (UID: \"f0531b4d-c954-4830-85f6-fe876c76437a\") " Nov 02 06:17:18.323577 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:17:18.323543 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0531b4d-c954-4830-85f6-fe876c76437a-trusted-ca" pod "f0531b4d-c954-4830-85f6-fe876c76437a" (UID: "f0531b4d-c954-4830-85f6-fe876c76437a"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:17:18.324221 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:17:18.324180 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0531b4d-c954-4830-85f6-fe876c76437a-registry-certificates" pod "f0531b4d-c954-4830-85f6-fe876c76437a" (UID: "f0531b4d-c954-4830-85f6-fe876c76437a"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:17:18.325473 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:17:18.325447 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0531b4d-c954-4830-85f6-fe876c76437a-bound-sa-token" pod "f0531b4d-c954-4830-85f6-fe876c76437a" (UID: "f0531b4d-c954-4830-85f6-fe876c76437a"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:17:18.325804 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:17:18.325766 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0531b4d-c954-4830-85f6-fe876c76437a-image-registry-private-configuration" pod "f0531b4d-c954-4830-85f6-fe876c76437a" (UID: "f0531b4d-c954-4830-85f6-fe876c76437a"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:17:18.325888 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:17:18.325804 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0531b4d-c954-4830-85f6-fe876c76437a-registry-tls" pod "f0531b4d-c954-4830-85f6-fe876c76437a" (UID: "f0531b4d-c954-4830-85f6-fe876c76437a"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:17:18.325973 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:17:18.325945 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0531b4d-c954-4830-85f6-fe876c76437a-installation-pull-secrets" pod "f0531b4d-c954-4830-85f6-fe876c76437a" (UID: "f0531b4d-c954-4830-85f6-fe876c76437a"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:17:18.326018 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:17:18.325978 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0531b4d-c954-4830-85f6-fe876c76437a-kube-api-access-m89g8" pod "f0531b4d-c954-4830-85f6-fe876c76437a" (UID: "f0531b4d-c954-4830-85f6-fe876c76437a"). InnerVolumeSpecName "kube-api-access-m89g8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:17:18.330190 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:17:18.330169 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0531b4d-c954-4830-85f6-fe876c76437a-ca-trust-extracted" pod "f0531b4d-c954-4830-85f6-fe876c76437a" (UID: "f0531b4d-c954-4830-85f6-fe876c76437a"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:17:18.424136 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:17:18.424101 5916 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f0531b4d-c954-4830-85f6-fe876c76437a-ca-trust-extracted\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:17:18.424136 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:17:18.424130 5916 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f0531b4d-c954-4830-85f6-fe876c76437a-registry-tls\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:17:18.424136 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:17:18.424140 5916 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f0531b4d-c954-4830-85f6-fe876c76437a-registry-certificates\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:17:18.424373 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:17:18.424150 5916 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f0531b4d-c954-4830-85f6-fe876c76437a-image-registry-private-configuration\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:17:18.424373 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:17:18.424162 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m89g8\" (UniqueName: \"kubernetes.io/projected/f0531b4d-c954-4830-85f6-fe876c76437a-kube-api-access-m89g8\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:17:18.424373 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:17:18.424179 5916 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f0531b4d-c954-4830-85f6-fe876c76437a-bound-sa-token\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:17:18.424373 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:17:18.424189 5916 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f0531b4d-c954-4830-85f6-fe876c76437a-installation-pull-secrets\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:17:18.424373 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:17:18.424197 5916 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f0531b4d-c954-4830-85f6-fe876c76437a-trusted-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:17:19.021687 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:17:19.021652 5916 generic.go:356] "Generic (PLEG): container finished" podID="f0531b4d-c954-4830-85f6-fe876c76437a" containerID="8d5f9daaf11f23bb864f8bc6039339327cfe071ed1a68abec1f2b1a7919f8a3d" exitCode=0 Nov 02 06:17:19.022152 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:17:19.021701 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6689c6d46b-vw558" Nov 02 06:17:19.022152 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:17:19.021737 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6689c6d46b-vw558" event={"ID":"f0531b4d-c954-4830-85f6-fe876c76437a","Type":"ContainerDied","Data":"8d5f9daaf11f23bb864f8bc6039339327cfe071ed1a68abec1f2b1a7919f8a3d"} Nov 02 06:17:19.022152 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:17:19.021788 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6689c6d46b-vw558" event={"ID":"f0531b4d-c954-4830-85f6-fe876c76437a","Type":"ContainerDied","Data":"a471a58ea4a732d84e89fb09f437def55b26113096801bc4a4e213c7894fbe5b"} Nov 02 06:17:19.022152 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:17:19.021807 5916 scope.go:122] "RemoveContainer" containerID="8d5f9daaf11f23bb864f8bc6039339327cfe071ed1a68abec1f2b1a7919f8a3d" Nov 02 06:17:19.030600 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:17:19.030581 5916 scope.go:122] "RemoveContainer" containerID="8d5f9daaf11f23bb864f8bc6039339327cfe071ed1a68abec1f2b1a7919f8a3d" Nov 02 06:17:19.030848 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:17:19.030830 5916 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d5f9daaf11f23bb864f8bc6039339327cfe071ed1a68abec1f2b1a7919f8a3d\": container with ID starting with 8d5f9daaf11f23bb864f8bc6039339327cfe071ed1a68abec1f2b1a7919f8a3d not found: ID does not exist" containerID="8d5f9daaf11f23bb864f8bc6039339327cfe071ed1a68abec1f2b1a7919f8a3d" Nov 02 06:17:19.030924 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:17:19.030855 5916 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d5f9daaf11f23bb864f8bc6039339327cfe071ed1a68abec1f2b1a7919f8a3d"} err="failed to get container status \"8d5f9daaf11f23bb864f8bc6039339327cfe071ed1a68abec1f2b1a7919f8a3d\": rpc error: code = NotFound desc = could not find container \"8d5f9daaf11f23bb864f8bc6039339327cfe071ed1a68abec1f2b1a7919f8a3d\": container with ID starting with 8d5f9daaf11f23bb864f8bc6039339327cfe071ed1a68abec1f2b1a7919f8a3d not found: ID does not exist" Nov 02 06:17:19.219759 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:17:19.219724 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6689c6d46b-vw558"] Nov 02 06:17:19.265739 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:17:19.265710 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6689c6d46b-vw558"] Nov 02 06:17:19.839924 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:17:19.839873 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0531b4d-c954-4830-85f6-fe876c76437a" path="/var/lib/kubelet/pods/f0531b4d-c954-4830-85f6-fe876c76437a/volumes" Nov 02 06:17:19.893746 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:17:19.893711 5916 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p98cq" containerName="extractor" Nov 02 06:17:22.691850 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:17:22.691818 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-ft7gs" Nov 02 06:17:34.793860 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:17:34.793787 5916 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-746899d75-dtjhs" podUID="fa9c5a91-d480-4d7b-9044-be6437a76a02" containerName="registry" containerID="cri-o://8ec81ab82bca12552c048070bb8e6d8403afabfda731aa44831bb2fb8e97cf17" gracePeriod=30 Nov 02 06:17:35.049798 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:17:35.049734 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-746899d75-dtjhs" Nov 02 06:17:35.080315 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:17:35.080288 5916 generic.go:356] "Generic (PLEG): container finished" podID="fa9c5a91-d480-4d7b-9044-be6437a76a02" containerID="8ec81ab82bca12552c048070bb8e6d8403afabfda731aa44831bb2fb8e97cf17" exitCode=0 Nov 02 06:17:35.080463 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:17:35.080327 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-746899d75-dtjhs" event={"ID":"fa9c5a91-d480-4d7b-9044-be6437a76a02","Type":"ContainerDied","Data":"8ec81ab82bca12552c048070bb8e6d8403afabfda731aa44831bb2fb8e97cf17"} Nov 02 06:17:35.080463 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:17:35.080332 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-746899d75-dtjhs" Nov 02 06:17:35.080463 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:17:35.080361 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-746899d75-dtjhs" event={"ID":"fa9c5a91-d480-4d7b-9044-be6437a76a02","Type":"ContainerDied","Data":"1309dfc627fd0eddb9d56bfeb55c303d58564eed884481f3f3119988a12fbe55"} Nov 02 06:17:35.080463 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:17:35.080380 5916 scope.go:122] "RemoveContainer" containerID="8ec81ab82bca12552c048070bb8e6d8403afabfda731aa44831bb2fb8e97cf17" Nov 02 06:17:35.087986 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:17:35.087969 5916 scope.go:122] "RemoveContainer" containerID="8ec81ab82bca12552c048070bb8e6d8403afabfda731aa44831bb2fb8e97cf17" Nov 02 06:17:35.088246 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:17:35.088225 5916 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ec81ab82bca12552c048070bb8e6d8403afabfda731aa44831bb2fb8e97cf17\": container with ID starting with 8ec81ab82bca12552c048070bb8e6d8403afabfda731aa44831bb2fb8e97cf17 not found: ID does not exist" containerID="8ec81ab82bca12552c048070bb8e6d8403afabfda731aa44831bb2fb8e97cf17" Nov 02 06:17:35.088311 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:17:35.088260 5916 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ec81ab82bca12552c048070bb8e6d8403afabfda731aa44831bb2fb8e97cf17"} err="failed to get container status \"8ec81ab82bca12552c048070bb8e6d8403afabfda731aa44831bb2fb8e97cf17\": rpc error: code = NotFound desc = could not find container \"8ec81ab82bca12552c048070bb8e6d8403afabfda731aa44831bb2fb8e97cf17\": container with ID starting with 8ec81ab82bca12552c048070bb8e6d8403afabfda731aa44831bb2fb8e97cf17 not found: ID does not exist" Nov 02 06:17:35.158458 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:17:35.158430 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa9c5a91-d480-4d7b-9044-be6437a76a02-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fa9c5a91-d480-4d7b-9044-be6437a76a02-ca-trust-extracted\") pod \"fa9c5a91-d480-4d7b-9044-be6437a76a02\" (UID: \"fa9c5a91-d480-4d7b-9044-be6437a76a02\") " Nov 02 06:17:35.158570 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:17:35.158490 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fa9c5a91-d480-4d7b-9044-be6437a76a02-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fa9c5a91-d480-4d7b-9044-be6437a76a02-installation-pull-secrets\") pod \"fa9c5a91-d480-4d7b-9044-be6437a76a02\" (UID: \"fa9c5a91-d480-4d7b-9044-be6437a76a02\") " Nov 02 06:17:35.158570 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:17:35.158535 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fa9c5a91-d480-4d7b-9044-be6437a76a02-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fa9c5a91-d480-4d7b-9044-be6437a76a02-image-registry-private-configuration\") pod \"fa9c5a91-d480-4d7b-9044-be6437a76a02\" (UID: \"fa9c5a91-d480-4d7b-9044-be6437a76a02\") " Nov 02 06:17:35.158570 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:17:35.158565 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa9c5a91-d480-4d7b-9044-be6437a76a02-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fa9c5a91-d480-4d7b-9044-be6437a76a02-registry-certificates\") pod \"fa9c5a91-d480-4d7b-9044-be6437a76a02\" (UID: \"fa9c5a91-d480-4d7b-9044-be6437a76a02\") " Nov 02 06:17:35.158739 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:17:35.158599 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa9c5a91-d480-4d7b-9044-be6437a76a02-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fa9c5a91-d480-4d7b-9044-be6437a76a02-bound-sa-token\") pod \"fa9c5a91-d480-4d7b-9044-be6437a76a02\" (UID: \"fa9c5a91-d480-4d7b-9044-be6437a76a02\") " Nov 02 06:17:35.158739 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:17:35.158628 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa9c5a91-d480-4d7b-9044-be6437a76a02-kube-api-access-cctkp\" (UniqueName: \"kubernetes.io/projected/fa9c5a91-d480-4d7b-9044-be6437a76a02-kube-api-access-cctkp\") pod \"fa9c5a91-d480-4d7b-9044-be6437a76a02\" (UID: \"fa9c5a91-d480-4d7b-9044-be6437a76a02\") " Nov 02 06:17:35.158739 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:17:35.158709 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa9c5a91-d480-4d7b-9044-be6437a76a02-registry-tls\" (UniqueName: \"kubernetes.io/projected/fa9c5a91-d480-4d7b-9044-be6437a76a02-registry-tls\") pod \"fa9c5a91-d480-4d7b-9044-be6437a76a02\" (UID: \"fa9c5a91-d480-4d7b-9044-be6437a76a02\") " Nov 02 06:17:35.158886 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:17:35.158777 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa9c5a91-d480-4d7b-9044-be6437a76a02-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fa9c5a91-d480-4d7b-9044-be6437a76a02-trusted-ca\") pod \"fa9c5a91-d480-4d7b-9044-be6437a76a02\" (UID: \"fa9c5a91-d480-4d7b-9044-be6437a76a02\") " Nov 02 06:17:35.159002 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:17:35.158974 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa9c5a91-d480-4d7b-9044-be6437a76a02-registry-certificates" pod "fa9c5a91-d480-4d7b-9044-be6437a76a02" (UID: "fa9c5a91-d480-4d7b-9044-be6437a76a02"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:17:35.159331 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:17:35.159308 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa9c5a91-d480-4d7b-9044-be6437a76a02-trusted-ca" pod "fa9c5a91-d480-4d7b-9044-be6437a76a02" (UID: "fa9c5a91-d480-4d7b-9044-be6437a76a02"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:17:35.161150 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:17:35.161119 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa9c5a91-d480-4d7b-9044-be6437a76a02-image-registry-private-configuration" pod "fa9c5a91-d480-4d7b-9044-be6437a76a02" (UID: "fa9c5a91-d480-4d7b-9044-be6437a76a02"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:17:35.161317 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:17:35.161299 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa9c5a91-d480-4d7b-9044-be6437a76a02-installation-pull-secrets" pod "fa9c5a91-d480-4d7b-9044-be6437a76a02" (UID: "fa9c5a91-d480-4d7b-9044-be6437a76a02"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:17:35.161382 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:17:35.161322 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa9c5a91-d480-4d7b-9044-be6437a76a02-bound-sa-token" pod "fa9c5a91-d480-4d7b-9044-be6437a76a02" (UID: "fa9c5a91-d480-4d7b-9044-be6437a76a02"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:17:35.161422 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:17:35.161399 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa9c5a91-d480-4d7b-9044-be6437a76a02-registry-tls" pod "fa9c5a91-d480-4d7b-9044-be6437a76a02" (UID: "fa9c5a91-d480-4d7b-9044-be6437a76a02"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:17:35.161608 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:17:35.161588 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa9c5a91-d480-4d7b-9044-be6437a76a02-kube-api-access-cctkp" pod "fa9c5a91-d480-4d7b-9044-be6437a76a02" (UID: "fa9c5a91-d480-4d7b-9044-be6437a76a02"). InnerVolumeSpecName "kube-api-access-cctkp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:17:35.165647 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:17:35.165545 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa9c5a91-d480-4d7b-9044-be6437a76a02-ca-trust-extracted" pod "fa9c5a91-d480-4d7b-9044-be6437a76a02" (UID: "fa9c5a91-d480-4d7b-9044-be6437a76a02"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:17:35.259612 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:17:35.259572 5916 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fa9c5a91-d480-4d7b-9044-be6437a76a02-registry-tls\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:17:35.259612 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:17:35.259599 5916 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fa9c5a91-d480-4d7b-9044-be6437a76a02-trusted-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:17:35.259612 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:17:35.259609 5916 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fa9c5a91-d480-4d7b-9044-be6437a76a02-ca-trust-extracted\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:17:35.259612 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:17:35.259619 5916 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fa9c5a91-d480-4d7b-9044-be6437a76a02-installation-pull-secrets\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:17:35.259984 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:17:35.259630 5916 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fa9c5a91-d480-4d7b-9044-be6437a76a02-image-registry-private-configuration\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:17:35.259984 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:17:35.259639 5916 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fa9c5a91-d480-4d7b-9044-be6437a76a02-registry-certificates\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:17:35.259984 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:17:35.259669 5916 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fa9c5a91-d480-4d7b-9044-be6437a76a02-bound-sa-token\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:17:35.259984 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:17:35.259679 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cctkp\" (UniqueName: \"kubernetes.io/projected/fa9c5a91-d480-4d7b-9044-be6437a76a02-kube-api-access-cctkp\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:17:35.492735 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:17:35.492700 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-746899d75-dtjhs"] Nov 02 06:17:35.497560 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:17:35.497502 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-746899d75-dtjhs"] Nov 02 06:17:35.840774 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:17:35.840695 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa9c5a91-d480-4d7b-9044-be6437a76a02" path="/var/lib/kubelet/pods/fa9c5a91-d480-4d7b-9044-be6437a76a02/volumes" Nov 02 06:17:49.893513 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:17:49.893472 5916 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p98cq" containerName="extractor" Nov 02 06:18:19.892490 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:18:19.892452 5916 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p98cq" containerName="extractor" Nov 02 06:18:49.893760 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:18:49.893680 5916 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p98cq" containerName="extractor" Nov 02 06:19:19.893046 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:19:19.893009 5916 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p98cq" containerName="extractor" Nov 02 06:19:49.892984 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:19:49.892938 5916 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p98cq" containerName="extractor" Nov 02 06:20:03.258523 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:03.258290 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agdmqn"] Nov 02 06:20:03.259011 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:03.258720 5916 cpu_manager.go:414] "containerMap: removing container" podUID="a4319db2-eda9-47a0-b742-985c8bdd7ea2" containerName="insights-gathering" Nov 02 06:20:03.259011 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:03.258739 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4319db2-eda9-47a0-b742-985c8bdd7ea2" containerName="insights-gathering" Nov 02 06:20:03.259011 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:03.258772 5916 cpu_manager.go:414] "containerMap: removing container" podUID="f0531b4d-c954-4830-85f6-fe876c76437a" containerName="registry" Nov 02 06:20:03.259011 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:03.258781 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0531b4d-c954-4830-85f6-fe876c76437a" containerName="registry" Nov 02 06:20:03.259011 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:03.258790 5916 cpu_manager.go:414] "containerMap: removing container" podUID="fa9c5a91-d480-4d7b-9044-be6437a76a02" containerName="registry" Nov 02 06:20:03.259011 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:03.258799 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa9c5a91-d480-4d7b-9044-be6437a76a02" containerName="registry" Nov 02 06:20:03.259011 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:03.258888 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0531b4d-c954-4830-85f6-fe876c76437a" containerName="registry" Nov 02 06:20:03.259011 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:03.258899 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa9c5a91-d480-4d7b-9044-be6437a76a02" containerName="registry" Nov 02 06:20:03.259011 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:03.258931 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4319db2-eda9-47a0-b742-985c8bdd7ea2" containerName="insights-gathering" Nov 02 06:20:03.262246 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:03.262225 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agdmqn" Nov 02 06:20:03.356310 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:03.356267 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-gt7gf\"" Nov 02 06:20:03.356310 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:03.356295 5916 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Nov 02 06:20:03.356540 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:03.356439 5916 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Nov 02 06:20:03.357145 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:03.357122 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agdmqn"] Nov 02 06:20:03.365376 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:03.365359 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jjdpk"] Nov 02 06:20:03.368722 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:03.368701 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jjdpk" Nov 02 06:20:03.394178 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:03.394146 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c48449eb-30cb-4b07-892a-b27192f49e0f-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agdmqn\" (UID: \"c48449eb-30cb-4b07-892a-b27192f49e0f\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agdmqn" Nov 02 06:20:03.394311 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:03.394181 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c48449eb-30cb-4b07-892a-b27192f49e0f-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agdmqn\" (UID: \"c48449eb-30cb-4b07-892a-b27192f49e0f\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agdmqn" Nov 02 06:20:03.394311 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:03.394257 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpvg7\" (UniqueName: \"kubernetes.io/projected/c48449eb-30cb-4b07-892a-b27192f49e0f-kube-api-access-zpvg7\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agdmqn\" (UID: \"c48449eb-30cb-4b07-892a-b27192f49e0f\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agdmqn" Nov 02 06:20:03.417246 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:03.417221 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-psnf7\"" Nov 02 06:20:03.427855 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:03.427824 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jjdpk"] Nov 02 06:20:03.463329 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:03.463290 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mfmhs"] Nov 02 06:20:03.466724 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:03.466704 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mfmhs" Nov 02 06:20:03.494791 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:03.494760 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpvg7\" (UniqueName: \"kubernetes.io/projected/c48449eb-30cb-4b07-892a-b27192f49e0f-kube-api-access-zpvg7\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agdmqn\" (UID: \"c48449eb-30cb-4b07-892a-b27192f49e0f\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agdmqn" Nov 02 06:20:03.494986 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:03.494801 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/70e3b8eb-ed1f-4552-8caf-113f26180ee5-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jjdpk\" (UID: \"70e3b8eb-ed1f-4552-8caf-113f26180ee5\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jjdpk" Nov 02 06:20:03.494986 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:03.494866 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/70e3b8eb-ed1f-4552-8caf-113f26180ee5-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jjdpk\" (UID: \"70e3b8eb-ed1f-4552-8caf-113f26180ee5\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jjdpk" Nov 02 06:20:03.494986 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:03.494970 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c48449eb-30cb-4b07-892a-b27192f49e0f-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agdmqn\" (UID: \"c48449eb-30cb-4b07-892a-b27192f49e0f\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agdmqn" Nov 02 06:20:03.495127 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:03.495039 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c48449eb-30cb-4b07-892a-b27192f49e0f-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agdmqn\" (UID: \"c48449eb-30cb-4b07-892a-b27192f49e0f\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agdmqn" Nov 02 06:20:03.495127 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:03.495070 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4gxr\" (UniqueName: \"kubernetes.io/projected/70e3b8eb-ed1f-4552-8caf-113f26180ee5-kube-api-access-d4gxr\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jjdpk\" (UID: \"70e3b8eb-ed1f-4552-8caf-113f26180ee5\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jjdpk" Nov 02 06:20:03.495334 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:03.495318 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c48449eb-30cb-4b07-892a-b27192f49e0f-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agdmqn\" (UID: \"c48449eb-30cb-4b07-892a-b27192f49e0f\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agdmqn" Nov 02 06:20:03.495383 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:03.495357 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c48449eb-30cb-4b07-892a-b27192f49e0f-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agdmqn\" (UID: \"c48449eb-30cb-4b07-892a-b27192f49e0f\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agdmqn" Nov 02 06:20:03.517213 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:03.517148 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-smj84\"" Nov 02 06:20:03.520815 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:03.520789 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mfmhs"] Nov 02 06:20:03.547995 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:03.547967 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpvg7\" (UniqueName: \"kubernetes.io/projected/c48449eb-30cb-4b07-892a-b27192f49e0f-kube-api-access-zpvg7\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agdmqn\" (UID: \"c48449eb-30cb-4b07-892a-b27192f49e0f\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agdmqn" Nov 02 06:20:03.560174 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:03.560143 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chkj82"] Nov 02 06:20:03.564003 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:03.563982 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chkj82" Nov 02 06:20:03.579449 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:03.579426 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agdmqn" Nov 02 06:20:03.596325 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:03.596295 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/70e3b8eb-ed1f-4552-8caf-113f26180ee5-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jjdpk\" (UID: \"70e3b8eb-ed1f-4552-8caf-113f26180ee5\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jjdpk" Nov 02 06:20:03.596456 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:03.596342 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aa5d2e0d-5b30-415e-86b0-d3a7e246ffa7-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mfmhs\" (UID: \"aa5d2e0d-5b30-415e-86b0-d3a7e246ffa7\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mfmhs" Nov 02 06:20:03.596456 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:03.596368 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jv28r\" (UniqueName: \"kubernetes.io/projected/aa5d2e0d-5b30-415e-86b0-d3a7e246ffa7-kube-api-access-jv28r\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mfmhs\" (UID: \"aa5d2e0d-5b30-415e-86b0-d3a7e246ffa7\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mfmhs" Nov 02 06:20:03.596456 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:03.596400 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aa5d2e0d-5b30-415e-86b0-d3a7e246ffa7-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mfmhs\" (UID: \"aa5d2e0d-5b30-415e-86b0-d3a7e246ffa7\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mfmhs" Nov 02 06:20:03.596618 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:03.596547 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4gxr\" (UniqueName: \"kubernetes.io/projected/70e3b8eb-ed1f-4552-8caf-113f26180ee5-kube-api-access-d4gxr\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jjdpk\" (UID: \"70e3b8eb-ed1f-4552-8caf-113f26180ee5\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jjdpk" Nov 02 06:20:03.596618 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:03.596593 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/70e3b8eb-ed1f-4552-8caf-113f26180ee5-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jjdpk\" (UID: \"70e3b8eb-ed1f-4552-8caf-113f26180ee5\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jjdpk" Nov 02 06:20:03.596717 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:03.596659 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/70e3b8eb-ed1f-4552-8caf-113f26180ee5-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jjdpk\" (UID: \"70e3b8eb-ed1f-4552-8caf-113f26180ee5\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jjdpk" Nov 02 06:20:03.596977 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:03.596955 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/70e3b8eb-ed1f-4552-8caf-113f26180ee5-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jjdpk\" (UID: \"70e3b8eb-ed1f-4552-8caf-113f26180ee5\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jjdpk" Nov 02 06:20:03.617366 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:03.617138 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chkj82"] Nov 02 06:20:03.617752 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:03.617563 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-rgnn7\"" Nov 02 06:20:03.650644 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:03.650615 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4gxr\" (UniqueName: \"kubernetes.io/projected/70e3b8eb-ed1f-4552-8caf-113f26180ee5-kube-api-access-d4gxr\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jjdpk\" (UID: \"70e3b8eb-ed1f-4552-8caf-113f26180ee5\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jjdpk" Nov 02 06:20:03.684526 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:03.684493 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jjdpk" Nov 02 06:20:03.697777 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:03.697752 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a7ef2c80-8b0c-4c7a-a5c1-ddb795124cbb-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chkj82\" (UID: \"a7ef2c80-8b0c-4c7a-a5c1-ddb795124cbb\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chkj82" Nov 02 06:20:03.697882 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:03.697820 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a7ef2c80-8b0c-4c7a-a5c1-ddb795124cbb-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chkj82\" (UID: \"a7ef2c80-8b0c-4c7a-a5c1-ddb795124cbb\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chkj82" Nov 02 06:20:03.697961 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:03.697945 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aa5d2e0d-5b30-415e-86b0-d3a7e246ffa7-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mfmhs\" (UID: \"aa5d2e0d-5b30-415e-86b0-d3a7e246ffa7\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mfmhs" Nov 02 06:20:03.698018 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:03.697995 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jv28r\" (UniqueName: \"kubernetes.io/projected/aa5d2e0d-5b30-415e-86b0-d3a7e246ffa7-kube-api-access-jv28r\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mfmhs\" (UID: \"aa5d2e0d-5b30-415e-86b0-d3a7e246ffa7\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mfmhs" Nov 02 06:20:03.698079 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:03.698038 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmf8j\" (UniqueName: \"kubernetes.io/projected/a7ef2c80-8b0c-4c7a-a5c1-ddb795124cbb-kube-api-access-kmf8j\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chkj82\" (UID: \"a7ef2c80-8b0c-4c7a-a5c1-ddb795124cbb\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chkj82" Nov 02 06:20:03.698079 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:03.698071 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aa5d2e0d-5b30-415e-86b0-d3a7e246ffa7-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mfmhs\" (UID: \"aa5d2e0d-5b30-415e-86b0-d3a7e246ffa7\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mfmhs" Nov 02 06:20:03.698526 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:03.698503 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aa5d2e0d-5b30-415e-86b0-d3a7e246ffa7-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mfmhs\" (UID: \"aa5d2e0d-5b30-415e-86b0-d3a7e246ffa7\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mfmhs" Nov 02 06:20:03.698625 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:03.698560 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aa5d2e0d-5b30-415e-86b0-d3a7e246ffa7-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mfmhs\" (UID: \"aa5d2e0d-5b30-415e-86b0-d3a7e246ffa7\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mfmhs" Nov 02 06:20:03.753502 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:03.753466 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jv28r\" (UniqueName: \"kubernetes.io/projected/aa5d2e0d-5b30-415e-86b0-d3a7e246ffa7-kube-api-access-jv28r\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mfmhs\" (UID: \"aa5d2e0d-5b30-415e-86b0-d3a7e246ffa7\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mfmhs" Nov 02 06:20:03.782325 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:03.782240 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mfmhs" Nov 02 06:20:03.798678 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:03.798648 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a7ef2c80-8b0c-4c7a-a5c1-ddb795124cbb-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chkj82\" (UID: \"a7ef2c80-8b0c-4c7a-a5c1-ddb795124cbb\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chkj82" Nov 02 06:20:03.798794 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:03.798744 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a7ef2c80-8b0c-4c7a-a5c1-ddb795124cbb-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chkj82\" (UID: \"a7ef2c80-8b0c-4c7a-a5c1-ddb795124cbb\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chkj82" Nov 02 06:20:03.798848 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:03.798798 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmf8j\" (UniqueName: \"kubernetes.io/projected/a7ef2c80-8b0c-4c7a-a5c1-ddb795124cbb-kube-api-access-kmf8j\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chkj82\" (UID: \"a7ef2c80-8b0c-4c7a-a5c1-ddb795124cbb\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chkj82" Nov 02 06:20:03.799019 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:03.798997 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a7ef2c80-8b0c-4c7a-a5c1-ddb795124cbb-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chkj82\" (UID: \"a7ef2c80-8b0c-4c7a-a5c1-ddb795124cbb\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chkj82" Nov 02 06:20:03.799215 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:03.799186 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a7ef2c80-8b0c-4c7a-a5c1-ddb795124cbb-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chkj82\" (UID: \"a7ef2c80-8b0c-4c7a-a5c1-ddb795124cbb\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chkj82" Nov 02 06:20:03.853109 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:03.853076 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmf8j\" (UniqueName: \"kubernetes.io/projected/a7ef2c80-8b0c-4c7a-a5c1-ddb795124cbb-kube-api-access-kmf8j\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chkj82\" (UID: \"a7ef2c80-8b0c-4c7a-a5c1-ddb795124cbb\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chkj82" Nov 02 06:20:03.880498 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:03.880467 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chkj82" Nov 02 06:20:03.905329 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:03.905303 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2rnzs"] Nov 02 06:20:03.909019 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:03.908997 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2rnzs" Nov 02 06:20:03.945747 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:03.945716 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agdmqn"] Nov 02 06:20:03.954958 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:20:03.954884 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc48449eb_30cb_4b07_892a_b27192f49e0f.slice/crio-a6cfc876b9b8d8fbfdc462627e6ac2e5b4d41308ede7c1c76a2d43302470e228 WatchSource:0}: Error finding container a6cfc876b9b8d8fbfdc462627e6ac2e5b4d41308ede7c1c76a2d43302470e228: Status 404 returned error can't find the container with id a6cfc876b9b8d8fbfdc462627e6ac2e5b4d41308ede7c1c76a2d43302470e228 Nov 02 06:20:03.957613 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:03.957589 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-7znv2\"" Nov 02 06:20:03.961479 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:03.961453 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2rnzs"] Nov 02 06:20:04.000788 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:04.000753 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/368fe2a8-af6b-4d60-a6cf-6a82dff993a0-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2rnzs\" (UID: \"368fe2a8-af6b-4d60-a6cf-6a82dff993a0\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2rnzs" Nov 02 06:20:04.000788 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:04.000793 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/368fe2a8-af6b-4d60-a6cf-6a82dff993a0-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2rnzs\" (UID: \"368fe2a8-af6b-4d60-a6cf-6a82dff993a0\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2rnzs" Nov 02 06:20:04.001044 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:04.000875 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntgl4\" (UniqueName: \"kubernetes.io/projected/368fe2a8-af6b-4d60-a6cf-6a82dff993a0-kube-api-access-ntgl4\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2rnzs\" (UID: \"368fe2a8-af6b-4d60-a6cf-6a82dff993a0\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2rnzs" Nov 02 06:20:04.055528 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:04.055444 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jjdpk"] Nov 02 06:20:04.062595 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:20:04.062567 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70e3b8eb_ed1f_4552_8caf_113f26180ee5.slice/crio-7cd5682b30d82cf837c97ee33e87c6fdd5a6c75be0a5b7cd695c64d81593e102 WatchSource:0}: Error finding container 7cd5682b30d82cf837c97ee33e87c6fdd5a6c75be0a5b7cd695c64d81593e102: Status 404 returned error can't find the container with id 7cd5682b30d82cf837c97ee33e87c6fdd5a6c75be0a5b7cd695c64d81593e102 Nov 02 06:20:04.102252 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:04.102215 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/368fe2a8-af6b-4d60-a6cf-6a82dff993a0-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2rnzs\" (UID: \"368fe2a8-af6b-4d60-a6cf-6a82dff993a0\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2rnzs" Nov 02 06:20:04.102252 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:04.102255 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/368fe2a8-af6b-4d60-a6cf-6a82dff993a0-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2rnzs\" (UID: \"368fe2a8-af6b-4d60-a6cf-6a82dff993a0\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2rnzs" Nov 02 06:20:04.102460 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:04.102289 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntgl4\" (UniqueName: \"kubernetes.io/projected/368fe2a8-af6b-4d60-a6cf-6a82dff993a0-kube-api-access-ntgl4\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2rnzs\" (UID: \"368fe2a8-af6b-4d60-a6cf-6a82dff993a0\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2rnzs" Nov 02 06:20:04.102639 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:04.102619 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/368fe2a8-af6b-4d60-a6cf-6a82dff993a0-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2rnzs\" (UID: \"368fe2a8-af6b-4d60-a6cf-6a82dff993a0\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2rnzs" Nov 02 06:20:04.102705 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:04.102685 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/368fe2a8-af6b-4d60-a6cf-6a82dff993a0-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2rnzs\" (UID: \"368fe2a8-af6b-4d60-a6cf-6a82dff993a0\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2rnzs" Nov 02 06:20:04.151517 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:04.151478 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mfmhs"] Nov 02 06:20:04.156199 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:04.156170 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntgl4\" (UniqueName: \"kubernetes.io/projected/368fe2a8-af6b-4d60-a6cf-6a82dff993a0-kube-api-access-ntgl4\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2rnzs\" (UID: \"368fe2a8-af6b-4d60-a6cf-6a82dff993a0\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2rnzs" Nov 02 06:20:04.158523 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:20:04.158482 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa5d2e0d_5b30_415e_86b0_d3a7e246ffa7.slice/crio-9d00ae0a72fb68e3788d325b7531ef09619ca2d5113967477380bf13c3a933e1 WatchSource:0}: Error finding container 9d00ae0a72fb68e3788d325b7531ef09619ca2d5113967477380bf13c3a933e1: Status 404 returned error can't find the container with id 9d00ae0a72fb68e3788d325b7531ef09619ca2d5113967477380bf13c3a933e1 Nov 02 06:20:04.226879 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:04.226833 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2rnzs" Nov 02 06:20:04.248236 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:04.248206 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chkj82"] Nov 02 06:20:04.254972 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:20:04.254939 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7ef2c80_8b0c_4c7a_a5c1_ddb795124cbb.slice/crio-979d7f5bc6a21f7f6f87e586e2b0e3442532472095900298c444f3bbd87d7f8f WatchSource:0}: Error finding container 979d7f5bc6a21f7f6f87e586e2b0e3442532472095900298c444f3bbd87d7f8f: Status 404 returned error can't find the container with id 979d7f5bc6a21f7f6f87e586e2b0e3442532472095900298c444f3bbd87d7f8f Nov 02 06:20:04.597147 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:04.597007 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2rnzs"] Nov 02 06:20:04.604856 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:20:04.604818 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod368fe2a8_af6b_4d60_a6cf_6a82dff993a0.slice/crio-1f20e247641fae0ede97bd0022b0fe4fe911ed91a0ba265e87e9ac38a45aab29 WatchSource:0}: Error finding container 1f20e247641fae0ede97bd0022b0fe4fe911ed91a0ba265e87e9ac38a45aab29: Status 404 returned error can't find the container with id 1f20e247641fae0ede97bd0022b0fe4fe911ed91a0ba265e87e9ac38a45aab29 Nov 02 06:20:04.614818 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:04.614781 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jjdpk" event={"ID":"70e3b8eb-ed1f-4552-8caf-113f26180ee5","Type":"ContainerStarted","Data":"7cd5682b30d82cf837c97ee33e87c6fdd5a6c75be0a5b7cd695c64d81593e102"} Nov 02 06:20:04.615955 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:04.615930 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mfmhs" event={"ID":"aa5d2e0d-5b30-415e-86b0-d3a7e246ffa7","Type":"ContainerStarted","Data":"9d00ae0a72fb68e3788d325b7531ef09619ca2d5113967477380bf13c3a933e1"} Nov 02 06:20:04.617267 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:04.617243 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agdmqn" event={"ID":"c48449eb-30cb-4b07-892a-b27192f49e0f","Type":"ContainerStarted","Data":"a6cfc876b9b8d8fbfdc462627e6ac2e5b4d41308ede7c1c76a2d43302470e228"} Nov 02 06:20:04.618393 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:04.618371 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chkj82" event={"ID":"a7ef2c80-8b0c-4c7a-a5c1-ddb795124cbb","Type":"ContainerStarted","Data":"979d7f5bc6a21f7f6f87e586e2b0e3442532472095900298c444f3bbd87d7f8f"} Nov 02 06:20:05.628574 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:05.628530 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2rnzs" event={"ID":"368fe2a8-af6b-4d60-a6cf-6a82dff993a0","Type":"ContainerStarted","Data":"1f20e247641fae0ede97bd0022b0fe4fe911ed91a0ba265e87e9ac38a45aab29"} Nov 02 06:20:05.795584 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:05.795543 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bnjjk"] Nov 02 06:20:05.799390 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:05.799363 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bnjjk" Nov 02 06:20:05.894436 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:05.894400 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-wflrh\"" Nov 02 06:20:05.895412 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:05.895386 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bnjjk"] Nov 02 06:20:05.917258 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:05.917224 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjmv7\" (UniqueName: \"kubernetes.io/projected/48b3fc06-ab52-4489-8e5d-84da4514f2da-kube-api-access-pjmv7\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bnjjk\" (UID: \"48b3fc06-ab52-4489-8e5d-84da4514f2da\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bnjjk" Nov 02 06:20:05.917415 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:05.917276 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/48b3fc06-ab52-4489-8e5d-84da4514f2da-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bnjjk\" (UID: \"48b3fc06-ab52-4489-8e5d-84da4514f2da\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bnjjk" Nov 02 06:20:05.917415 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:05.917324 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/48b3fc06-ab52-4489-8e5d-84da4514f2da-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bnjjk\" (UID: \"48b3fc06-ab52-4489-8e5d-84da4514f2da\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bnjjk" Nov 02 06:20:06.018123 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:06.018079 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjmv7\" (UniqueName: \"kubernetes.io/projected/48b3fc06-ab52-4489-8e5d-84da4514f2da-kube-api-access-pjmv7\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bnjjk\" (UID: \"48b3fc06-ab52-4489-8e5d-84da4514f2da\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bnjjk" Nov 02 06:20:06.018322 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:06.018141 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/48b3fc06-ab52-4489-8e5d-84da4514f2da-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bnjjk\" (UID: \"48b3fc06-ab52-4489-8e5d-84da4514f2da\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bnjjk" Nov 02 06:20:06.018322 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:06.018208 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/48b3fc06-ab52-4489-8e5d-84da4514f2da-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bnjjk\" (UID: \"48b3fc06-ab52-4489-8e5d-84da4514f2da\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bnjjk" Nov 02 06:20:06.018662 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:06.018641 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/48b3fc06-ab52-4489-8e5d-84da4514f2da-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bnjjk\" (UID: \"48b3fc06-ab52-4489-8e5d-84da4514f2da\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bnjjk" Nov 02 06:20:06.019280 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:06.019228 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/48b3fc06-ab52-4489-8e5d-84da4514f2da-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bnjjk\" (UID: \"48b3fc06-ab52-4489-8e5d-84da4514f2da\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bnjjk" Nov 02 06:20:06.072421 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:06.072365 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjmv7\" (UniqueName: \"kubernetes.io/projected/48b3fc06-ab52-4489-8e5d-84da4514f2da-kube-api-access-pjmv7\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bnjjk\" (UID: \"48b3fc06-ab52-4489-8e5d-84da4514f2da\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bnjjk" Nov 02 06:20:06.122024 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:06.121983 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bnjjk" Nov 02 06:20:06.518700 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:06.518609 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bnjjk"] Nov 02 06:20:06.536757 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:20:06.536720 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48b3fc06_ab52_4489_8e5d_84da4514f2da.slice/crio-a0dd6f46ac1857b1a6a31fd0cb94f8fa7b1db3d796d50b10a5e8bedf45228060 WatchSource:0}: Error finding container a0dd6f46ac1857b1a6a31fd0cb94f8fa7b1db3d796d50b10a5e8bedf45228060: Status 404 returned error can't find the container with id a0dd6f46ac1857b1a6a31fd0cb94f8fa7b1db3d796d50b10a5e8bedf45228060 Nov 02 06:20:06.636940 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:06.636882 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bnjjk" event={"ID":"48b3fc06-ab52-4489-8e5d-84da4514f2da","Type":"ContainerStarted","Data":"a0dd6f46ac1857b1a6a31fd0cb94f8fa7b1db3d796d50b10a5e8bedf45228060"} Nov 02 06:20:08.275155 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:08.275118 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xr9qf"] Nov 02 06:20:08.278991 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:08.278966 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xr9qf" Nov 02 06:20:08.328587 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:08.328556 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-tt4bx\"" Nov 02 06:20:08.335382 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:08.335140 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xr9qf"] Nov 02 06:20:08.444666 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:08.444629 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9ndg\" (UniqueName: \"kubernetes.io/projected/05e071cb-0da9-4fff-9b10-5ab9b325b1c6-kube-api-access-x9ndg\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xr9qf\" (UID: \"05e071cb-0da9-4fff-9b10-5ab9b325b1c6\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xr9qf" Nov 02 06:20:08.444860 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:08.444679 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/05e071cb-0da9-4fff-9b10-5ab9b325b1c6-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xr9qf\" (UID: \"05e071cb-0da9-4fff-9b10-5ab9b325b1c6\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xr9qf" Nov 02 06:20:08.444860 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:08.444707 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/05e071cb-0da9-4fff-9b10-5ab9b325b1c6-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xr9qf\" (UID: \"05e071cb-0da9-4fff-9b10-5ab9b325b1c6\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xr9qf" Nov 02 06:20:08.546475 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:08.545530 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9ndg\" (UniqueName: \"kubernetes.io/projected/05e071cb-0da9-4fff-9b10-5ab9b325b1c6-kube-api-access-x9ndg\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xr9qf\" (UID: \"05e071cb-0da9-4fff-9b10-5ab9b325b1c6\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xr9qf" Nov 02 06:20:08.546475 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:08.545608 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/05e071cb-0da9-4fff-9b10-5ab9b325b1c6-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xr9qf\" (UID: \"05e071cb-0da9-4fff-9b10-5ab9b325b1c6\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xr9qf" Nov 02 06:20:08.546475 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:08.545648 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/05e071cb-0da9-4fff-9b10-5ab9b325b1c6-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xr9qf\" (UID: \"05e071cb-0da9-4fff-9b10-5ab9b325b1c6\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xr9qf" Nov 02 06:20:08.546475 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:08.546397 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/05e071cb-0da9-4fff-9b10-5ab9b325b1c6-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xr9qf\" (UID: \"05e071cb-0da9-4fff-9b10-5ab9b325b1c6\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xr9qf" Nov 02 06:20:08.546890 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:08.546867 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/05e071cb-0da9-4fff-9b10-5ab9b325b1c6-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xr9qf\" (UID: \"05e071cb-0da9-4fff-9b10-5ab9b325b1c6\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xr9qf" Nov 02 06:20:08.603349 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:08.603290 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9ndg\" (UniqueName: \"kubernetes.io/projected/05e071cb-0da9-4fff-9b10-5ab9b325b1c6-kube-api-access-x9ndg\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xr9qf\" (UID: \"05e071cb-0da9-4fff-9b10-5ab9b325b1c6\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xr9qf" Nov 02 06:20:08.900062 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:08.900023 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xr9qf" Nov 02 06:20:09.323162 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:09.323066 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xr9qf"] Nov 02 06:20:09.330432 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:20:09.330404 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05e071cb_0da9_4fff_9b10_5ab9b325b1c6.slice/crio-ff2c10212c57340bd560778fe35ba7c673129e18ab9dc8c86085fc1e5443735f WatchSource:0}: Error finding container ff2c10212c57340bd560778fe35ba7c673129e18ab9dc8c86085fc1e5443735f: Status 404 returned error can't find the container with id ff2c10212c57340bd560778fe35ba7c673129e18ab9dc8c86085fc1e5443735f Nov 02 06:20:09.669086 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:09.669041 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xr9qf" event={"ID":"05e071cb-0da9-4fff-9b10-5ab9b325b1c6","Type":"ContainerStarted","Data":"ff2c10212c57340bd560778fe35ba7c673129e18ab9dc8c86085fc1e5443735f"} Nov 02 06:20:10.662092 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:10.662056 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crhbtq"] Nov 02 06:20:10.665870 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:10.665849 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crhbtq" Nov 02 06:20:10.690895 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:10.688606 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7626s"] Nov 02 06:20:10.694205 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:10.693714 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7626s" Nov 02 06:20:10.721068 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:10.719347 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76v4x7"] Nov 02 06:20:10.723309 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:10.723282 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76v4x7" Nov 02 06:20:10.735731 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:10.735665 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crhbtq"] Nov 02 06:20:10.765961 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:10.764589 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j2zjn"] Nov 02 06:20:10.768688 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:10.768642 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5a659a95-82ea-47c7-937d-17b4407d1eb2-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crhbtq\" (UID: \"5a659a95-82ea-47c7-937d-17b4407d1eb2\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crhbtq" Nov 02 06:20:10.768896 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:10.768696 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fc77s\" (UniqueName: \"kubernetes.io/projected/5a659a95-82ea-47c7-937d-17b4407d1eb2-kube-api-access-fc77s\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crhbtq\" (UID: \"5a659a95-82ea-47c7-937d-17b4407d1eb2\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crhbtq" Nov 02 06:20:10.768896 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:10.768745 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5a659a95-82ea-47c7-937d-17b4407d1eb2-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crhbtq\" (UID: \"5a659a95-82ea-47c7-937d-17b4407d1eb2\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crhbtq" Nov 02 06:20:10.769198 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:10.769177 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7626s"] Nov 02 06:20:10.769325 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:10.769299 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j2zjn" Nov 02 06:20:10.774584 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:10.774533 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76v4x7"] Nov 02 06:20:10.859485 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:10.859451 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j2zjn"] Nov 02 06:20:10.871007 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:10.869349 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/999d5daf-1bbf-46df-8184-e3454fae166b-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7626s\" (UID: \"999d5daf-1bbf-46df-8184-e3454fae166b\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7626s" Nov 02 06:20:10.871007 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:10.869404 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ng4cf\" (UniqueName: \"kubernetes.io/projected/999d5daf-1bbf-46df-8184-e3454fae166b-kube-api-access-ng4cf\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7626s\" (UID: \"999d5daf-1bbf-46df-8184-e3454fae166b\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7626s" Nov 02 06:20:10.871007 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:10.869447 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d93a0ea2-39a5-4bfa-b4c9-46df030a2dad-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76v4x7\" (UID: \"d93a0ea2-39a5-4bfa-b4c9-46df030a2dad\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76v4x7" Nov 02 06:20:10.871007 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:10.869481 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ef4df80a-7c4f-46fd-9cfe-5ba25a89f869-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j2zjn\" (UID: \"ef4df80a-7c4f-46fd-9cfe-5ba25a89f869\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j2zjn" Nov 02 06:20:10.871007 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:10.869511 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7gtc\" (UniqueName: \"kubernetes.io/projected/ef4df80a-7c4f-46fd-9cfe-5ba25a89f869-kube-api-access-n7gtc\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j2zjn\" (UID: \"ef4df80a-7c4f-46fd-9cfe-5ba25a89f869\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j2zjn" Nov 02 06:20:10.871007 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:10.869554 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5a659a95-82ea-47c7-937d-17b4407d1eb2-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crhbtq\" (UID: \"5a659a95-82ea-47c7-937d-17b4407d1eb2\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crhbtq" Nov 02 06:20:10.871007 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:10.869591 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/999d5daf-1bbf-46df-8184-e3454fae166b-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7626s\" (UID: \"999d5daf-1bbf-46df-8184-e3454fae166b\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7626s" Nov 02 06:20:10.871007 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:10.869621 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fc77s\" (UniqueName: \"kubernetes.io/projected/5a659a95-82ea-47c7-937d-17b4407d1eb2-kube-api-access-fc77s\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crhbtq\" (UID: \"5a659a95-82ea-47c7-937d-17b4407d1eb2\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crhbtq" Nov 02 06:20:10.871007 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:10.869667 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d93a0ea2-39a5-4bfa-b4c9-46df030a2dad-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76v4x7\" (UID: \"d93a0ea2-39a5-4bfa-b4c9-46df030a2dad\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76v4x7" Nov 02 06:20:10.871007 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:10.869706 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5a659a95-82ea-47c7-937d-17b4407d1eb2-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crhbtq\" (UID: \"5a659a95-82ea-47c7-937d-17b4407d1eb2\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crhbtq" Nov 02 06:20:10.871007 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:10.869733 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rrfj\" (UniqueName: \"kubernetes.io/projected/d93a0ea2-39a5-4bfa-b4c9-46df030a2dad-kube-api-access-4rrfj\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76v4x7\" (UID: \"d93a0ea2-39a5-4bfa-b4c9-46df030a2dad\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76v4x7" Nov 02 06:20:10.871007 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:10.869772 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ef4df80a-7c4f-46fd-9cfe-5ba25a89f869-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j2zjn\" (UID: \"ef4df80a-7c4f-46fd-9cfe-5ba25a89f869\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j2zjn" Nov 02 06:20:10.871007 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:10.870309 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5a659a95-82ea-47c7-937d-17b4407d1eb2-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crhbtq\" (UID: \"5a659a95-82ea-47c7-937d-17b4407d1eb2\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crhbtq" Nov 02 06:20:10.871007 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:10.870937 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5a659a95-82ea-47c7-937d-17b4407d1eb2-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crhbtq\" (UID: \"5a659a95-82ea-47c7-937d-17b4407d1eb2\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crhbtq" Nov 02 06:20:10.930105 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:10.929961 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fc77s\" (UniqueName: \"kubernetes.io/projected/5a659a95-82ea-47c7-937d-17b4407d1eb2-kube-api-access-fc77s\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crhbtq\" (UID: \"5a659a95-82ea-47c7-937d-17b4407d1eb2\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crhbtq" Nov 02 06:20:10.973281 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:10.971068 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/999d5daf-1bbf-46df-8184-e3454fae166b-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7626s\" (UID: \"999d5daf-1bbf-46df-8184-e3454fae166b\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7626s" Nov 02 06:20:10.973281 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:10.971142 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d93a0ea2-39a5-4bfa-b4c9-46df030a2dad-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76v4x7\" (UID: \"d93a0ea2-39a5-4bfa-b4c9-46df030a2dad\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76v4x7" Nov 02 06:20:10.973281 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:10.971180 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rrfj\" (UniqueName: \"kubernetes.io/projected/d93a0ea2-39a5-4bfa-b4c9-46df030a2dad-kube-api-access-4rrfj\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76v4x7\" (UID: \"d93a0ea2-39a5-4bfa-b4c9-46df030a2dad\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76v4x7" Nov 02 06:20:10.973281 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:10.971220 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ef4df80a-7c4f-46fd-9cfe-5ba25a89f869-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j2zjn\" (UID: \"ef4df80a-7c4f-46fd-9cfe-5ba25a89f869\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j2zjn" Nov 02 06:20:10.973281 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:10.971272 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/999d5daf-1bbf-46df-8184-e3454fae166b-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7626s\" (UID: \"999d5daf-1bbf-46df-8184-e3454fae166b\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7626s" Nov 02 06:20:10.973281 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:10.971307 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ng4cf\" (UniqueName: \"kubernetes.io/projected/999d5daf-1bbf-46df-8184-e3454fae166b-kube-api-access-ng4cf\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7626s\" (UID: \"999d5daf-1bbf-46df-8184-e3454fae166b\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7626s" Nov 02 06:20:10.973281 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:10.971347 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d93a0ea2-39a5-4bfa-b4c9-46df030a2dad-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76v4x7\" (UID: \"d93a0ea2-39a5-4bfa-b4c9-46df030a2dad\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76v4x7" Nov 02 06:20:10.973281 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:10.971383 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ef4df80a-7c4f-46fd-9cfe-5ba25a89f869-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j2zjn\" (UID: \"ef4df80a-7c4f-46fd-9cfe-5ba25a89f869\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j2zjn" Nov 02 06:20:10.973281 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:10.971414 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7gtc\" (UniqueName: \"kubernetes.io/projected/ef4df80a-7c4f-46fd-9cfe-5ba25a89f869-kube-api-access-n7gtc\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j2zjn\" (UID: \"ef4df80a-7c4f-46fd-9cfe-5ba25a89f869\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j2zjn" Nov 02 06:20:10.973281 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:10.972041 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/999d5daf-1bbf-46df-8184-e3454fae166b-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7626s\" (UID: \"999d5daf-1bbf-46df-8184-e3454fae166b\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7626s" Nov 02 06:20:10.973281 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:10.972328 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d93a0ea2-39a5-4bfa-b4c9-46df030a2dad-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76v4x7\" (UID: \"d93a0ea2-39a5-4bfa-b4c9-46df030a2dad\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76v4x7" Nov 02 06:20:10.973281 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:10.972793 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ef4df80a-7c4f-46fd-9cfe-5ba25a89f869-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j2zjn\" (UID: \"ef4df80a-7c4f-46fd-9cfe-5ba25a89f869\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j2zjn" Nov 02 06:20:10.973281 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:10.973013 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/999d5daf-1bbf-46df-8184-e3454fae166b-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7626s\" (UID: \"999d5daf-1bbf-46df-8184-e3454fae166b\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7626s" Nov 02 06:20:10.973281 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:10.973232 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d93a0ea2-39a5-4bfa-b4c9-46df030a2dad-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76v4x7\" (UID: \"d93a0ea2-39a5-4bfa-b4c9-46df030a2dad\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76v4x7" Nov 02 06:20:10.973281 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:10.973237 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ef4df80a-7c4f-46fd-9cfe-5ba25a89f869-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j2zjn\" (UID: \"ef4df80a-7c4f-46fd-9cfe-5ba25a89f869\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j2zjn" Nov 02 06:20:10.990811 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:10.990566 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crhbtq" Nov 02 06:20:11.028801 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:11.028750 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7gtc\" (UniqueName: \"kubernetes.io/projected/ef4df80a-7c4f-46fd-9cfe-5ba25a89f869-kube-api-access-n7gtc\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j2zjn\" (UID: \"ef4df80a-7c4f-46fd-9cfe-5ba25a89f869\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j2zjn" Nov 02 06:20:11.032284 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:11.032219 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ng4cf\" (UniqueName: \"kubernetes.io/projected/999d5daf-1bbf-46df-8184-e3454fae166b-kube-api-access-ng4cf\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7626s\" (UID: \"999d5daf-1bbf-46df-8184-e3454fae166b\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7626s" Nov 02 06:20:11.032284 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:11.032231 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rrfj\" (UniqueName: \"kubernetes.io/projected/d93a0ea2-39a5-4bfa-b4c9-46df030a2dad-kube-api-access-4rrfj\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76v4x7\" (UID: \"d93a0ea2-39a5-4bfa-b4c9-46df030a2dad\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76v4x7" Nov 02 06:20:11.045049 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:11.044841 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76v4x7" Nov 02 06:20:11.093074 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:11.093036 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j2zjn" Nov 02 06:20:11.316644 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:11.316363 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7626s" Nov 02 06:20:11.392052 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:11.391964 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crhbtq"] Nov 02 06:20:11.397427 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:20:11.397393 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a659a95_82ea_47c7_937d_17b4407d1eb2.slice/crio-e5fdbd631e1414863682656c7089820e6c40bbd9d0abe0b481d4510cccfb6fd5 WatchSource:0}: Error finding container e5fdbd631e1414863682656c7089820e6c40bbd9d0abe0b481d4510cccfb6fd5: Status 404 returned error can't find the container with id e5fdbd631e1414863682656c7089820e6c40bbd9d0abe0b481d4510cccfb6fd5 Nov 02 06:20:11.493100 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:11.493068 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j2zjn"] Nov 02 06:20:11.530805 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:11.530733 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76v4x7"] Nov 02 06:20:11.539574 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:20:11.539530 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd93a0ea2_39a5_4bfa_b4c9_46df030a2dad.slice/crio-be0e42e3fd39b0cebcd20693a6b06e59800872722745f00a64420cc3f78273fe WatchSource:0}: Error finding container be0e42e3fd39b0cebcd20693a6b06e59800872722745f00a64420cc3f78273fe: Status 404 returned error can't find the container with id be0e42e3fd39b0cebcd20693a6b06e59800872722745f00a64420cc3f78273fe Nov 02 06:20:11.693176 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:11.693071 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j2zjn" event={"ID":"ef4df80a-7c4f-46fd-9cfe-5ba25a89f869","Type":"ContainerStarted","Data":"eb3749f02b042230bf6e763207204acf9239fadf6bf651ff4a91a5d80e11932f"} Nov 02 06:20:11.695209 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:11.695167 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76v4x7" event={"ID":"d93a0ea2-39a5-4bfa-b4c9-46df030a2dad","Type":"ContainerStarted","Data":"be0e42e3fd39b0cebcd20693a6b06e59800872722745f00a64420cc3f78273fe"} Nov 02 06:20:11.697477 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:11.697382 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crhbtq" event={"ID":"5a659a95-82ea-47c7-937d-17b4407d1eb2","Type":"ContainerStarted","Data":"e5fdbd631e1414863682656c7089820e6c40bbd9d0abe0b481d4510cccfb6fd5"} Nov 02 06:20:11.756804 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:11.756765 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7626s"] Nov 02 06:20:11.765099 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:20:11.765062 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod999d5daf_1bbf_46df_8184_e3454fae166b.slice/crio-87b8f60610570e25af52a13cd6518213e7afb0f10cbcd7108aab45e1bbe217a5 WatchSource:0}: Error finding container 87b8f60610570e25af52a13cd6518213e7afb0f10cbcd7108aab45e1bbe217a5: Status 404 returned error can't find the container with id 87b8f60610570e25af52a13cd6518213e7afb0f10cbcd7108aab45e1bbe217a5 Nov 02 06:20:12.725946 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:12.725892 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7626s" event={"ID":"999d5daf-1bbf-46df-8184-e3454fae166b","Type":"ContainerStarted","Data":"87b8f60610570e25af52a13cd6518213e7afb0f10cbcd7108aab45e1bbe217a5"} Nov 02 06:20:13.106485 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:13.106398 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-44p69"] Nov 02 06:20:13.111731 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:13.111704 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-44p69" Nov 02 06:20:13.178312 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:13.178277 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-hptbf\"" Nov 02 06:20:13.183987 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:13.183943 5916 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 02 06:20:13.225899 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:13.225860 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-44p69"] Nov 02 06:20:13.313937 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:13.305288 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fccb2c65-8f82-4808-bb0f-7d9333d8932a-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-44p69\" (UID: \"fccb2c65-8f82-4808-bb0f-7d9333d8932a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-44p69" Nov 02 06:20:13.313937 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:13.305388 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fccb2c65-8f82-4808-bb0f-7d9333d8932a-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-44p69\" (UID: \"fccb2c65-8f82-4808-bb0f-7d9333d8932a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-44p69" Nov 02 06:20:13.313937 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:13.305453 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fccb2c65-8f82-4808-bb0f-7d9333d8932a-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-44p69\" (UID: \"fccb2c65-8f82-4808-bb0f-7d9333d8932a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-44p69" Nov 02 06:20:13.313937 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:13.305477 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fccb2c65-8f82-4808-bb0f-7d9333d8932a-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-44p69\" (UID: \"fccb2c65-8f82-4808-bb0f-7d9333d8932a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-44p69" Nov 02 06:20:13.313937 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:13.305505 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fccb2c65-8f82-4808-bb0f-7d9333d8932a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-44p69\" (UID: \"fccb2c65-8f82-4808-bb0f-7d9333d8932a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-44p69" Nov 02 06:20:13.313937 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:13.305545 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fccb2c65-8f82-4808-bb0f-7d9333d8932a-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-44p69\" (UID: \"fccb2c65-8f82-4808-bb0f-7d9333d8932a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-44p69" Nov 02 06:20:13.313937 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:13.305592 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mhph\" (UniqueName: \"kubernetes.io/projected/fccb2c65-8f82-4808-bb0f-7d9333d8932a-kube-api-access-7mhph\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-44p69\" (UID: \"fccb2c65-8f82-4808-bb0f-7d9333d8932a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-44p69" Nov 02 06:20:13.313937 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:13.305624 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fccb2c65-8f82-4808-bb0f-7d9333d8932a-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-44p69\" (UID: \"fccb2c65-8f82-4808-bb0f-7d9333d8932a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-44p69" Nov 02 06:20:13.313937 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:13.305647 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fccb2c65-8f82-4808-bb0f-7d9333d8932a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-44p69\" (UID: \"fccb2c65-8f82-4808-bb0f-7d9333d8932a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-44p69" Nov 02 06:20:13.396091 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:13.396060 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-dh56v"] Nov 02 06:20:13.401204 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:13.401177 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-dh56v" Nov 02 06:20:13.406159 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:13.406136 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fccb2c65-8f82-4808-bb0f-7d9333d8932a-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-44p69\" (UID: \"fccb2c65-8f82-4808-bb0f-7d9333d8932a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-44p69" Nov 02 06:20:13.406302 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:13.406169 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fccb2c65-8f82-4808-bb0f-7d9333d8932a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-44p69\" (UID: \"fccb2c65-8f82-4808-bb0f-7d9333d8932a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-44p69" Nov 02 06:20:13.406302 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:13.406229 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fccb2c65-8f82-4808-bb0f-7d9333d8932a-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-44p69\" (UID: \"fccb2c65-8f82-4808-bb0f-7d9333d8932a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-44p69" Nov 02 06:20:13.406302 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:13.406262 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fccb2c65-8f82-4808-bb0f-7d9333d8932a-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-44p69\" (UID: \"fccb2c65-8f82-4808-bb0f-7d9333d8932a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-44p69" Nov 02 06:20:13.406570 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:13.406318 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fccb2c65-8f82-4808-bb0f-7d9333d8932a-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-44p69\" (UID: \"fccb2c65-8f82-4808-bb0f-7d9333d8932a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-44p69" Nov 02 06:20:13.406570 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:13.406345 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fccb2c65-8f82-4808-bb0f-7d9333d8932a-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-44p69\" (UID: \"fccb2c65-8f82-4808-bb0f-7d9333d8932a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-44p69" Nov 02 06:20:13.406570 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:13.406371 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fccb2c65-8f82-4808-bb0f-7d9333d8932a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-44p69\" (UID: \"fccb2c65-8f82-4808-bb0f-7d9333d8932a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-44p69" Nov 02 06:20:13.406570 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:13.406461 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fccb2c65-8f82-4808-bb0f-7d9333d8932a-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-44p69\" (UID: \"fccb2c65-8f82-4808-bb0f-7d9333d8932a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-44p69" Nov 02 06:20:13.406570 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:13.406529 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mhph\" (UniqueName: \"kubernetes.io/projected/fccb2c65-8f82-4808-bb0f-7d9333d8932a-kube-api-access-7mhph\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-44p69\" (UID: \"fccb2c65-8f82-4808-bb0f-7d9333d8932a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-44p69" Nov 02 06:20:13.409179 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:13.408039 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fccb2c65-8f82-4808-bb0f-7d9333d8932a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-44p69\" (UID: \"fccb2c65-8f82-4808-bb0f-7d9333d8932a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-44p69" Nov 02 06:20:13.409179 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:13.408325 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fccb2c65-8f82-4808-bb0f-7d9333d8932a-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-44p69\" (UID: \"fccb2c65-8f82-4808-bb0f-7d9333d8932a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-44p69" Nov 02 06:20:13.409179 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:13.408584 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fccb2c65-8f82-4808-bb0f-7d9333d8932a-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-44p69\" (UID: \"fccb2c65-8f82-4808-bb0f-7d9333d8932a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-44p69" Nov 02 06:20:13.409179 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:13.408827 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fccb2c65-8f82-4808-bb0f-7d9333d8932a-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-44p69\" (UID: \"fccb2c65-8f82-4808-bb0f-7d9333d8932a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-44p69" Nov 02 06:20:13.410521 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:13.409802 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fccb2c65-8f82-4808-bb0f-7d9333d8932a-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-44p69\" (UID: \"fccb2c65-8f82-4808-bb0f-7d9333d8932a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-44p69" Nov 02 06:20:13.420812 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:13.420781 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fccb2c65-8f82-4808-bb0f-7d9333d8932a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-44p69\" (UID: \"fccb2c65-8f82-4808-bb0f-7d9333d8932a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-44p69" Nov 02 06:20:13.427344 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:13.427307 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fccb2c65-8f82-4808-bb0f-7d9333d8932a-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-44p69\" (UID: \"fccb2c65-8f82-4808-bb0f-7d9333d8932a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-44p69" Nov 02 06:20:13.450208 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:13.450162 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-g78tr\"" Nov 02 06:20:13.458558 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:13.458420 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-dh56v"] Nov 02 06:20:13.469242 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:13.469205 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fccb2c65-8f82-4808-bb0f-7d9333d8932a-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-44p69\" (UID: \"fccb2c65-8f82-4808-bb0f-7d9333d8932a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-44p69" Nov 02 06:20:13.472512 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:13.472485 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mhph\" (UniqueName: \"kubernetes.io/projected/fccb2c65-8f82-4808-bb0f-7d9333d8932a-kube-api-access-7mhph\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-44p69\" (UID: \"fccb2c65-8f82-4808-bb0f-7d9333d8932a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-44p69" Nov 02 06:20:13.508123 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:13.508089 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e12f5580-a2a6-4e2a-a549-af9693a63b86-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-dh56v\" (UID: \"e12f5580-a2a6-4e2a-a549-af9693a63b86\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-dh56v" Nov 02 06:20:13.508297 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:13.508141 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e12f5580-a2a6-4e2a-a549-af9693a63b86-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-dh56v\" (UID: \"e12f5580-a2a6-4e2a-a549-af9693a63b86\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-dh56v" Nov 02 06:20:13.508297 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:13.508187 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e12f5580-a2a6-4e2a-a549-af9693a63b86-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-dh56v\" (UID: \"e12f5580-a2a6-4e2a-a549-af9693a63b86\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-dh56v" Nov 02 06:20:13.508297 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:13.508229 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e12f5580-a2a6-4e2a-a549-af9693a63b86-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-dh56v\" (UID: \"e12f5580-a2a6-4e2a-a549-af9693a63b86\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-dh56v" Nov 02 06:20:13.508297 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:13.508263 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e12f5580-a2a6-4e2a-a549-af9693a63b86-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-dh56v\" (UID: \"e12f5580-a2a6-4e2a-a549-af9693a63b86\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-dh56v" Nov 02 06:20:13.508297 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:13.508294 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsqq5\" (UniqueName: \"kubernetes.io/projected/e12f5580-a2a6-4e2a-a549-af9693a63b86-kube-api-access-fsqq5\") pod \"openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-dh56v\" (UID: \"e12f5580-a2a6-4e2a-a549-af9693a63b86\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-dh56v" Nov 02 06:20:13.508561 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:13.508320 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e12f5580-a2a6-4e2a-a549-af9693a63b86-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-dh56v\" (UID: \"e12f5580-a2a6-4e2a-a549-af9693a63b86\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-dh56v" Nov 02 06:20:13.508561 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:13.508340 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e12f5580-a2a6-4e2a-a549-af9693a63b86-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-dh56v\" (UID: \"e12f5580-a2a6-4e2a-a549-af9693a63b86\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-dh56v" Nov 02 06:20:13.508561 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:13.508359 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e12f5580-a2a6-4e2a-a549-af9693a63b86-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-dh56v\" (UID: \"e12f5580-a2a6-4e2a-a549-af9693a63b86\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-dh56v" Nov 02 06:20:13.613172 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:13.609530 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e12f5580-a2a6-4e2a-a549-af9693a63b86-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-dh56v\" (UID: \"e12f5580-a2a6-4e2a-a549-af9693a63b86\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-dh56v" Nov 02 06:20:13.613172 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:13.609590 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e12f5580-a2a6-4e2a-a549-af9693a63b86-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-dh56v\" (UID: \"e12f5580-a2a6-4e2a-a549-af9693a63b86\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-dh56v" Nov 02 06:20:13.613172 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:13.609671 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e12f5580-a2a6-4e2a-a549-af9693a63b86-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-dh56v\" (UID: \"e12f5580-a2a6-4e2a-a549-af9693a63b86\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-dh56v" Nov 02 06:20:13.613172 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:13.609714 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e12f5580-a2a6-4e2a-a549-af9693a63b86-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-dh56v\" (UID: \"e12f5580-a2a6-4e2a-a549-af9693a63b86\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-dh56v" Nov 02 06:20:13.613172 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:13.609747 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e12f5580-a2a6-4e2a-a549-af9693a63b86-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-dh56v\" (UID: \"e12f5580-a2a6-4e2a-a549-af9693a63b86\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-dh56v" Nov 02 06:20:13.613172 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:13.609774 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsqq5\" (UniqueName: \"kubernetes.io/projected/e12f5580-a2a6-4e2a-a549-af9693a63b86-kube-api-access-fsqq5\") pod \"openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-dh56v\" (UID: \"e12f5580-a2a6-4e2a-a549-af9693a63b86\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-dh56v" Nov 02 06:20:13.613172 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:13.609818 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e12f5580-a2a6-4e2a-a549-af9693a63b86-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-dh56v\" (UID: \"e12f5580-a2a6-4e2a-a549-af9693a63b86\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-dh56v" Nov 02 06:20:13.613172 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:13.609852 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e12f5580-a2a6-4e2a-a549-af9693a63b86-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-dh56v\" (UID: \"e12f5580-a2a6-4e2a-a549-af9693a63b86\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-dh56v" Nov 02 06:20:13.613172 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:13.609887 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e12f5580-a2a6-4e2a-a549-af9693a63b86-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-dh56v\" (UID: \"e12f5580-a2a6-4e2a-a549-af9693a63b86\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-dh56v" Nov 02 06:20:13.613172 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:13.610415 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e12f5580-a2a6-4e2a-a549-af9693a63b86-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-dh56v\" (UID: \"e12f5580-a2a6-4e2a-a549-af9693a63b86\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-dh56v" Nov 02 06:20:13.613172 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:13.611092 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e12f5580-a2a6-4e2a-a549-af9693a63b86-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-dh56v\" (UID: \"e12f5580-a2a6-4e2a-a549-af9693a63b86\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-dh56v" Nov 02 06:20:13.613172 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:13.611317 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e12f5580-a2a6-4e2a-a549-af9693a63b86-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-dh56v\" (UID: \"e12f5580-a2a6-4e2a-a549-af9693a63b86\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-dh56v" Nov 02 06:20:13.613172 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:13.612308 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e12f5580-a2a6-4e2a-a549-af9693a63b86-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-dh56v\" (UID: \"e12f5580-a2a6-4e2a-a549-af9693a63b86\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-dh56v" Nov 02 06:20:13.613172 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:13.612759 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e12f5580-a2a6-4e2a-a549-af9693a63b86-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-dh56v\" (UID: \"e12f5580-a2a6-4e2a-a549-af9693a63b86\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-dh56v" Nov 02 06:20:13.618530 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:13.618475 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e12f5580-a2a6-4e2a-a549-af9693a63b86-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-dh56v\" (UID: \"e12f5580-a2a6-4e2a-a549-af9693a63b86\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-dh56v" Nov 02 06:20:13.623082 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:13.623054 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e12f5580-a2a6-4e2a-a549-af9693a63b86-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-dh56v\" (UID: \"e12f5580-a2a6-4e2a-a549-af9693a63b86\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-dh56v" Nov 02 06:20:13.679151 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:13.679043 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsqq5\" (UniqueName: \"kubernetes.io/projected/e12f5580-a2a6-4e2a-a549-af9693a63b86-kube-api-access-fsqq5\") pod \"openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-dh56v\" (UID: \"e12f5580-a2a6-4e2a-a549-af9693a63b86\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-dh56v" Nov 02 06:20:13.681135 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:13.681107 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e12f5580-a2a6-4e2a-a549-af9693a63b86-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-dh56v\" (UID: \"e12f5580-a2a6-4e2a-a549-af9693a63b86\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-dh56v" Nov 02 06:20:13.729443 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:13.729414 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-dh56v" Nov 02 06:20:13.729958 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:13.729780 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-44p69" Nov 02 06:20:14.266382 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:14.266336 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-dh56v"] Nov 02 06:20:14.269182 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:14.269029 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-44p69"] Nov 02 06:20:14.316691 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:20:14.316653 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfccb2c65_8f82_4808_bb0f_7d9333d8932a.slice/crio-d9677834eac5820f26e5c4eb45c0704a8c06232ea5fdbaf6d3da206207cb5169 WatchSource:0}: Error finding container d9677834eac5820f26e5c4eb45c0704a8c06232ea5fdbaf6d3da206207cb5169: Status 404 returned error can't find the container with id d9677834eac5820f26e5c4eb45c0704a8c06232ea5fdbaf6d3da206207cb5169 Nov 02 06:20:14.760856 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:14.760728 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-44p69" event={"ID":"fccb2c65-8f82-4808-bb0f-7d9333d8932a","Type":"ContainerStarted","Data":"d9677834eac5820f26e5c4eb45c0704a8c06232ea5fdbaf6d3da206207cb5169"} Nov 02 06:20:14.766219 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:14.766116 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-dh56v" event={"ID":"e12f5580-a2a6-4e2a-a549-af9693a63b86","Type":"ContainerStarted","Data":"af01239eed1171a9a6c5cb6f0f4ba513489f224028621227f07c1dbd93dc451c"} Nov 02 06:20:19.922919 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:20:19.922802 5916 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p98cq" containerName="extractor" Nov 02 06:20:25.935264 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:25.935226 5916 generic.go:356] "Generic (PLEG): container finished" podID="48b3fc06-ab52-4489-8e5d-84da4514f2da" containerID="2114c498cb4c056b688b3b6d0dbfe2027d1a7130aae6e744aeb33075161418af" exitCode=0 Nov 02 06:20:25.935789 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:25.935353 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bnjjk" event={"ID":"48b3fc06-ab52-4489-8e5d-84da4514f2da","Type":"ContainerDied","Data":"2114c498cb4c056b688b3b6d0dbfe2027d1a7130aae6e744aeb33075161418af"} Nov 02 06:20:25.937740 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:25.937395 5916 generic.go:356] "Generic (PLEG): container finished" podID="368fe2a8-af6b-4d60-a6cf-6a82dff993a0" containerID="72038a85c92d988e37bf41684af7317195a4758295d2f5f24ee27e38153e22a2" exitCode=0 Nov 02 06:20:25.937740 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:25.937493 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2rnzs" event={"ID":"368fe2a8-af6b-4d60-a6cf-6a82dff993a0","Type":"ContainerDied","Data":"72038a85c92d988e37bf41684af7317195a4758295d2f5f24ee27e38153e22a2"} Nov 02 06:20:26.942953 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:26.942921 5916 generic.go:356] "Generic (PLEG): container finished" podID="d93a0ea2-39a5-4bfa-b4c9-46df030a2dad" containerID="b7aa9bc3eb0202b7793f3f1441e310dead5fe85ce7208bc84284e8d39b3b1645" exitCode=0 Nov 02 06:20:26.943421 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:26.943015 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76v4x7" event={"ID":"d93a0ea2-39a5-4bfa-b4c9-46df030a2dad","Type":"ContainerDied","Data":"b7aa9bc3eb0202b7793f3f1441e310dead5fe85ce7208bc84284e8d39b3b1645"} Nov 02 06:20:26.944564 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:26.944543 5916 generic.go:356] "Generic (PLEG): container finished" podID="70e3b8eb-ed1f-4552-8caf-113f26180ee5" containerID="c3be80154a3fd89db528ad6e069a1f26bff989c0e41f236a7a8938d19dbc18b0" exitCode=0 Nov 02 06:20:26.944672 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:26.944606 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jjdpk" event={"ID":"70e3b8eb-ed1f-4552-8caf-113f26180ee5","Type":"ContainerDied","Data":"c3be80154a3fd89db528ad6e069a1f26bff989c0e41f236a7a8938d19dbc18b0"} Nov 02 06:20:27.245142 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:27.245108 5916 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 06:20:27.245251 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:27.245190 5916 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 06:20:27.245292 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:27.245255 5916 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 06:20:27.950438 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:27.950407 5916 generic.go:356] "Generic (PLEG): container finished" podID="05e071cb-0da9-4fff-9b10-5ab9b325b1c6" containerID="e22c6b4cceeb191551e13ca95c7c34622bc40d0f5e1fdc0a88f96f5464becd0f" exitCode=0 Nov 02 06:20:27.950883 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:27.950483 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xr9qf" event={"ID":"05e071cb-0da9-4fff-9b10-5ab9b325b1c6","Type":"ContainerDied","Data":"e22c6b4cceeb191551e13ca95c7c34622bc40d0f5e1fdc0a88f96f5464becd0f"} Nov 02 06:20:27.952059 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:27.952040 5916 generic.go:356] "Generic (PLEG): container finished" podID="999d5daf-1bbf-46df-8184-e3454fae166b" containerID="1ff7e8cb6b0ef735405f3120367ccafe62986ea2aa69bcf952b058d373cc62ee" exitCode=0 Nov 02 06:20:27.952149 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:27.952106 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7626s" event={"ID":"999d5daf-1bbf-46df-8184-e3454fae166b","Type":"ContainerDied","Data":"1ff7e8cb6b0ef735405f3120367ccafe62986ea2aa69bcf952b058d373cc62ee"} Nov 02 06:20:27.953601 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:27.953584 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-dh56v" event={"ID":"e12f5580-a2a6-4e2a-a549-af9693a63b86","Type":"ContainerStarted","Data":"1d2d79ca57dc9e1e8c65fb24b2b716c948377dced4c64e0c71e8e9a0127866ba"} Nov 02 06:20:27.955033 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:27.955012 5916 generic.go:356] "Generic (PLEG): container finished" podID="c48449eb-30cb-4b07-892a-b27192f49e0f" containerID="94e004e38a1df830f6ff503fef6dc1d4e7c9edd57e231859bf99de894651694c" exitCode=0 Nov 02 06:20:27.955113 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:27.955089 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agdmqn" event={"ID":"c48449eb-30cb-4b07-892a-b27192f49e0f","Type":"ContainerDied","Data":"94e004e38a1df830f6ff503fef6dc1d4e7c9edd57e231859bf99de894651694c"} Nov 02 06:20:28.095861 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:28.095832 5916 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 06:20:28.095979 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:28.095894 5916 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 06:20:28.095979 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:28.095941 5916 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 06:20:28.173652 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:28.173566 5916 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-dh56v" podStartSLOduration=2.211969875 podStartE2EDuration="15.173546787s" totalImagesPullingTime="12.961576912s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:20:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:20:28.173002719 +0000 UTC m=+216.444823114" watchObservedRunningTime="2026-11-02 06:20:28.173546787 +0000 UTC m=+216.445367180" Nov 02 06:20:28.730051 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:28.730019 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-dh56v" Nov 02 06:20:28.730201 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:28.730073 5916 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-dh56v" Nov 02 06:20:28.735061 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:28.735035 5916 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-dh56v" Nov 02 06:20:28.961220 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:28.961191 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-44p69" event={"ID":"fccb2c65-8f82-4808-bb0f-7d9333d8932a","Type":"ContainerStarted","Data":"0433bc04f4a45475241fe4622439501b58a25a1db75351c76111fbbbbe571ef6"} Nov 02 06:20:28.963075 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:28.963047 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bnjjk" event={"ID":"48b3fc06-ab52-4489-8e5d-84da4514f2da","Type":"ContainerStarted","Data":"f695eb4d5a8f75e5d75d05a53835fadcdc5ad0c8c26d9476d742230e2a7ef5f7"} Nov 02 06:20:28.964725 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:28.964689 5916 generic.go:356] "Generic (PLEG): container finished" podID="aa5d2e0d-5b30-415e-86b0-d3a7e246ffa7" containerID="3f71a08da85a2fbcab4d4d216d4613cdf00fff80510bc092a0f50226dd6b6630" exitCode=0 Nov 02 06:20:28.964826 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:28.964767 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mfmhs" event={"ID":"aa5d2e0d-5b30-415e-86b0-d3a7e246ffa7","Type":"ContainerDied","Data":"3f71a08da85a2fbcab4d4d216d4613cdf00fff80510bc092a0f50226dd6b6630"} Nov 02 06:20:28.966359 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:28.966338 5916 generic.go:356] "Generic (PLEG): container finished" podID="ef4df80a-7c4f-46fd-9cfe-5ba25a89f869" containerID="1506dbb4d9b934ccaf8ccf69aa19ddec797f06c3b0c1b50ad08ae939ef8bcbac" exitCode=0 Nov 02 06:20:28.966460 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:28.966419 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j2zjn" event={"ID":"ef4df80a-7c4f-46fd-9cfe-5ba25a89f869","Type":"ContainerDied","Data":"1506dbb4d9b934ccaf8ccf69aa19ddec797f06c3b0c1b50ad08ae939ef8bcbac"} Nov 02 06:20:28.968180 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:28.968116 5916 generic.go:356] "Generic (PLEG): container finished" podID="a7ef2c80-8b0c-4c7a-a5c1-ddb795124cbb" containerID="4161770366c4192c4b45a7221cf55546e04c77220c64a04f3468f2e71ba0cdc5" exitCode=0 Nov 02 06:20:28.968313 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:28.968294 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chkj82" event={"ID":"a7ef2c80-8b0c-4c7a-a5c1-ddb795124cbb","Type":"ContainerDied","Data":"4161770366c4192c4b45a7221cf55546e04c77220c64a04f3468f2e71ba0cdc5"} Nov 02 06:20:28.970220 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:28.970197 5916 generic.go:356] "Generic (PLEG): container finished" podID="5a659a95-82ea-47c7-937d-17b4407d1eb2" containerID="ea4147fd7a849bcc2b779d99aa1049d08e0d5ed2dcaa2bd46b6d07b6ad8d8a39" exitCode=0 Nov 02 06:20:28.970324 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:28.970286 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crhbtq" event={"ID":"5a659a95-82ea-47c7-937d-17b4407d1eb2","Type":"ContainerDied","Data":"ea4147fd7a849bcc2b779d99aa1049d08e0d5ed2dcaa2bd46b6d07b6ad8d8a39"} Nov 02 06:20:28.972108 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:28.972087 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-dh56v" Nov 02 06:20:29.072517 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:29.072471 5916 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-44p69" podStartSLOduration=2.299046265 podStartE2EDuration="16.072456817s" totalImagesPullingTime="13.773410552s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:20:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:20:29.07231483 +0000 UTC m=+217.344135225" watchObservedRunningTime="2026-11-02 06:20:29.072456817 +0000 UTC m=+217.344277202" Nov 02 06:20:29.730020 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:29.729982 5916 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-44p69" Nov 02 06:20:29.735383 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:29.735353 5916 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-44p69" Nov 02 06:20:29.975788 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:29.975758 5916 generic.go:356] "Generic (PLEG): container finished" podID="48b3fc06-ab52-4489-8e5d-84da4514f2da" containerID="f695eb4d5a8f75e5d75d05a53835fadcdc5ad0c8c26d9476d742230e2a7ef5f7" exitCode=0 Nov 02 06:20:29.976173 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:29.975816 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bnjjk" event={"ID":"48b3fc06-ab52-4489-8e5d-84da4514f2da","Type":"ContainerDied","Data":"f695eb4d5a8f75e5d75d05a53835fadcdc5ad0c8c26d9476d742230e2a7ef5f7"} Nov 02 06:20:29.977783 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:29.977760 5916 generic.go:356] "Generic (PLEG): container finished" podID="d93a0ea2-39a5-4bfa-b4c9-46df030a2dad" containerID="37c56561335e4990c48fd81283d50b7c42583130b358e0364c83e78e33f8eb4d" exitCode=0 Nov 02 06:20:29.977892 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:29.977844 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76v4x7" event={"ID":"d93a0ea2-39a5-4bfa-b4c9-46df030a2dad","Type":"ContainerDied","Data":"37c56561335e4990c48fd81283d50b7c42583130b358e0364c83e78e33f8eb4d"} Nov 02 06:20:29.979815 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:29.979795 5916 generic.go:356] "Generic (PLEG): container finished" podID="70e3b8eb-ed1f-4552-8caf-113f26180ee5" containerID="d2097d48e4cff9f5739343601e6d20b9cf34061a364e6144fce1bed5fd4ed1ea" exitCode=0 Nov 02 06:20:29.979934 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:29.979882 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jjdpk" event={"ID":"70e3b8eb-ed1f-4552-8caf-113f26180ee5","Type":"ContainerDied","Data":"d2097d48e4cff9f5739343601e6d20b9cf34061a364e6144fce1bed5fd4ed1ea"} Nov 02 06:20:29.980125 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:29.980091 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-44p69" Nov 02 06:20:29.981118 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:29.981103 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-44p69" Nov 02 06:20:30.988209 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:30.988174 5916 generic.go:356] "Generic (PLEG): container finished" podID="aa5d2e0d-5b30-415e-86b0-d3a7e246ffa7" containerID="6fb81b9b55327ed9925d6f5f95539bd62552e5e1d3df2893a64c8436e1f19990" exitCode=0 Nov 02 06:20:30.988652 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:30.988240 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mfmhs" event={"ID":"aa5d2e0d-5b30-415e-86b0-d3a7e246ffa7","Type":"ContainerDied","Data":"6fb81b9b55327ed9925d6f5f95539bd62552e5e1d3df2893a64c8436e1f19990"} Nov 02 06:20:30.991424 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:30.991401 5916 generic.go:356] "Generic (PLEG): container finished" podID="ef4df80a-7c4f-46fd-9cfe-5ba25a89f869" containerID="10c5a20de6bd27a80ce401a5c7c2d8ce48742e3ffeb776059646b5f2c94adecf" exitCode=0 Nov 02 06:20:30.991538 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:30.991485 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j2zjn" event={"ID":"ef4df80a-7c4f-46fd-9cfe-5ba25a89f869","Type":"ContainerDied","Data":"10c5a20de6bd27a80ce401a5c7c2d8ce48742e3ffeb776059646b5f2c94adecf"} Nov 02 06:20:30.993892 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:30.993753 5916 generic.go:356] "Generic (PLEG): container finished" podID="999d5daf-1bbf-46df-8184-e3454fae166b" containerID="24eea5c1b67c0fbf164551f5474a1e63db131efd39c93b98186c6f06be4a4052" exitCode=0 Nov 02 06:20:30.994088 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:30.993925 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7626s" event={"ID":"999d5daf-1bbf-46df-8184-e3454fae166b","Type":"ContainerDied","Data":"24eea5c1b67c0fbf164551f5474a1e63db131efd39c93b98186c6f06be4a4052"} Nov 02 06:20:30.996275 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:30.996247 5916 generic.go:356] "Generic (PLEG): container finished" podID="c48449eb-30cb-4b07-892a-b27192f49e0f" containerID="83ecbbf7dfea2ffffb22b5105c3149f2481707a872a9a9ccfc079b00cc750cf2" exitCode=0 Nov 02 06:20:30.997264 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:30.997239 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agdmqn" event={"ID":"c48449eb-30cb-4b07-892a-b27192f49e0f","Type":"ContainerDied","Data":"83ecbbf7dfea2ffffb22b5105c3149f2481707a872a9a9ccfc079b00cc750cf2"} Nov 02 06:20:32.008565 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:32.008519 5916 generic.go:356] "Generic (PLEG): container finished" podID="a7ef2c80-8b0c-4c7a-a5c1-ddb795124cbb" containerID="08538046f60178c50c099bbc96229f72aabc085f63d59a2e29f926ee12d80a25" exitCode=0 Nov 02 06:20:32.009098 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:32.008598 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chkj82" event={"ID":"a7ef2c80-8b0c-4c7a-a5c1-ddb795124cbb","Type":"ContainerDied","Data":"08538046f60178c50c099bbc96229f72aabc085f63d59a2e29f926ee12d80a25"} Nov 02 06:20:32.013366 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:32.013320 5916 generic.go:356] "Generic (PLEG): container finished" podID="5a659a95-82ea-47c7-937d-17b4407d1eb2" containerID="6f9e8efba52ff545b144cd9ca7d2e25fa49e0f49d3fcaf2d3c6380bb3f53f247" exitCode=0 Nov 02 06:20:32.013995 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:32.013967 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crhbtq" event={"ID":"5a659a95-82ea-47c7-937d-17b4407d1eb2","Type":"ContainerDied","Data":"6f9e8efba52ff545b144cd9ca7d2e25fa49e0f49d3fcaf2d3c6380bb3f53f247"} Nov 02 06:20:33.029697 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:33.029669 5916 generic.go:356] "Generic (PLEG): container finished" podID="05e071cb-0da9-4fff-9b10-5ab9b325b1c6" containerID="b6f1bbd2927aa893abc75ba8ff94885c095d904b089a2afc2a1b8897e6adb820" exitCode=0 Nov 02 06:20:33.030107 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:33.029813 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xr9qf" event={"ID":"05e071cb-0da9-4fff-9b10-5ab9b325b1c6","Type":"ContainerDied","Data":"b6f1bbd2927aa893abc75ba8ff94885c095d904b089a2afc2a1b8897e6adb820"} Nov 02 06:20:34.059172 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:34.059118 5916 generic.go:356] "Generic (PLEG): container finished" podID="368fe2a8-af6b-4d60-a6cf-6a82dff993a0" containerID="d6d053c4657c1a8d05c0b53795a73c9066b0c852a423ede7f7cea05827db22ab" exitCode=0 Nov 02 06:20:34.059728 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:34.059300 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2rnzs" event={"ID":"368fe2a8-af6b-4d60-a6cf-6a82dff993a0","Type":"ContainerDied","Data":"d6d053c4657c1a8d05c0b53795a73c9066b0c852a423ede7f7cea05827db22ab"} Nov 02 06:20:49.915225 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:20:49.915176 5916 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p98cq" containerName="extractor" Nov 02 06:20:52.329676 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:52.329587 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jjdpk" event={"ID":"70e3b8eb-ed1f-4552-8caf-113f26180ee5","Type":"ContainerStarted","Data":"837b6034453719887d516c7672887d5d6bcf3237a6383db13e5efa39451cec4d"} Nov 02 06:20:53.350469 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:53.350428 5916 generic.go:356] "Generic (PLEG): container finished" podID="48b3fc06-ab52-4489-8e5d-84da4514f2da" containerID="0b4c12e2aea3ab8ba62ec63de463c5f0724dc1d0cf4560c87bc665e79e788208" exitCode=0 Nov 02 06:20:53.350986 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:53.350566 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bnjjk" event={"ID":"48b3fc06-ab52-4489-8e5d-84da4514f2da","Type":"ContainerDied","Data":"0b4c12e2aea3ab8ba62ec63de463c5f0724dc1d0cf4560c87bc665e79e788208"} Nov 02 06:20:53.356516 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:53.356445 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j2zjn" event={"ID":"ef4df80a-7c4f-46fd-9cfe-5ba25a89f869","Type":"ContainerStarted","Data":"f2a65a425b22131777ed9a4ccaaca034d72fa82b1d4917c6379929f21d269b2b"} Nov 02 06:20:53.364518 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:53.364487 5916 generic.go:356] "Generic (PLEG): container finished" podID="d93a0ea2-39a5-4bfa-b4c9-46df030a2dad" containerID="19f913d0defcb6e93301b07e79aea5606a26a8cef2144590533a8a478e3c23ac" exitCode=0 Nov 02 06:20:53.364677 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:53.364617 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76v4x7" event={"ID":"d93a0ea2-39a5-4bfa-b4c9-46df030a2dad","Type":"ContainerDied","Data":"19f913d0defcb6e93301b07e79aea5606a26a8cef2144590533a8a478e3c23ac"} Nov 02 06:20:53.368759 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:53.368731 5916 generic.go:356] "Generic (PLEG): container finished" podID="70e3b8eb-ed1f-4552-8caf-113f26180ee5" containerID="837b6034453719887d516c7672887d5d6bcf3237a6383db13e5efa39451cec4d" exitCode=0 Nov 02 06:20:53.368890 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:53.368779 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jjdpk" event={"ID":"70e3b8eb-ed1f-4552-8caf-113f26180ee5","Type":"ContainerDied","Data":"837b6034453719887d516c7672887d5d6bcf3237a6383db13e5efa39451cec4d"} Nov 02 06:20:53.460100 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:53.460036 5916 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jjdpk" podStartSLOduration=3.8228536010000003 podStartE2EDuration="50.459608391s" totalImagesPullingTime="46.63675479s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:20:03 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:20:52.440200698 +0000 UTC m=+240.712021090" watchObservedRunningTime="2026-11-02 06:20:53.459608391 +0000 UTC m=+241.731428783" Nov 02 06:20:53.788701 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:53.788640 5916 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j2zjn" podStartSLOduration=2.661473803 podStartE2EDuration="43.788619832s" totalImagesPullingTime="41.127146029s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:20:10 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:20:53.786993596 +0000 UTC m=+242.058813991" watchObservedRunningTime="2026-11-02 06:20:53.788619832 +0000 UTC m=+242.060440207" Nov 02 06:20:54.381582 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:54.380899 5916 generic.go:356] "Generic (PLEG): container finished" podID="ef4df80a-7c4f-46fd-9cfe-5ba25a89f869" containerID="f2a65a425b22131777ed9a4ccaaca034d72fa82b1d4917c6379929f21d269b2b" exitCode=0 Nov 02 06:20:54.381582 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:54.381030 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j2zjn" event={"ID":"ef4df80a-7c4f-46fd-9cfe-5ba25a89f869","Type":"ContainerDied","Data":"f2a65a425b22131777ed9a4ccaaca034d72fa82b1d4917c6379929f21d269b2b"} Nov 02 06:20:54.384368 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:54.384331 5916 generic.go:356] "Generic (PLEG): container finished" podID="999d5daf-1bbf-46df-8184-e3454fae166b" containerID="f3eb9a93fa3eaaa46bf27571df9d918270000de2f77a6154c1420d9e5bcea853" exitCode=0 Nov 02 06:20:54.384481 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:54.384434 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7626s" event={"ID":"999d5daf-1bbf-46df-8184-e3454fae166b","Type":"ContainerDied","Data":"f3eb9a93fa3eaaa46bf27571df9d918270000de2f77a6154c1420d9e5bcea853"} Nov 02 06:20:54.391080 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:54.390969 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chkj82" event={"ID":"a7ef2c80-8b0c-4c7a-a5c1-ddb795124cbb","Type":"ContainerStarted","Data":"0239c5f825e39d043d5170be02dff3b30bfd005d921cf327b339213248000617"} Nov 02 06:20:54.602293 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:54.602260 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76v4x7" Nov 02 06:20:54.602579 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:54.602526 5916 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chkj82" podStartSLOduration=3.398412908 podStartE2EDuration="51.602507324s" totalImagesPullingTime="48.204094416s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:20:03 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:20:54.600588734 +0000 UTC m=+242.872409155" watchObservedRunningTime="2026-11-02 06:20:54.602507324 +0000 UTC m=+242.874327754" Nov 02 06:20:54.641339 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:54.641305 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bnjjk" Nov 02 06:20:54.654745 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:54.654622 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jjdpk" Nov 02 06:20:54.771894 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:54.771808 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70e3b8eb-ed1f-4552-8caf-113f26180ee5-util\" (UniqueName: \"kubernetes.io/empty-dir/70e3b8eb-ed1f-4552-8caf-113f26180ee5-util\") pod \"70e3b8eb-ed1f-4552-8caf-113f26180ee5\" (UID: \"70e3b8eb-ed1f-4552-8caf-113f26180ee5\") " Nov 02 06:20:54.771894 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:54.771858 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48b3fc06-ab52-4489-8e5d-84da4514f2da-util\" (UniqueName: \"kubernetes.io/empty-dir/48b3fc06-ab52-4489-8e5d-84da4514f2da-util\") pod \"48b3fc06-ab52-4489-8e5d-84da4514f2da\" (UID: \"48b3fc06-ab52-4489-8e5d-84da4514f2da\") " Nov 02 06:20:54.772115 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:54.771935 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70e3b8eb-ed1f-4552-8caf-113f26180ee5-bundle\" (UniqueName: \"kubernetes.io/empty-dir/70e3b8eb-ed1f-4552-8caf-113f26180ee5-bundle\") pod \"70e3b8eb-ed1f-4552-8caf-113f26180ee5\" (UID: \"70e3b8eb-ed1f-4552-8caf-113f26180ee5\") " Nov 02 06:20:54.772115 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:54.771988 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70e3b8eb-ed1f-4552-8caf-113f26180ee5-kube-api-access-d4gxr\" (UniqueName: \"kubernetes.io/projected/70e3b8eb-ed1f-4552-8caf-113f26180ee5-kube-api-access-d4gxr\") pod \"70e3b8eb-ed1f-4552-8caf-113f26180ee5\" (UID: \"70e3b8eb-ed1f-4552-8caf-113f26180ee5\") " Nov 02 06:20:54.772115 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:54.772040 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d93a0ea2-39a5-4bfa-b4c9-46df030a2dad-util\" (UniqueName: \"kubernetes.io/empty-dir/d93a0ea2-39a5-4bfa-b4c9-46df030a2dad-util\") pod \"d93a0ea2-39a5-4bfa-b4c9-46df030a2dad\" (UID: \"d93a0ea2-39a5-4bfa-b4c9-46df030a2dad\") " Nov 02 06:20:54.772115 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:54.772094 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48b3fc06-ab52-4489-8e5d-84da4514f2da-kube-api-access-pjmv7\" (UniqueName: \"kubernetes.io/projected/48b3fc06-ab52-4489-8e5d-84da4514f2da-kube-api-access-pjmv7\") pod \"48b3fc06-ab52-4489-8e5d-84da4514f2da\" (UID: \"48b3fc06-ab52-4489-8e5d-84da4514f2da\") " Nov 02 06:20:54.772358 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:54.772133 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d93a0ea2-39a5-4bfa-b4c9-46df030a2dad-bundle\" (UniqueName: \"kubernetes.io/empty-dir/d93a0ea2-39a5-4bfa-b4c9-46df030a2dad-bundle\") pod \"d93a0ea2-39a5-4bfa-b4c9-46df030a2dad\" (UID: \"d93a0ea2-39a5-4bfa-b4c9-46df030a2dad\") " Nov 02 06:20:54.772358 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:54.772169 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d93a0ea2-39a5-4bfa-b4c9-46df030a2dad-kube-api-access-4rrfj\" (UniqueName: \"kubernetes.io/projected/d93a0ea2-39a5-4bfa-b4c9-46df030a2dad-kube-api-access-4rrfj\") pod \"d93a0ea2-39a5-4bfa-b4c9-46df030a2dad\" (UID: \"d93a0ea2-39a5-4bfa-b4c9-46df030a2dad\") " Nov 02 06:20:54.772358 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:54.772206 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48b3fc06-ab52-4489-8e5d-84da4514f2da-bundle\" (UniqueName: \"kubernetes.io/empty-dir/48b3fc06-ab52-4489-8e5d-84da4514f2da-bundle\") pod \"48b3fc06-ab52-4489-8e5d-84da4514f2da\" (UID: \"48b3fc06-ab52-4489-8e5d-84da4514f2da\") " Nov 02 06:20:54.773257 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:54.773229 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d93a0ea2-39a5-4bfa-b4c9-46df030a2dad-bundle" pod "d93a0ea2-39a5-4bfa-b4c9-46df030a2dad" (UID: "d93a0ea2-39a5-4bfa-b4c9-46df030a2dad"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:20:54.775382 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:54.775354 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48b3fc06-ab52-4489-8e5d-84da4514f2da-kube-api-access-pjmv7" pod "48b3fc06-ab52-4489-8e5d-84da4514f2da" (UID: "48b3fc06-ab52-4489-8e5d-84da4514f2da"). InnerVolumeSpecName "kube-api-access-pjmv7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:20:54.775572 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:54.775545 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48b3fc06-ab52-4489-8e5d-84da4514f2da-bundle" pod "48b3fc06-ab52-4489-8e5d-84da4514f2da" (UID: "48b3fc06-ab52-4489-8e5d-84da4514f2da"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:20:54.776500 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:54.776473 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70e3b8eb-ed1f-4552-8caf-113f26180ee5-bundle" pod "70e3b8eb-ed1f-4552-8caf-113f26180ee5" (UID: "70e3b8eb-ed1f-4552-8caf-113f26180ee5"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:20:54.779157 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:54.779130 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d93a0ea2-39a5-4bfa-b4c9-46df030a2dad-util" pod "d93a0ea2-39a5-4bfa-b4c9-46df030a2dad" (UID: "d93a0ea2-39a5-4bfa-b4c9-46df030a2dad"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:20:54.786380 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:54.786357 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d93a0ea2-39a5-4bfa-b4c9-46df030a2dad-kube-api-access-4rrfj" pod "d93a0ea2-39a5-4bfa-b4c9-46df030a2dad" (UID: "d93a0ea2-39a5-4bfa-b4c9-46df030a2dad"). InnerVolumeSpecName "kube-api-access-4rrfj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:20:54.787816 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:54.787791 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48b3fc06-ab52-4489-8e5d-84da4514f2da-util" pod "48b3fc06-ab52-4489-8e5d-84da4514f2da" (UID: "48b3fc06-ab52-4489-8e5d-84da4514f2da"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:20:54.787979 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:54.787824 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70e3b8eb-ed1f-4552-8caf-113f26180ee5-kube-api-access-d4gxr" pod "70e3b8eb-ed1f-4552-8caf-113f26180ee5" (UID: "70e3b8eb-ed1f-4552-8caf-113f26180ee5"). InnerVolumeSpecName "kube-api-access-d4gxr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:20:54.788463 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:54.788444 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70e3b8eb-ed1f-4552-8caf-113f26180ee5-util" pod "70e3b8eb-ed1f-4552-8caf-113f26180ee5" (UID: "70e3b8eb-ed1f-4552-8caf-113f26180ee5"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:20:54.873656 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:54.873630 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4gxr\" (UniqueName: \"kubernetes.io/projected/70e3b8eb-ed1f-4552-8caf-113f26180ee5-kube-api-access-d4gxr\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:20:54.873656 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:54.873655 5916 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d93a0ea2-39a5-4bfa-b4c9-46df030a2dad-util\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:20:54.873832 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:54.873671 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjmv7\" (UniqueName: \"kubernetes.io/projected/48b3fc06-ab52-4489-8e5d-84da4514f2da-kube-api-access-pjmv7\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:20:54.873832 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:54.873684 5916 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d93a0ea2-39a5-4bfa-b4c9-46df030a2dad-bundle\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:20:54.873832 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:54.873698 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rrfj\" (UniqueName: \"kubernetes.io/projected/d93a0ea2-39a5-4bfa-b4c9-46df030a2dad-kube-api-access-4rrfj\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:20:54.873832 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:54.873706 5916 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/48b3fc06-ab52-4489-8e5d-84da4514f2da-bundle\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:20:54.873832 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:54.873715 5916 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/70e3b8eb-ed1f-4552-8caf-113f26180ee5-util\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:20:54.873832 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:54.873723 5916 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/48b3fc06-ab52-4489-8e5d-84da4514f2da-util\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:20:54.873832 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:54.873731 5916 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/70e3b8eb-ed1f-4552-8caf-113f26180ee5-bundle\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:20:55.396730 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:55.396696 5916 generic.go:356] "Generic (PLEG): container finished" podID="aa5d2e0d-5b30-415e-86b0-d3a7e246ffa7" containerID="88737675bb254664832400693fccbd8d7734f1a2a75ae1bec1be6d8634ca167f" exitCode=0 Nov 02 06:20:55.397219 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:55.396764 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mfmhs" event={"ID":"aa5d2e0d-5b30-415e-86b0-d3a7e246ffa7","Type":"ContainerDied","Data":"88737675bb254664832400693fccbd8d7734f1a2a75ae1bec1be6d8634ca167f"} Nov 02 06:20:55.399535 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:55.399508 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2rnzs" event={"ID":"368fe2a8-af6b-4d60-a6cf-6a82dff993a0","Type":"ContainerStarted","Data":"0a698e01c1c2c57b7c23118dab24fe0c501d98636d3d64c97b663ee43e215f79"} Nov 02 06:20:55.401443 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:55.401423 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76v4x7" Nov 02 06:20:55.401558 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:55.401444 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76v4x7" event={"ID":"d93a0ea2-39a5-4bfa-b4c9-46df030a2dad","Type":"ContainerDied","Data":"be0e42e3fd39b0cebcd20693a6b06e59800872722745f00a64420cc3f78273fe"} Nov 02 06:20:55.401558 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:55.401485 5916 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="be0e42e3fd39b0cebcd20693a6b06e59800872722745f00a64420cc3f78273fe" Nov 02 06:20:55.403680 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:55.403654 5916 generic.go:356] "Generic (PLEG): container finished" podID="c48449eb-30cb-4b07-892a-b27192f49e0f" containerID="b23e3a5e04a7bc0e7cf79fbd2b916bed841499c7052a5cfd2a1ed67ff3f470f6" exitCode=0 Nov 02 06:20:55.403808 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:55.403740 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agdmqn" event={"ID":"c48449eb-30cb-4b07-892a-b27192f49e0f","Type":"ContainerDied","Data":"b23e3a5e04a7bc0e7cf79fbd2b916bed841499c7052a5cfd2a1ed67ff3f470f6"} Nov 02 06:20:55.406300 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:55.406197 5916 generic.go:356] "Generic (PLEG): container finished" podID="a7ef2c80-8b0c-4c7a-a5c1-ddb795124cbb" containerID="0239c5f825e39d043d5170be02dff3b30bfd005d921cf327b339213248000617" exitCode=0 Nov 02 06:20:55.406300 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:55.406266 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chkj82" event={"ID":"a7ef2c80-8b0c-4c7a-a5c1-ddb795124cbb","Type":"ContainerDied","Data":"0239c5f825e39d043d5170be02dff3b30bfd005d921cf327b339213248000617"} Nov 02 06:20:55.408168 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:55.408142 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jjdpk" event={"ID":"70e3b8eb-ed1f-4552-8caf-113f26180ee5","Type":"ContainerDied","Data":"7cd5682b30d82cf837c97ee33e87c6fdd5a6c75be0a5b7cd695c64d81593e102"} Nov 02 06:20:55.408168 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:55.408152 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jjdpk" Nov 02 06:20:55.408319 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:55.408171 5916 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7cd5682b30d82cf837c97ee33e87c6fdd5a6c75be0a5b7cd695c64d81593e102" Nov 02 06:20:55.411700 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:55.411676 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crhbtq" event={"ID":"5a659a95-82ea-47c7-937d-17b4407d1eb2","Type":"ContainerStarted","Data":"b2855e02e805303ddf9cb98eb27fb0eeeb88560f08ba3110734f1b57498bccbc"} Nov 02 06:20:55.414111 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:55.414091 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xr9qf" event={"ID":"05e071cb-0da9-4fff-9b10-5ab9b325b1c6","Type":"ContainerStarted","Data":"73ea59a6eb3a7ec385683a9b8f09515b7ddb3df8e1c0d50c0a4f67ae0bec07ab"} Nov 02 06:20:55.415930 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:55.415893 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bnjjk" event={"ID":"48b3fc06-ab52-4489-8e5d-84da4514f2da","Type":"ContainerDied","Data":"a0dd6f46ac1857b1a6a31fd0cb94f8fa7b1db3d796d50b10a5e8bedf45228060"} Nov 02 06:20:55.416001 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:55.415939 5916 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a0dd6f46ac1857b1a6a31fd0cb94f8fa7b1db3d796d50b10a5e8bedf45228060" Nov 02 06:20:55.416001 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:55.415946 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bnjjk" Nov 02 06:20:55.589054 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:55.588977 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7626s" Nov 02 06:20:55.592711 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:55.592689 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j2zjn" Nov 02 06:20:55.683575 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:55.683542 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef4df80a-7c4f-46fd-9cfe-5ba25a89f869-kube-api-access-n7gtc\" (UniqueName: \"kubernetes.io/projected/ef4df80a-7c4f-46fd-9cfe-5ba25a89f869-kube-api-access-n7gtc\") pod \"ef4df80a-7c4f-46fd-9cfe-5ba25a89f869\" (UID: \"ef4df80a-7c4f-46fd-9cfe-5ba25a89f869\") " Nov 02 06:20:55.683723 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:55.683591 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/999d5daf-1bbf-46df-8184-e3454fae166b-bundle\" (UniqueName: \"kubernetes.io/empty-dir/999d5daf-1bbf-46df-8184-e3454fae166b-bundle\") pod \"999d5daf-1bbf-46df-8184-e3454fae166b\" (UID: \"999d5daf-1bbf-46df-8184-e3454fae166b\") " Nov 02 06:20:55.683723 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:55.683623 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef4df80a-7c4f-46fd-9cfe-5ba25a89f869-util\" (UniqueName: \"kubernetes.io/empty-dir/ef4df80a-7c4f-46fd-9cfe-5ba25a89f869-util\") pod \"ef4df80a-7c4f-46fd-9cfe-5ba25a89f869\" (UID: \"ef4df80a-7c4f-46fd-9cfe-5ba25a89f869\") " Nov 02 06:20:55.683807 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:55.683720 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/999d5daf-1bbf-46df-8184-e3454fae166b-util\" (UniqueName: \"kubernetes.io/empty-dir/999d5daf-1bbf-46df-8184-e3454fae166b-util\") pod \"999d5daf-1bbf-46df-8184-e3454fae166b\" (UID: \"999d5daf-1bbf-46df-8184-e3454fae166b\") " Nov 02 06:20:55.683807 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:55.683755 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef4df80a-7c4f-46fd-9cfe-5ba25a89f869-bundle\" (UniqueName: \"kubernetes.io/empty-dir/ef4df80a-7c4f-46fd-9cfe-5ba25a89f869-bundle\") pod \"ef4df80a-7c4f-46fd-9cfe-5ba25a89f869\" (UID: \"ef4df80a-7c4f-46fd-9cfe-5ba25a89f869\") " Nov 02 06:20:55.683807 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:55.683786 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/999d5daf-1bbf-46df-8184-e3454fae166b-kube-api-access-ng4cf\" (UniqueName: \"kubernetes.io/projected/999d5daf-1bbf-46df-8184-e3454fae166b-kube-api-access-ng4cf\") pod \"999d5daf-1bbf-46df-8184-e3454fae166b\" (UID: \"999d5daf-1bbf-46df-8184-e3454fae166b\") " Nov 02 06:20:55.684257 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:55.684224 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef4df80a-7c4f-46fd-9cfe-5ba25a89f869-bundle" pod "ef4df80a-7c4f-46fd-9cfe-5ba25a89f869" (UID: "ef4df80a-7c4f-46fd-9cfe-5ba25a89f869"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:20:55.684452 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:55.684427 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/999d5daf-1bbf-46df-8184-e3454fae166b-bundle" pod "999d5daf-1bbf-46df-8184-e3454fae166b" (UID: "999d5daf-1bbf-46df-8184-e3454fae166b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:20:55.685689 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:55.685667 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef4df80a-7c4f-46fd-9cfe-5ba25a89f869-kube-api-access-n7gtc" pod "ef4df80a-7c4f-46fd-9cfe-5ba25a89f869" (UID: "ef4df80a-7c4f-46fd-9cfe-5ba25a89f869"). InnerVolumeSpecName "kube-api-access-n7gtc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:20:55.685835 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:55.685816 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/999d5daf-1bbf-46df-8184-e3454fae166b-kube-api-access-ng4cf" pod "999d5daf-1bbf-46df-8184-e3454fae166b" (UID: "999d5daf-1bbf-46df-8184-e3454fae166b"). InnerVolumeSpecName "kube-api-access-ng4cf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:20:55.689761 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:55.689738 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef4df80a-7c4f-46fd-9cfe-5ba25a89f869-util" pod "ef4df80a-7c4f-46fd-9cfe-5ba25a89f869" (UID: "ef4df80a-7c4f-46fd-9cfe-5ba25a89f869"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:20:55.690086 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:55.690067 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/999d5daf-1bbf-46df-8184-e3454fae166b-util" pod "999d5daf-1bbf-46df-8184-e3454fae166b" (UID: "999d5daf-1bbf-46df-8184-e3454fae166b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:20:55.784723 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:55.784692 5916 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/999d5daf-1bbf-46df-8184-e3454fae166b-util\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:20:55.784723 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:55.784717 5916 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ef4df80a-7c4f-46fd-9cfe-5ba25a89f869-bundle\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:20:55.784900 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:55.784730 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ng4cf\" (UniqueName: \"kubernetes.io/projected/999d5daf-1bbf-46df-8184-e3454fae166b-kube-api-access-ng4cf\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:20:55.784900 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:55.784743 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7gtc\" (UniqueName: \"kubernetes.io/projected/ef4df80a-7c4f-46fd-9cfe-5ba25a89f869-kube-api-access-n7gtc\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:20:55.784900 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:55.784757 5916 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/999d5daf-1bbf-46df-8184-e3454fae166b-bundle\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:20:55.784900 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:55.784767 5916 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ef4df80a-7c4f-46fd-9cfe-5ba25a89f869-util\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:20:55.921645 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:55.921590 5916 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crhbtq" podStartSLOduration=3.537645087 podStartE2EDuration="45.921576459s" totalImagesPullingTime="42.383931372s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:20:10 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:20:55.921432521 +0000 UTC m=+244.193252914" watchObservedRunningTime="2026-11-02 06:20:55.921576459 +0000 UTC m=+244.193396853" Nov 02 06:20:56.031028 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:56.030979 5916 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xr9qf" podStartSLOduration=3.406687309 podStartE2EDuration="48.030964726s" totalImagesPullingTime="44.624277417s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:20:08 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:20:56.028799472 +0000 UTC m=+244.300619866" watchObservedRunningTime="2026-11-02 06:20:56.030964726 +0000 UTC m=+244.302785120" Nov 02 06:20:56.185144 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:56.185092 5916 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2rnzs" podStartSLOduration=4.389971769 podStartE2EDuration="53.185075925s" totalImagesPullingTime="48.795104156s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:20:03 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:20:56.184205019 +0000 UTC m=+244.456025412" watchObservedRunningTime="2026-11-02 06:20:56.185075925 +0000 UTC m=+244.456896347" Nov 02 06:20:56.423007 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:56.422971 5916 generic.go:356] "Generic (PLEG): container finished" podID="05e071cb-0da9-4fff-9b10-5ab9b325b1c6" containerID="73ea59a6eb3a7ec385683a9b8f09515b7ddb3df8e1c0d50c0a4f67ae0bec07ab" exitCode=0 Nov 02 06:20:56.423435 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:56.423057 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xr9qf" event={"ID":"05e071cb-0da9-4fff-9b10-5ab9b325b1c6","Type":"ContainerDied","Data":"73ea59a6eb3a7ec385683a9b8f09515b7ddb3df8e1c0d50c0a4f67ae0bec07ab"} Nov 02 06:20:56.425368 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:56.425346 5916 generic.go:356] "Generic (PLEG): container finished" podID="368fe2a8-af6b-4d60-a6cf-6a82dff993a0" containerID="0a698e01c1c2c57b7c23118dab24fe0c501d98636d3d64c97b663ee43e215f79" exitCode=0 Nov 02 06:20:56.425474 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:56.425423 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2rnzs" event={"ID":"368fe2a8-af6b-4d60-a6cf-6a82dff993a0","Type":"ContainerDied","Data":"0a698e01c1c2c57b7c23118dab24fe0c501d98636d3d64c97b663ee43e215f79"} Nov 02 06:20:56.427320 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:56.427295 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j2zjn" event={"ID":"ef4df80a-7c4f-46fd-9cfe-5ba25a89f869","Type":"ContainerDied","Data":"eb3749f02b042230bf6e763207204acf9239fadf6bf651ff4a91a5d80e11932f"} Nov 02 06:20:56.427320 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:56.427319 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j2zjn" Nov 02 06:20:56.427477 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:56.427335 5916 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eb3749f02b042230bf6e763207204acf9239fadf6bf651ff4a91a5d80e11932f" Nov 02 06:20:56.429478 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:56.429458 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7626s" Nov 02 06:20:56.429626 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:56.429493 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7626s" event={"ID":"999d5daf-1bbf-46df-8184-e3454fae166b","Type":"ContainerDied","Data":"87b8f60610570e25af52a13cd6518213e7afb0f10cbcd7108aab45e1bbe217a5"} Nov 02 06:20:56.429626 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:56.429530 5916 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="87b8f60610570e25af52a13cd6518213e7afb0f10cbcd7108aab45e1bbe217a5" Nov 02 06:20:56.432390 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:56.432367 5916 generic.go:356] "Generic (PLEG): container finished" podID="5a659a95-82ea-47c7-937d-17b4407d1eb2" containerID="b2855e02e805303ddf9cb98eb27fb0eeeb88560f08ba3110734f1b57498bccbc" exitCode=0 Nov 02 06:20:56.432496 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:56.432457 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crhbtq" event={"ID":"5a659a95-82ea-47c7-937d-17b4407d1eb2","Type":"ContainerDied","Data":"b2855e02e805303ddf9cb98eb27fb0eeeb88560f08ba3110734f1b57498bccbc"} Nov 02 06:20:56.563410 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:56.563383 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mfmhs" Nov 02 06:20:56.590165 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:56.590134 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa5d2e0d-5b30-415e-86b0-d3a7e246ffa7-bundle\" (UniqueName: \"kubernetes.io/empty-dir/aa5d2e0d-5b30-415e-86b0-d3a7e246ffa7-bundle\") pod \"aa5d2e0d-5b30-415e-86b0-d3a7e246ffa7\" (UID: \"aa5d2e0d-5b30-415e-86b0-d3a7e246ffa7\") " Nov 02 06:20:56.590309 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:56.590194 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa5d2e0d-5b30-415e-86b0-d3a7e246ffa7-util\" (UniqueName: \"kubernetes.io/empty-dir/aa5d2e0d-5b30-415e-86b0-d3a7e246ffa7-util\") pod \"aa5d2e0d-5b30-415e-86b0-d3a7e246ffa7\" (UID: \"aa5d2e0d-5b30-415e-86b0-d3a7e246ffa7\") " Nov 02 06:20:56.590309 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:56.590247 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa5d2e0d-5b30-415e-86b0-d3a7e246ffa7-kube-api-access-jv28r\" (UniqueName: \"kubernetes.io/projected/aa5d2e0d-5b30-415e-86b0-d3a7e246ffa7-kube-api-access-jv28r\") pod \"aa5d2e0d-5b30-415e-86b0-d3a7e246ffa7\" (UID: \"aa5d2e0d-5b30-415e-86b0-d3a7e246ffa7\") " Nov 02 06:20:56.590765 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:56.590734 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa5d2e0d-5b30-415e-86b0-d3a7e246ffa7-bundle" pod "aa5d2e0d-5b30-415e-86b0-d3a7e246ffa7" (UID: "aa5d2e0d-5b30-415e-86b0-d3a7e246ffa7"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:20:56.593257 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:56.593229 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa5d2e0d-5b30-415e-86b0-d3a7e246ffa7-kube-api-access-jv28r" pod "aa5d2e0d-5b30-415e-86b0-d3a7e246ffa7" (UID: "aa5d2e0d-5b30-415e-86b0-d3a7e246ffa7"). InnerVolumeSpecName "kube-api-access-jv28r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:20:56.596939 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:56.596885 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa5d2e0d-5b30-415e-86b0-d3a7e246ffa7-util" pod "aa5d2e0d-5b30-415e-86b0-d3a7e246ffa7" (UID: "aa5d2e0d-5b30-415e-86b0-d3a7e246ffa7"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:20:56.601023 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:56.601001 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chkj82" Nov 02 06:20:56.623507 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:56.623490 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agdmqn" Nov 02 06:20:56.690879 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:56.690848 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7ef2c80-8b0c-4c7a-a5c1-ddb795124cbb-util\" (UniqueName: \"kubernetes.io/empty-dir/a7ef2c80-8b0c-4c7a-a5c1-ddb795124cbb-util\") pod \"a7ef2c80-8b0c-4c7a-a5c1-ddb795124cbb\" (UID: \"a7ef2c80-8b0c-4c7a-a5c1-ddb795124cbb\") " Nov 02 06:20:56.691010 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:56.690946 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c48449eb-30cb-4b07-892a-b27192f49e0f-kube-api-access-zpvg7\" (UniqueName: \"kubernetes.io/projected/c48449eb-30cb-4b07-892a-b27192f49e0f-kube-api-access-zpvg7\") pod \"c48449eb-30cb-4b07-892a-b27192f49e0f\" (UID: \"c48449eb-30cb-4b07-892a-b27192f49e0f\") " Nov 02 06:20:56.691047 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:56.691013 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7ef2c80-8b0c-4c7a-a5c1-ddb795124cbb-kube-api-access-kmf8j\" (UniqueName: \"kubernetes.io/projected/a7ef2c80-8b0c-4c7a-a5c1-ddb795124cbb-kube-api-access-kmf8j\") pod \"a7ef2c80-8b0c-4c7a-a5c1-ddb795124cbb\" (UID: \"a7ef2c80-8b0c-4c7a-a5c1-ddb795124cbb\") " Nov 02 06:20:56.691092 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:56.691065 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c48449eb-30cb-4b07-892a-b27192f49e0f-util\" (UniqueName: \"kubernetes.io/empty-dir/c48449eb-30cb-4b07-892a-b27192f49e0f-util\") pod \"c48449eb-30cb-4b07-892a-b27192f49e0f\" (UID: \"c48449eb-30cb-4b07-892a-b27192f49e0f\") " Nov 02 06:20:56.691297 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:56.691282 5916 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aa5d2e0d-5b30-415e-86b0-d3a7e246ffa7-bundle\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:20:56.691375 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:56.691299 5916 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aa5d2e0d-5b30-415e-86b0-d3a7e246ffa7-util\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:20:56.691375 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:56.691308 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jv28r\" (UniqueName: \"kubernetes.io/projected/aa5d2e0d-5b30-415e-86b0-d3a7e246ffa7-kube-api-access-jv28r\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:20:56.693024 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:56.692992 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c48449eb-30cb-4b07-892a-b27192f49e0f-kube-api-access-zpvg7" pod "c48449eb-30cb-4b07-892a-b27192f49e0f" (UID: "c48449eb-30cb-4b07-892a-b27192f49e0f"). InnerVolumeSpecName "kube-api-access-zpvg7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:20:56.693098 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:56.693053 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7ef2c80-8b0c-4c7a-a5c1-ddb795124cbb-kube-api-access-kmf8j" pod "a7ef2c80-8b0c-4c7a-a5c1-ddb795124cbb" (UID: "a7ef2c80-8b0c-4c7a-a5c1-ddb795124cbb"). InnerVolumeSpecName "kube-api-access-kmf8j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:20:56.696490 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:56.696472 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7ef2c80-8b0c-4c7a-a5c1-ddb795124cbb-util" pod "a7ef2c80-8b0c-4c7a-a5c1-ddb795124cbb" (UID: "a7ef2c80-8b0c-4c7a-a5c1-ddb795124cbb"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:20:56.696592 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:56.696572 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c48449eb-30cb-4b07-892a-b27192f49e0f-util" pod "c48449eb-30cb-4b07-892a-b27192f49e0f" (UID: "c48449eb-30cb-4b07-892a-b27192f49e0f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:20:56.791895 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:56.791833 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c48449eb-30cb-4b07-892a-b27192f49e0f-bundle\" (UniqueName: \"kubernetes.io/empty-dir/c48449eb-30cb-4b07-892a-b27192f49e0f-bundle\") pod \"c48449eb-30cb-4b07-892a-b27192f49e0f\" (UID: \"c48449eb-30cb-4b07-892a-b27192f49e0f\") " Nov 02 06:20:56.791895 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:56.791870 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7ef2c80-8b0c-4c7a-a5c1-ddb795124cbb-bundle\" (UniqueName: \"kubernetes.io/empty-dir/a7ef2c80-8b0c-4c7a-a5c1-ddb795124cbb-bundle\") pod \"a7ef2c80-8b0c-4c7a-a5c1-ddb795124cbb\" (UID: \"a7ef2c80-8b0c-4c7a-a5c1-ddb795124cbb\") " Nov 02 06:20:56.792152 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:56.792133 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kmf8j\" (UniqueName: \"kubernetes.io/projected/a7ef2c80-8b0c-4c7a-a5c1-ddb795124cbb-kube-api-access-kmf8j\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:20:56.792188 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:56.792159 5916 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c48449eb-30cb-4b07-892a-b27192f49e0f-util\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:20:56.792188 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:56.792176 5916 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a7ef2c80-8b0c-4c7a-a5c1-ddb795124cbb-util\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:20:56.792251 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:56.792189 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zpvg7\" (UniqueName: \"kubernetes.io/projected/c48449eb-30cb-4b07-892a-b27192f49e0f-kube-api-access-zpvg7\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:20:56.792375 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:56.792355 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7ef2c80-8b0c-4c7a-a5c1-ddb795124cbb-bundle" pod "a7ef2c80-8b0c-4c7a-a5c1-ddb795124cbb" (UID: "a7ef2c80-8b0c-4c7a-a5c1-ddb795124cbb"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:20:56.792520 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:56.792503 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c48449eb-30cb-4b07-892a-b27192f49e0f-bundle" pod "c48449eb-30cb-4b07-892a-b27192f49e0f" (UID: "c48449eb-30cb-4b07-892a-b27192f49e0f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:20:56.892592 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:56.892563 5916 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c48449eb-30cb-4b07-892a-b27192f49e0f-bundle\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:20:56.892592 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:56.892592 5916 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a7ef2c80-8b0c-4c7a-a5c1-ddb795124cbb-bundle\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:20:57.439571 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:57.439543 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mfmhs" Nov 02 06:20:57.439571 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:57.439553 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mfmhs" event={"ID":"aa5d2e0d-5b30-415e-86b0-d3a7e246ffa7","Type":"ContainerDied","Data":"9d00ae0a72fb68e3788d325b7531ef09619ca2d5113967477380bf13c3a933e1"} Nov 02 06:20:57.440109 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:57.439604 5916 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9d00ae0a72fb68e3788d325b7531ef09619ca2d5113967477380bf13c3a933e1" Nov 02 06:20:57.441577 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:57.441548 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agdmqn" Nov 02 06:20:57.441577 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:57.441555 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agdmqn" event={"ID":"c48449eb-30cb-4b07-892a-b27192f49e0f","Type":"ContainerDied","Data":"a6cfc876b9b8d8fbfdc462627e6ac2e5b4d41308ede7c1c76a2d43302470e228"} Nov 02 06:20:57.441738 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:57.441594 5916 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a6cfc876b9b8d8fbfdc462627e6ac2e5b4d41308ede7c1c76a2d43302470e228" Nov 02 06:20:57.443496 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:57.443470 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chkj82" event={"ID":"a7ef2c80-8b0c-4c7a-a5c1-ddb795124cbb","Type":"ContainerDied","Data":"979d7f5bc6a21f7f6f87e586e2b0e3442532472095900298c444f3bbd87d7f8f"} Nov 02 06:20:57.443598 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:57.443514 5916 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="979d7f5bc6a21f7f6f87e586e2b0e3442532472095900298c444f3bbd87d7f8f" Nov 02 06:20:57.443598 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:57.443548 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chkj82" Nov 02 06:20:57.592955 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:57.592931 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crhbtq" Nov 02 06:20:57.597579 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:57.597553 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a659a95-82ea-47c7-937d-17b4407d1eb2-bundle\" (UniqueName: \"kubernetes.io/empty-dir/5a659a95-82ea-47c7-937d-17b4407d1eb2-bundle\") pod \"5a659a95-82ea-47c7-937d-17b4407d1eb2\" (UID: \"5a659a95-82ea-47c7-937d-17b4407d1eb2\") " Nov 02 06:20:57.598227 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:57.598195 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a659a95-82ea-47c7-937d-17b4407d1eb2-bundle" pod "5a659a95-82ea-47c7-937d-17b4407d1eb2" (UID: "5a659a95-82ea-47c7-937d-17b4407d1eb2"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:20:57.620606 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:57.620585 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xr9qf" Nov 02 06:20:57.623212 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:57.623189 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2rnzs" Nov 02 06:20:57.698390 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:57.698321 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05e071cb-0da9-4fff-9b10-5ab9b325b1c6-bundle\" (UniqueName: \"kubernetes.io/empty-dir/05e071cb-0da9-4fff-9b10-5ab9b325b1c6-bundle\") pod \"05e071cb-0da9-4fff-9b10-5ab9b325b1c6\" (UID: \"05e071cb-0da9-4fff-9b10-5ab9b325b1c6\") " Nov 02 06:20:57.698390 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:57.698362 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/368fe2a8-af6b-4d60-a6cf-6a82dff993a0-util\" (UniqueName: \"kubernetes.io/empty-dir/368fe2a8-af6b-4d60-a6cf-6a82dff993a0-util\") pod \"368fe2a8-af6b-4d60-a6cf-6a82dff993a0\" (UID: \"368fe2a8-af6b-4d60-a6cf-6a82dff993a0\") " Nov 02 06:20:57.698390 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:57.698393 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/368fe2a8-af6b-4d60-a6cf-6a82dff993a0-kube-api-access-ntgl4\" (UniqueName: \"kubernetes.io/projected/368fe2a8-af6b-4d60-a6cf-6a82dff993a0-kube-api-access-ntgl4\") pod \"368fe2a8-af6b-4d60-a6cf-6a82dff993a0\" (UID: \"368fe2a8-af6b-4d60-a6cf-6a82dff993a0\") " Nov 02 06:20:57.698662 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:57.698409 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05e071cb-0da9-4fff-9b10-5ab9b325b1c6-kube-api-access-x9ndg\" (UniqueName: \"kubernetes.io/projected/05e071cb-0da9-4fff-9b10-5ab9b325b1c6-kube-api-access-x9ndg\") pod \"05e071cb-0da9-4fff-9b10-5ab9b325b1c6\" (UID: \"05e071cb-0da9-4fff-9b10-5ab9b325b1c6\") " Nov 02 06:20:57.698662 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:57.698442 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a659a95-82ea-47c7-937d-17b4407d1eb2-util\" (UniqueName: \"kubernetes.io/empty-dir/5a659a95-82ea-47c7-937d-17b4407d1eb2-util\") pod \"5a659a95-82ea-47c7-937d-17b4407d1eb2\" (UID: \"5a659a95-82ea-47c7-937d-17b4407d1eb2\") " Nov 02 06:20:57.698662 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:57.698465 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/368fe2a8-af6b-4d60-a6cf-6a82dff993a0-bundle\" (UniqueName: \"kubernetes.io/empty-dir/368fe2a8-af6b-4d60-a6cf-6a82dff993a0-bundle\") pod \"368fe2a8-af6b-4d60-a6cf-6a82dff993a0\" (UID: \"368fe2a8-af6b-4d60-a6cf-6a82dff993a0\") " Nov 02 06:20:57.698662 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:57.698486 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05e071cb-0da9-4fff-9b10-5ab9b325b1c6-util\" (UniqueName: \"kubernetes.io/empty-dir/05e071cb-0da9-4fff-9b10-5ab9b325b1c6-util\") pod \"05e071cb-0da9-4fff-9b10-5ab9b325b1c6\" (UID: \"05e071cb-0da9-4fff-9b10-5ab9b325b1c6\") " Nov 02 06:20:57.698662 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:57.698525 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a659a95-82ea-47c7-937d-17b4407d1eb2-kube-api-access-fc77s\" (UniqueName: \"kubernetes.io/projected/5a659a95-82ea-47c7-937d-17b4407d1eb2-kube-api-access-fc77s\") pod \"5a659a95-82ea-47c7-937d-17b4407d1eb2\" (UID: \"5a659a95-82ea-47c7-937d-17b4407d1eb2\") " Nov 02 06:20:57.698861 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:57.698687 5916 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5a659a95-82ea-47c7-937d-17b4407d1eb2-bundle\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:20:57.698861 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:57.698760 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05e071cb-0da9-4fff-9b10-5ab9b325b1c6-bundle" pod "05e071cb-0da9-4fff-9b10-5ab9b325b1c6" (UID: "05e071cb-0da9-4fff-9b10-5ab9b325b1c6"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:20:57.699298 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:57.699270 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/368fe2a8-af6b-4d60-a6cf-6a82dff993a0-bundle" pod "368fe2a8-af6b-4d60-a6cf-6a82dff993a0" (UID: "368fe2a8-af6b-4d60-a6cf-6a82dff993a0"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:20:57.701069 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:57.701045 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a659a95-82ea-47c7-937d-17b4407d1eb2-kube-api-access-fc77s" pod "5a659a95-82ea-47c7-937d-17b4407d1eb2" (UID: "5a659a95-82ea-47c7-937d-17b4407d1eb2"). InnerVolumeSpecName "kube-api-access-fc77s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:20:57.701171 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:57.701108 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/368fe2a8-af6b-4d60-a6cf-6a82dff993a0-kube-api-access-ntgl4" pod "368fe2a8-af6b-4d60-a6cf-6a82dff993a0" (UID: "368fe2a8-af6b-4d60-a6cf-6a82dff993a0"). InnerVolumeSpecName "kube-api-access-ntgl4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:20:57.701275 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:57.701256 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05e071cb-0da9-4fff-9b10-5ab9b325b1c6-kube-api-access-x9ndg" pod "05e071cb-0da9-4fff-9b10-5ab9b325b1c6" (UID: "05e071cb-0da9-4fff-9b10-5ab9b325b1c6"). InnerVolumeSpecName "kube-api-access-x9ndg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:20:57.705176 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:57.705155 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/368fe2a8-af6b-4d60-a6cf-6a82dff993a0-util" pod "368fe2a8-af6b-4d60-a6cf-6a82dff993a0" (UID: "368fe2a8-af6b-4d60-a6cf-6a82dff993a0"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:20:57.705388 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:57.705368 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a659a95-82ea-47c7-937d-17b4407d1eb2-util" pod "5a659a95-82ea-47c7-937d-17b4407d1eb2" (UID: "5a659a95-82ea-47c7-937d-17b4407d1eb2"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:20:57.705972 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:57.705947 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05e071cb-0da9-4fff-9b10-5ab9b325b1c6-util" pod "05e071cb-0da9-4fff-9b10-5ab9b325b1c6" (UID: "05e071cb-0da9-4fff-9b10-5ab9b325b1c6"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:20:57.799105 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:57.799067 5916 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/05e071cb-0da9-4fff-9b10-5ab9b325b1c6-bundle\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:20:57.799105 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:57.799099 5916 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/368fe2a8-af6b-4d60-a6cf-6a82dff993a0-util\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:20:57.799105 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:57.799109 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ntgl4\" (UniqueName: \"kubernetes.io/projected/368fe2a8-af6b-4d60-a6cf-6a82dff993a0-kube-api-access-ntgl4\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:20:57.799327 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:57.799121 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9ndg\" (UniqueName: \"kubernetes.io/projected/05e071cb-0da9-4fff-9b10-5ab9b325b1c6-kube-api-access-x9ndg\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:20:57.799327 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:57.799130 5916 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5a659a95-82ea-47c7-937d-17b4407d1eb2-util\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:20:57.799327 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:57.799138 5916 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/368fe2a8-af6b-4d60-a6cf-6a82dff993a0-bundle\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:20:57.799327 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:57.799145 5916 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/05e071cb-0da9-4fff-9b10-5ab9b325b1c6-util\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:20:57.799327 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:57.799153 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fc77s\" (UniqueName: \"kubernetes.io/projected/5a659a95-82ea-47c7-937d-17b4407d1eb2-kube-api-access-fc77s\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:20:58.450656 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:58.450626 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2rnzs" Nov 02 06:20:58.450656 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:58.450635 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2rnzs" event={"ID":"368fe2a8-af6b-4d60-a6cf-6a82dff993a0","Type":"ContainerDied","Data":"1f20e247641fae0ede97bd0022b0fe4fe911ed91a0ba265e87e9ac38a45aab29"} Nov 02 06:20:58.451139 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:58.450694 5916 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1f20e247641fae0ede97bd0022b0fe4fe911ed91a0ba265e87e9ac38a45aab29" Nov 02 06:20:58.452769 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:58.452736 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crhbtq" event={"ID":"5a659a95-82ea-47c7-937d-17b4407d1eb2","Type":"ContainerDied","Data":"e5fdbd631e1414863682656c7089820e6c40bbd9d0abe0b481d4510cccfb6fd5"} Nov 02 06:20:58.452769 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:58.452761 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crhbtq" Nov 02 06:20:58.452975 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:58.452782 5916 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e5fdbd631e1414863682656c7089820e6c40bbd9d0abe0b481d4510cccfb6fd5" Nov 02 06:20:58.454930 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:58.454892 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xr9qf" event={"ID":"05e071cb-0da9-4fff-9b10-5ab9b325b1c6","Type":"ContainerDied","Data":"ff2c10212c57340bd560778fe35ba7c673129e18ab9dc8c86085fc1e5443735f"} Nov 02 06:20:58.455069 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:58.454933 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xr9qf" Nov 02 06:20:58.455069 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:20:58.454941 5916 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ff2c10212c57340bd560778fe35ba7c673129e18ab9dc8c86085fc1e5443735f" Nov 02 06:21:00.104796 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:00.104755 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-565d9484c7-jpgdm"] Nov 02 06:21:00.105242 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:00.105075 5916 cpu_manager.go:414] "containerMap: removing container" podUID="368fe2a8-af6b-4d60-a6cf-6a82dff993a0" containerName="pull" Nov 02 06:21:00.105242 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:00.105086 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="368fe2a8-af6b-4d60-a6cf-6a82dff993a0" containerName="pull" Nov 02 06:21:00.105242 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:00.105098 5916 cpu_manager.go:414] "containerMap: removing container" podUID="48b3fc06-ab52-4489-8e5d-84da4514f2da" containerName="extract" Nov 02 06:21:00.105242 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:00.105102 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48b3fc06-ab52-4489-8e5d-84da4514f2da" containerName="extract" Nov 02 06:21:00.105242 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:00.105111 5916 cpu_manager.go:414] "containerMap: removing container" podUID="05e071cb-0da9-4fff-9b10-5ab9b325b1c6" containerName="extract" Nov 02 06:21:00.105242 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:00.105116 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05e071cb-0da9-4fff-9b10-5ab9b325b1c6" containerName="extract" Nov 02 06:21:00.105242 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:00.105124 5916 cpu_manager.go:414] "containerMap: removing container" podUID="ef4df80a-7c4f-46fd-9cfe-5ba25a89f869" containerName="util" Nov 02 06:21:00.105242 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:00.105128 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef4df80a-7c4f-46fd-9cfe-5ba25a89f869" containerName="util" Nov 02 06:21:00.105242 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:00.105134 5916 cpu_manager.go:414] "containerMap: removing container" podUID="05e071cb-0da9-4fff-9b10-5ab9b325b1c6" containerName="pull" Nov 02 06:21:00.105242 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:00.105142 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05e071cb-0da9-4fff-9b10-5ab9b325b1c6" containerName="pull" Nov 02 06:21:00.105242 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:00.105152 5916 cpu_manager.go:414] "containerMap: removing container" podUID="c48449eb-30cb-4b07-892a-b27192f49e0f" containerName="extract" Nov 02 06:21:00.105242 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:00.105159 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c48449eb-30cb-4b07-892a-b27192f49e0f" containerName="extract" Nov 02 06:21:00.105242 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:00.105170 5916 cpu_manager.go:414] "containerMap: removing container" podUID="c48449eb-30cb-4b07-892a-b27192f49e0f" containerName="util" Nov 02 06:21:00.105242 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:00.105175 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c48449eb-30cb-4b07-892a-b27192f49e0f" containerName="util" Nov 02 06:21:00.105242 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:00.105183 5916 cpu_manager.go:414] "containerMap: removing container" podUID="a7ef2c80-8b0c-4c7a-a5c1-ddb795124cbb" containerName="pull" Nov 02 06:21:00.105242 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:00.105187 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7ef2c80-8b0c-4c7a-a5c1-ddb795124cbb" containerName="pull" Nov 02 06:21:00.105242 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:00.105193 5916 cpu_manager.go:414] "containerMap: removing container" podUID="d93a0ea2-39a5-4bfa-b4c9-46df030a2dad" containerName="extract" Nov 02 06:21:00.105242 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:00.105197 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d93a0ea2-39a5-4bfa-b4c9-46df030a2dad" containerName="extract" Nov 02 06:21:00.105242 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:00.105204 5916 cpu_manager.go:414] "containerMap: removing container" podUID="ef4df80a-7c4f-46fd-9cfe-5ba25a89f869" containerName="extract" Nov 02 06:21:00.105242 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:00.105208 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef4df80a-7c4f-46fd-9cfe-5ba25a89f869" containerName="extract" Nov 02 06:21:00.105242 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:00.105216 5916 cpu_manager.go:414] "containerMap: removing container" podUID="a7ef2c80-8b0c-4c7a-a5c1-ddb795124cbb" containerName="extract" Nov 02 06:21:00.105242 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:00.105220 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7ef2c80-8b0c-4c7a-a5c1-ddb795124cbb" containerName="extract" Nov 02 06:21:00.105242 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:00.105228 5916 cpu_manager.go:414] "containerMap: removing container" podUID="05e071cb-0da9-4fff-9b10-5ab9b325b1c6" containerName="util" Nov 02 06:21:00.105242 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:00.105235 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05e071cb-0da9-4fff-9b10-5ab9b325b1c6" containerName="util" Nov 02 06:21:00.105242 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:00.105244 5916 cpu_manager.go:414] "containerMap: removing container" podUID="d93a0ea2-39a5-4bfa-b4c9-46df030a2dad" containerName="pull" Nov 02 06:21:00.105242 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:00.105252 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d93a0ea2-39a5-4bfa-b4c9-46df030a2dad" containerName="pull" Nov 02 06:21:00.106048 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:00.105260 5916 cpu_manager.go:414] "containerMap: removing container" podUID="999d5daf-1bbf-46df-8184-e3454fae166b" containerName="pull" Nov 02 06:21:00.106048 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:00.105264 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="999d5daf-1bbf-46df-8184-e3454fae166b" containerName="pull" Nov 02 06:21:00.106048 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:00.105270 5916 cpu_manager.go:414] "containerMap: removing container" podUID="368fe2a8-af6b-4d60-a6cf-6a82dff993a0" containerName="extract" Nov 02 06:21:00.106048 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:00.105275 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="368fe2a8-af6b-4d60-a6cf-6a82dff993a0" containerName="extract" Nov 02 06:21:00.106048 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:00.105281 5916 cpu_manager.go:414] "containerMap: removing container" podUID="48b3fc06-ab52-4489-8e5d-84da4514f2da" containerName="util" Nov 02 06:21:00.106048 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:00.105285 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48b3fc06-ab52-4489-8e5d-84da4514f2da" containerName="util" Nov 02 06:21:00.106048 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:00.105293 5916 cpu_manager.go:414] "containerMap: removing container" podUID="999d5daf-1bbf-46df-8184-e3454fae166b" containerName="extract" Nov 02 06:21:00.106048 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:00.105296 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="999d5daf-1bbf-46df-8184-e3454fae166b" containerName="extract" Nov 02 06:21:00.106048 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:00.105307 5916 cpu_manager.go:414] "containerMap: removing container" podUID="48b3fc06-ab52-4489-8e5d-84da4514f2da" containerName="pull" Nov 02 06:21:00.106048 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:00.105311 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48b3fc06-ab52-4489-8e5d-84da4514f2da" containerName="pull" Nov 02 06:21:00.106048 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:00.105320 5916 cpu_manager.go:414] "containerMap: removing container" podUID="aa5d2e0d-5b30-415e-86b0-d3a7e246ffa7" containerName="extract" Nov 02 06:21:00.106048 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:00.105326 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa5d2e0d-5b30-415e-86b0-d3a7e246ffa7" containerName="extract" Nov 02 06:21:00.106048 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:00.105333 5916 cpu_manager.go:414] "containerMap: removing container" podUID="999d5daf-1bbf-46df-8184-e3454fae166b" containerName="util" Nov 02 06:21:00.106048 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:00.105337 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="999d5daf-1bbf-46df-8184-e3454fae166b" containerName="util" Nov 02 06:21:00.106048 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:00.105344 5916 cpu_manager.go:414] "containerMap: removing container" podUID="aa5d2e0d-5b30-415e-86b0-d3a7e246ffa7" containerName="util" Nov 02 06:21:00.106048 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:00.105349 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa5d2e0d-5b30-415e-86b0-d3a7e246ffa7" containerName="util" Nov 02 06:21:00.106048 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:00.105353 5916 cpu_manager.go:414] "containerMap: removing container" podUID="70e3b8eb-ed1f-4552-8caf-113f26180ee5" containerName="pull" Nov 02 06:21:00.106048 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:00.105357 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70e3b8eb-ed1f-4552-8caf-113f26180ee5" containerName="pull" Nov 02 06:21:00.106048 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:00.105363 5916 cpu_manager.go:414] "containerMap: removing container" podUID="5a659a95-82ea-47c7-937d-17b4407d1eb2" containerName="extract" Nov 02 06:21:00.106048 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:00.105367 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a659a95-82ea-47c7-937d-17b4407d1eb2" containerName="extract" Nov 02 06:21:00.106048 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:00.105375 5916 cpu_manager.go:414] "containerMap: removing container" podUID="70e3b8eb-ed1f-4552-8caf-113f26180ee5" containerName="extract" Nov 02 06:21:00.106048 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:00.105379 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70e3b8eb-ed1f-4552-8caf-113f26180ee5" containerName="extract" Nov 02 06:21:00.106048 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:00.105384 5916 cpu_manager.go:414] "containerMap: removing container" podUID="d93a0ea2-39a5-4bfa-b4c9-46df030a2dad" containerName="util" Nov 02 06:21:00.106048 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:00.105388 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d93a0ea2-39a5-4bfa-b4c9-46df030a2dad" containerName="util" Nov 02 06:21:00.106048 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:00.105395 5916 cpu_manager.go:414] "containerMap: removing container" podUID="5a659a95-82ea-47c7-937d-17b4407d1eb2" containerName="util" Nov 02 06:21:00.106048 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:00.105402 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a659a95-82ea-47c7-937d-17b4407d1eb2" containerName="util" Nov 02 06:21:00.106048 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:00.105409 5916 cpu_manager.go:414] "containerMap: removing container" podUID="c48449eb-30cb-4b07-892a-b27192f49e0f" containerName="pull" Nov 02 06:21:00.106048 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:00.105414 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c48449eb-30cb-4b07-892a-b27192f49e0f" containerName="pull" Nov 02 06:21:00.106048 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:00.105419 5916 cpu_manager.go:414] "containerMap: removing container" podUID="5a659a95-82ea-47c7-937d-17b4407d1eb2" containerName="pull" Nov 02 06:21:00.106048 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:00.105424 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a659a95-82ea-47c7-937d-17b4407d1eb2" containerName="pull" Nov 02 06:21:00.106048 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:00.105431 5916 cpu_manager.go:414] "containerMap: removing container" podUID="a7ef2c80-8b0c-4c7a-a5c1-ddb795124cbb" containerName="util" Nov 02 06:21:00.106048 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:00.105436 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7ef2c80-8b0c-4c7a-a5c1-ddb795124cbb" containerName="util" Nov 02 06:21:00.106048 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:00.105444 5916 cpu_manager.go:414] "containerMap: removing container" podUID="70e3b8eb-ed1f-4552-8caf-113f26180ee5" containerName="util" Nov 02 06:21:00.106048 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:00.105449 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70e3b8eb-ed1f-4552-8caf-113f26180ee5" containerName="util" Nov 02 06:21:00.106048 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:00.105454 5916 cpu_manager.go:414] "containerMap: removing container" podUID="ef4df80a-7c4f-46fd-9cfe-5ba25a89f869" containerName="pull" Nov 02 06:21:00.106048 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:00.105458 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef4df80a-7c4f-46fd-9cfe-5ba25a89f869" containerName="pull" Nov 02 06:21:00.106048 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:00.105465 5916 cpu_manager.go:414] "containerMap: removing container" podUID="aa5d2e0d-5b30-415e-86b0-d3a7e246ffa7" containerName="pull" Nov 02 06:21:00.106048 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:00.105469 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa5d2e0d-5b30-415e-86b0-d3a7e246ffa7" containerName="pull" Nov 02 06:21:00.106048 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:00.105480 5916 cpu_manager.go:414] "containerMap: removing container" podUID="368fe2a8-af6b-4d60-a6cf-6a82dff993a0" containerName="util" Nov 02 06:21:00.106048 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:00.105490 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="368fe2a8-af6b-4d60-a6cf-6a82dff993a0" containerName="util" Nov 02 06:21:00.106048 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:00.105538 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a659a95-82ea-47c7-937d-17b4407d1eb2" containerName="extract" Nov 02 06:21:00.106048 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:00.105544 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="70e3b8eb-ed1f-4552-8caf-113f26180ee5" containerName="extract" Nov 02 06:21:00.106048 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:00.105553 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="48b3fc06-ab52-4489-8e5d-84da4514f2da" containerName="extract" Nov 02 06:21:00.106048 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:00.105561 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="05e071cb-0da9-4fff-9b10-5ab9b325b1c6" containerName="extract" Nov 02 06:21:00.106048 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:00.105569 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="c48449eb-30cb-4b07-892a-b27192f49e0f" containerName="extract" Nov 02 06:21:00.106048 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:00.105575 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="d93a0ea2-39a5-4bfa-b4c9-46df030a2dad" containerName="extract" Nov 02 06:21:00.106048 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:00.105580 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef4df80a-7c4f-46fd-9cfe-5ba25a89f869" containerName="extract" Nov 02 06:21:00.106048 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:00.105584 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7ef2c80-8b0c-4c7a-a5c1-ddb795124cbb" containerName="extract" Nov 02 06:21:00.106048 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:00.105589 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="368fe2a8-af6b-4d60-a6cf-6a82dff993a0" containerName="extract" Nov 02 06:21:00.106048 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:00.105595 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="999d5daf-1bbf-46df-8184-e3454fae166b" containerName="extract" Nov 02 06:21:00.106048 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:00.105600 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa5d2e0d-5b30-415e-86b0-d3a7e246ffa7" containerName="extract" Nov 02 06:21:00.109901 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:00.109883 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-565d9484c7-jpgdm" Nov 02 06:21:00.115767 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:00.115744 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/298877e8-24cc-4e94-bb69-4258662fd1cf-manifests\") pod \"opendatahub-operator-controller-manager-565d9484c7-jpgdm\" (UID: \"298877e8-24cc-4e94-bb69-4258662fd1cf\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-565d9484c7-jpgdm" Nov 02 06:21:00.115947 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:00.115829 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4f74f\" (UniqueName: \"kubernetes.io/projected/298877e8-24cc-4e94-bb69-4258662fd1cf-kube-api-access-4f74f\") pod \"opendatahub-operator-controller-manager-565d9484c7-jpgdm\" (UID: \"298877e8-24cc-4e94-bb69-4258662fd1cf\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-565d9484c7-jpgdm" Nov 02 06:21:00.115947 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:00.115884 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/298877e8-24cc-4e94-bb69-4258662fd1cf-metrics-cert\") pod \"opendatahub-operator-controller-manager-565d9484c7-jpgdm\" (UID: \"298877e8-24cc-4e94-bb69-4258662fd1cf\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-565d9484c7-jpgdm" Nov 02 06:21:00.116121 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:00.115985 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/298877e8-24cc-4e94-bb69-4258662fd1cf-webhook-cert\") pod \"opendatahub-operator-controller-manager-565d9484c7-jpgdm\" (UID: \"298877e8-24cc-4e94-bb69-4258662fd1cf\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-565d9484c7-jpgdm" Nov 02 06:21:00.116121 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:00.116038 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/298877e8-24cc-4e94-bb69-4258662fd1cf-tmp\") pod \"opendatahub-operator-controller-manager-565d9484c7-jpgdm\" (UID: \"298877e8-24cc-4e94-bb69-4258662fd1cf\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-565d9484c7-jpgdm" Nov 02 06:21:00.116545 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:00.116155 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/298877e8-24cc-4e94-bb69-4258662fd1cf-apiservice-cert\") pod \"opendatahub-operator-controller-manager-565d9484c7-jpgdm\" (UID: \"298877e8-24cc-4e94-bb69-4258662fd1cf\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-565d9484c7-jpgdm" Nov 02 06:21:00.158678 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:00.158643 5916 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Nov 02 06:21:00.158939 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:00.158740 5916 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Nov 02 06:21:00.159125 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:00.159108 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Nov 02 06:21:00.159177 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:00.159143 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Nov 02 06:21:00.204791 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:00.204757 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-5hv8f\"" Nov 02 06:21:00.204791 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:00.204772 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Nov 02 06:21:00.208683 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:00.208657 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-565d9484c7-jpgdm"] Nov 02 06:21:00.217515 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:00.217490 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/298877e8-24cc-4e94-bb69-4258662fd1cf-metrics-cert\") pod \"opendatahub-operator-controller-manager-565d9484c7-jpgdm\" (UID: \"298877e8-24cc-4e94-bb69-4258662fd1cf\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-565d9484c7-jpgdm" Nov 02 06:21:00.217626 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:00.217530 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/298877e8-24cc-4e94-bb69-4258662fd1cf-webhook-cert\") pod \"opendatahub-operator-controller-manager-565d9484c7-jpgdm\" (UID: \"298877e8-24cc-4e94-bb69-4258662fd1cf\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-565d9484c7-jpgdm" Nov 02 06:21:00.217626 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:00.217566 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/298877e8-24cc-4e94-bb69-4258662fd1cf-tmp\") pod \"opendatahub-operator-controller-manager-565d9484c7-jpgdm\" (UID: \"298877e8-24cc-4e94-bb69-4258662fd1cf\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-565d9484c7-jpgdm" Nov 02 06:21:00.217626 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:00.217597 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/298877e8-24cc-4e94-bb69-4258662fd1cf-apiservice-cert\") pod \"opendatahub-operator-controller-manager-565d9484c7-jpgdm\" (UID: \"298877e8-24cc-4e94-bb69-4258662fd1cf\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-565d9484c7-jpgdm" Nov 02 06:21:00.217782 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:00.217687 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/298877e8-24cc-4e94-bb69-4258662fd1cf-manifests\") pod \"opendatahub-operator-controller-manager-565d9484c7-jpgdm\" (UID: \"298877e8-24cc-4e94-bb69-4258662fd1cf\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-565d9484c7-jpgdm" Nov 02 06:21:00.217782 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:00.217712 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4f74f\" (UniqueName: \"kubernetes.io/projected/298877e8-24cc-4e94-bb69-4258662fd1cf-kube-api-access-4f74f\") pod \"opendatahub-operator-controller-manager-565d9484c7-jpgdm\" (UID: \"298877e8-24cc-4e94-bb69-4258662fd1cf\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-565d9484c7-jpgdm" Nov 02 06:21:00.218174 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:00.218148 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/298877e8-24cc-4e94-bb69-4258662fd1cf-manifests\") pod \"opendatahub-operator-controller-manager-565d9484c7-jpgdm\" (UID: \"298877e8-24cc-4e94-bb69-4258662fd1cf\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-565d9484c7-jpgdm" Nov 02 06:21:00.220183 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:00.220162 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/298877e8-24cc-4e94-bb69-4258662fd1cf-webhook-cert\") pod \"opendatahub-operator-controller-manager-565d9484c7-jpgdm\" (UID: \"298877e8-24cc-4e94-bb69-4258662fd1cf\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-565d9484c7-jpgdm" Nov 02 06:21:00.220295 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:00.220209 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/298877e8-24cc-4e94-bb69-4258662fd1cf-metrics-cert\") pod \"opendatahub-operator-controller-manager-565d9484c7-jpgdm\" (UID: \"298877e8-24cc-4e94-bb69-4258662fd1cf\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-565d9484c7-jpgdm" Nov 02 06:21:00.220428 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:00.220407 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/298877e8-24cc-4e94-bb69-4258662fd1cf-tmp\") pod \"opendatahub-operator-controller-manager-565d9484c7-jpgdm\" (UID: \"298877e8-24cc-4e94-bb69-4258662fd1cf\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-565d9484c7-jpgdm" Nov 02 06:21:00.220481 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:00.220463 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/298877e8-24cc-4e94-bb69-4258662fd1cf-apiservice-cert\") pod \"opendatahub-operator-controller-manager-565d9484c7-jpgdm\" (UID: \"298877e8-24cc-4e94-bb69-4258662fd1cf\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-565d9484c7-jpgdm" Nov 02 06:21:00.271721 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:00.271692 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4f74f\" (UniqueName: \"kubernetes.io/projected/298877e8-24cc-4e94-bb69-4258662fd1cf-kube-api-access-4f74f\") pod \"opendatahub-operator-controller-manager-565d9484c7-jpgdm\" (UID: \"298877e8-24cc-4e94-bb69-4258662fd1cf\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-565d9484c7-jpgdm" Nov 02 06:21:00.425886 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:00.425850 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-565d9484c7-jpgdm" Nov 02 06:21:00.910188 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:00.910157 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-565d9484c7-jpgdm"] Nov 02 06:21:00.917233 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:21:00.917196 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod298877e8_24cc_4e94_bb69_4258662fd1cf.slice/crio-46fb6458f739aec7e402ec2f75f281f2a145cf371970499c71e1880d1260e478 WatchSource:0}: Error finding container 46fb6458f739aec7e402ec2f75f281f2a145cf371970499c71e1880d1260e478: Status 404 returned error can't find the container with id 46fb6458f739aec7e402ec2f75f281f2a145cf371970499c71e1880d1260e478 Nov 02 06:21:01.471981 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:01.471940 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-565d9484c7-jpgdm" event={"ID":"298877e8-24cc-4e94-bb69-4258662fd1cf","Type":"ContainerStarted","Data":"46fb6458f739aec7e402ec2f75f281f2a145cf371970499c71e1880d1260e478"} Nov 02 06:21:07.502750 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:07.502714 5916 generic.go:356] "Generic (PLEG): container finished" podID="298877e8-24cc-4e94-bb69-4258662fd1cf" containerID="87becee3be883282ff75b04c89c5567bc589d354f60f26a6f5cdcf33ac4d83fd" exitCode=0 Nov 02 06:21:07.503168 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:07.502802 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-565d9484c7-jpgdm" event={"ID":"298877e8-24cc-4e94-bb69-4258662fd1cf","Type":"ContainerDied","Data":"87becee3be883282ff75b04c89c5567bc589d354f60f26a6f5cdcf33ac4d83fd"} Nov 02 06:21:08.509020 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:08.508988 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-565d9484c7-jpgdm" event={"ID":"298877e8-24cc-4e94-bb69-4258662fd1cf","Type":"ContainerStarted","Data":"f3eb3e1aa6b8bf5f2d1494e751f8b16881ae9864517f63ce7e65b82a0c4d2ede"} Nov 02 06:21:08.509409 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:08.509124 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-565d9484c7-jpgdm" Nov 02 06:21:08.722913 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:08.722844 5916 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-565d9484c7-jpgdm" podStartSLOduration=2.869682574 podStartE2EDuration="9.722828934s" totalImagesPullingTime="6.379425279s" totalInitContainerRuntime="473.721081ms" isStatefulPod=false podCreationTimestamp="2026-11-02 06:20:59 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:21:08.675606915 +0000 UTC m=+256.947427310" watchObservedRunningTime="2026-11-02 06:21:08.722828934 +0000 UTC m=+256.994649327" Nov 02 06:21:11.624207 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:11.624165 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-8c429"] Nov 02 06:21:11.627770 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:11.627748 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-8c429" Nov 02 06:21:11.676121 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:11.676098 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-8c429"] Nov 02 06:21:11.720334 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:11.720307 5916 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 02 06:21:11.720477 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:11.720316 5916 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 02 06:21:11.720477 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:11.720351 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-4lbcv\"" Nov 02 06:21:11.720562 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:11.720474 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8eea0eab-e7ff-4a11-a7e7-9b25f7830221-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-8c429\" (UID: \"8eea0eab-e7ff-4a11-a7e7-9b25f7830221\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-8c429" Nov 02 06:21:11.720562 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:11.720510 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pwkw\" (UniqueName: \"kubernetes.io/projected/8eea0eab-e7ff-4a11-a7e7-9b25f7830221-kube-api-access-7pwkw\") pod \"kuadrant-operator-controller-manager-cf8469d7b-8c429\" (UID: \"8eea0eab-e7ff-4a11-a7e7-9b25f7830221\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-8c429" Nov 02 06:21:11.821600 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:11.821571 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8eea0eab-e7ff-4a11-a7e7-9b25f7830221-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-8c429\" (UID: \"8eea0eab-e7ff-4a11-a7e7-9b25f7830221\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-8c429" Nov 02 06:21:11.821716 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:11.821608 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pwkw\" (UniqueName: \"kubernetes.io/projected/8eea0eab-e7ff-4a11-a7e7-9b25f7830221-kube-api-access-7pwkw\") pod \"kuadrant-operator-controller-manager-cf8469d7b-8c429\" (UID: \"8eea0eab-e7ff-4a11-a7e7-9b25f7830221\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-8c429" Nov 02 06:21:11.821966 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:11.821947 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8eea0eab-e7ff-4a11-a7e7-9b25f7830221-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-8c429\" (UID: \"8eea0eab-e7ff-4a11-a7e7-9b25f7830221\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-8c429" Nov 02 06:21:11.878144 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:11.878079 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pwkw\" (UniqueName: \"kubernetes.io/projected/8eea0eab-e7ff-4a11-a7e7-9b25f7830221-kube-api-access-7pwkw\") pod \"kuadrant-operator-controller-manager-cf8469d7b-8c429\" (UID: \"8eea0eab-e7ff-4a11-a7e7-9b25f7830221\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-8c429" Nov 02 06:21:11.940159 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:11.940128 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-8c429" Nov 02 06:21:12.364619 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:12.364586 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-8c429"] Nov 02 06:21:12.372158 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:21:12.372133 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8eea0eab_e7ff_4a11_a7e7_9b25f7830221.slice/crio-91fb958cbfd914c763d9ab85d9d20bda1ea356f8da1a4e12205546a008ca2894 WatchSource:0}: Error finding container 91fb958cbfd914c763d9ab85d9d20bda1ea356f8da1a4e12205546a008ca2894: Status 404 returned error can't find the container with id 91fb958cbfd914c763d9ab85d9d20bda1ea356f8da1a4e12205546a008ca2894 Nov 02 06:21:12.528071 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:12.528033 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-8c429" event={"ID":"8eea0eab-e7ff-4a11-a7e7-9b25f7830221","Type":"ContainerStarted","Data":"91fb958cbfd914c763d9ab85d9d20bda1ea356f8da1a4e12205546a008ca2894"} Nov 02 06:21:16.211974 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:16.211936 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-d4659"] Nov 02 06:21:16.241454 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:16.241425 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-d4659" Nov 02 06:21:16.275531 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:16.275499 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-d4659"] Nov 02 06:21:16.319750 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:16.319710 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-t8qlr\"" Nov 02 06:21:16.360779 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:16.360735 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spdpl\" (UniqueName: \"kubernetes.io/projected/8cad89fe-40eb-443c-95e6-d4ae30c800e4-kube-api-access-spdpl\") pod \"limitador-operator-controller-manager-5778bb4449-d4659\" (UID: \"8cad89fe-40eb-443c-95e6-d4ae30c800e4\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-d4659" Nov 02 06:21:16.461948 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:16.461877 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spdpl\" (UniqueName: \"kubernetes.io/projected/8cad89fe-40eb-443c-95e6-d4ae30c800e4-kube-api-access-spdpl\") pod \"limitador-operator-controller-manager-5778bb4449-d4659\" (UID: \"8cad89fe-40eb-443c-95e6-d4ae30c800e4\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-d4659" Nov 02 06:21:16.519406 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:16.519335 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spdpl\" (UniqueName: \"kubernetes.io/projected/8cad89fe-40eb-443c-95e6-d4ae30c800e4-kube-api-access-spdpl\") pod \"limitador-operator-controller-manager-5778bb4449-d4659\" (UID: \"8cad89fe-40eb-443c-95e6-d4ae30c800e4\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-d4659" Nov 02 06:21:16.572885 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:16.572854 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-d4659" Nov 02 06:21:17.014622 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:17.014590 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-d4659"] Nov 02 06:21:17.021473 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:21:17.021448 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cad89fe_40eb_443c_95e6_d4ae30c800e4.slice/crio-629d9059613373b1501e0de091182c43064623553ee251cc202a85b435e989c9 WatchSource:0}: Error finding container 629d9059613373b1501e0de091182c43064623553ee251cc202a85b435e989c9: Status 404 returned error can't find the container with id 629d9059613373b1501e0de091182c43064623553ee251cc202a85b435e989c9 Nov 02 06:21:17.559365 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:17.559321 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-d4659" event={"ID":"8cad89fe-40eb-443c-95e6-d4ae30c800e4","Type":"ContainerStarted","Data":"629d9059613373b1501e0de091182c43064623553ee251cc202a85b435e989c9"} Nov 02 06:21:18.634410 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:18.634372 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-79vf9"] Nov 02 06:21:18.638194 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:18.638173 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-79vf9" Nov 02 06:21:18.686841 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:18.686812 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-79vf9"] Nov 02 06:21:18.730644 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:18.730614 5916 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Nov 02 06:21:18.730820 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:18.730669 5916 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Nov 02 06:21:18.730820 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:18.730669 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-df2vf\"" Nov 02 06:21:18.783479 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:18.783446 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qhwl\" (UniqueName: \"kubernetes.io/projected/c0eaf282-9bc4-4a39-90d1-c82555ac0951-kube-api-access-2qhwl\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-79vf9\" (UID: \"c0eaf282-9bc4-4a39-90d1-c82555ac0951\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-79vf9" Nov 02 06:21:18.783678 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:18.783573 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c0eaf282-9bc4-4a39-90d1-c82555ac0951-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-79vf9\" (UID: \"c0eaf282-9bc4-4a39-90d1-c82555ac0951\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-79vf9" Nov 02 06:21:18.884625 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:18.884541 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c0eaf282-9bc4-4a39-90d1-c82555ac0951-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-79vf9\" (UID: \"c0eaf282-9bc4-4a39-90d1-c82555ac0951\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-79vf9" Nov 02 06:21:18.884625 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:18.884603 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qhwl\" (UniqueName: \"kubernetes.io/projected/c0eaf282-9bc4-4a39-90d1-c82555ac0951-kube-api-access-2qhwl\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-79vf9\" (UID: \"c0eaf282-9bc4-4a39-90d1-c82555ac0951\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-79vf9" Nov 02 06:21:18.885031 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:18.885013 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c0eaf282-9bc4-4a39-90d1-c82555ac0951-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-79vf9\" (UID: \"c0eaf282-9bc4-4a39-90d1-c82555ac0951\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-79vf9" Nov 02 06:21:18.942343 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:18.942310 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qhwl\" (UniqueName: \"kubernetes.io/projected/c0eaf282-9bc4-4a39-90d1-c82555ac0951-kube-api-access-2qhwl\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-79vf9\" (UID: \"c0eaf282-9bc4-4a39-90d1-c82555ac0951\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-79vf9" Nov 02 06:21:18.950115 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:18.950086 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-79vf9" Nov 02 06:21:19.373692 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:19.373612 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-79vf9"] Nov 02 06:21:19.380650 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:21:19.380625 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0eaf282_9bc4_4a39_90d1_c82555ac0951.slice/crio-dac4292b8f427b1ad775ec4d3e526f4b4769fdf297189de38713bbb39a67edd0 WatchSource:0}: Error finding container dac4292b8f427b1ad775ec4d3e526f4b4769fdf297189de38713bbb39a67edd0: Status 404 returned error can't find the container with id dac4292b8f427b1ad775ec4d3e526f4b4769fdf297189de38713bbb39a67edd0 Nov 02 06:21:19.516402 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:19.516376 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-565d9484c7-jpgdm" Nov 02 06:21:19.573774 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:19.573726 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-79vf9" event={"ID":"c0eaf282-9bc4-4a39-90d1-c82555ac0951","Type":"ContainerStarted","Data":"dac4292b8f427b1ad775ec4d3e526f4b4769fdf297189de38713bbb39a67edd0"} Nov 02 06:21:19.894396 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:21:19.894362 5916 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p98cq" containerName="extractor" Nov 02 06:21:20.587061 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:20.587025 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-lg9rv"] Nov 02 06:21:20.590231 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:20.590213 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-lg9rv" Nov 02 06:21:20.641547 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:20.641515 5916 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Nov 02 06:21:20.641716 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:20.641571 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-pjcsm\"" Nov 02 06:21:20.650149 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:20.650122 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-lg9rv"] Nov 02 06:21:20.699886 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:20.699851 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4przv\" (UniqueName: \"kubernetes.io/projected/018f84ec-02be-4b19-8471-01b899a3b9b7-kube-api-access-4przv\") pod \"dns-operator-controller-manager-8557cbdb84-lg9rv\" (UID: \"018f84ec-02be-4b19-8471-01b899a3b9b7\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-lg9rv" Nov 02 06:21:20.800571 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:20.800535 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4przv\" (UniqueName: \"kubernetes.io/projected/018f84ec-02be-4b19-8471-01b899a3b9b7-kube-api-access-4przv\") pod \"dns-operator-controller-manager-8557cbdb84-lg9rv\" (UID: \"018f84ec-02be-4b19-8471-01b899a3b9b7\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-lg9rv" Nov 02 06:21:20.854267 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:20.854199 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4przv\" (UniqueName: \"kubernetes.io/projected/018f84ec-02be-4b19-8471-01b899a3b9b7-kube-api-access-4przv\") pod \"dns-operator-controller-manager-8557cbdb84-lg9rv\" (UID: \"018f84ec-02be-4b19-8471-01b899a3b9b7\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-lg9rv" Nov 02 06:21:20.903945 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:20.903897 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-lg9rv" Nov 02 06:21:21.279308 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:21.279273 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-lg9rv"] Nov 02 06:21:21.285659 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:21:21.285630 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod018f84ec_02be_4b19_8471_01b899a3b9b7.slice/crio-c7b76d8a7a9897367154c1b44c0d9acef06d0657b95be0b12d93233cfb611bfc WatchSource:0}: Error finding container c7b76d8a7a9897367154c1b44c0d9acef06d0657b95be0b12d93233cfb611bfc: Status 404 returned error can't find the container with id c7b76d8a7a9897367154c1b44c0d9acef06d0657b95be0b12d93233cfb611bfc Nov 02 06:21:21.587333 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:21.587241 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-lg9rv" event={"ID":"018f84ec-02be-4b19-8471-01b899a3b9b7","Type":"ContainerStarted","Data":"c7b76d8a7a9897367154c1b44c0d9acef06d0657b95be0b12d93233cfb611bfc"} Nov 02 06:21:23.600639 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:23.600544 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-79vf9" event={"ID":"c0eaf282-9bc4-4a39-90d1-c82555ac0951","Type":"ContainerStarted","Data":"49f093c1743d9eddfc9257e1905e69b2984150af54a68499999fb06039d3baaa"} Nov 02 06:21:23.603178 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:23.602683 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-d4659" event={"ID":"8cad89fe-40eb-443c-95e6-d4ae30c800e4","Type":"ContainerStarted","Data":"d1317b64c4369c70b86bc08acaa9d566c57d84bcabb74a3255b2b958a2359613"} Nov 02 06:21:23.603178 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:23.602821 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-d4659" Nov 02 06:21:23.803162 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:23.803113 5916 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-79vf9" podStartSLOduration=1.971068869 podStartE2EDuration="5.803098125s" totalImagesPullingTime="3.832029256s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:21:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:21:23.757011958 +0000 UTC m=+272.028832351" watchObservedRunningTime="2026-11-02 06:21:23.803098125 +0000 UTC m=+272.074918513" Nov 02 06:21:23.870874 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:23.870751 5916 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-d4659" podStartSLOduration=1.6742555449999998 podStartE2EDuration="7.870730973s" totalImagesPullingTime="6.196475428s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:21:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:21:23.869945562 +0000 UTC m=+272.141765959" watchObservedRunningTime="2026-11-02 06:21:23.870730973 +0000 UTC m=+272.142551368" Nov 02 06:21:24.609981 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:24.609878 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-8c429" event={"ID":"8eea0eab-e7ff-4a11-a7e7-9b25f7830221","Type":"ContainerStarted","Data":"15feed1b417a13df68a040080728d23213b5e2ad1849fa6a56a90047663fb321"} Nov 02 06:21:24.610385 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:24.610269 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-8c429" Nov 02 06:21:24.765467 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:24.765408 5916 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-8c429" podStartSLOduration=2.28591159 podStartE2EDuration="13.765396751s" totalImagesPullingTime="11.479485161s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:21:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:21:24.764436497 +0000 UTC m=+273.036256891" watchObservedRunningTime="2026-11-02 06:21:24.765396751 +0000 UTC m=+273.037217142" Nov 02 06:21:24.835507 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:24.835474 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-ztxsk"] Nov 02 06:21:24.839536 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:24.839509 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-ztxsk" Nov 02 06:21:24.891846 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:24.891813 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-ztxsk"] Nov 02 06:21:24.893462 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:24.893442 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-458fz\"" Nov 02 06:21:24.942525 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:24.942488 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-js2vm\" (UniqueName: \"kubernetes.io/projected/93c99741-8f15-41ce-982c-2ddd87bb9608-kube-api-access-js2vm\") pod \"authorino-operator-7b7bb4bc77-ztxsk\" (UID: \"93c99741-8f15-41ce-982c-2ddd87bb9608\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-ztxsk" Nov 02 06:21:25.043149 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:25.043111 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-js2vm\" (UniqueName: \"kubernetes.io/projected/93c99741-8f15-41ce-982c-2ddd87bb9608-kube-api-access-js2vm\") pod \"authorino-operator-7b7bb4bc77-ztxsk\" (UID: \"93c99741-8f15-41ce-982c-2ddd87bb9608\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-ztxsk" Nov 02 06:21:25.099070 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:25.099039 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-js2vm\" (UniqueName: \"kubernetes.io/projected/93c99741-8f15-41ce-982c-2ddd87bb9608-kube-api-access-js2vm\") pod \"authorino-operator-7b7bb4bc77-ztxsk\" (UID: \"93c99741-8f15-41ce-982c-2ddd87bb9608\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-ztxsk" Nov 02 06:21:25.152058 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:25.151983 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-ztxsk" Nov 02 06:21:25.528092 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:25.527993 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-ztxsk"] Nov 02 06:21:25.534558 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:21:25.534527 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93c99741_8f15_41ce_982c_2ddd87bb9608.slice/crio-b72571d5d5b70643584f0170e8d99bc2534bdcbe02f333edbf9450653af6a5b5 WatchSource:0}: Error finding container b72571d5d5b70643584f0170e8d99bc2534bdcbe02f333edbf9450653af6a5b5: Status 404 returned error can't find the container with id b72571d5d5b70643584f0170e8d99bc2534bdcbe02f333edbf9450653af6a5b5 Nov 02 06:21:25.616667 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:25.616633 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-ztxsk" event={"ID":"93c99741-8f15-41ce-982c-2ddd87bb9608","Type":"ContainerStarted","Data":"b72571d5d5b70643584f0170e8d99bc2534bdcbe02f333edbf9450653af6a5b5"} Nov 02 06:21:28.634820 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:28.634784 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-lg9rv" event={"ID":"018f84ec-02be-4b19-8471-01b899a3b9b7","Type":"ContainerStarted","Data":"056fad81ee641373caf4e5dc36f18ff6dd13426e2c54af3551d24199d732eb8b"} Nov 02 06:21:28.635313 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:28.634852 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-lg9rv" Nov 02 06:21:28.743161 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:28.743103 5916 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-lg9rv" podStartSLOduration=1.954687898 podStartE2EDuration="8.743084043s" totalImagesPullingTime="6.788396145s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:21:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:21:28.740791398 +0000 UTC m=+277.012611791" watchObservedRunningTime="2026-11-02 06:21:28.743084043 +0000 UTC m=+277.014904438" Nov 02 06:21:29.646422 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:29.646373 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-ztxsk" event={"ID":"93c99741-8f15-41ce-982c-2ddd87bb9608","Type":"ContainerStarted","Data":"2a2981ca7acde8b9d1f96ee35ccd9584c70457072aca200130fc64749f308d50"} Nov 02 06:21:29.646851 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:29.646621 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-ztxsk" Nov 02 06:21:29.753642 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:29.753592 5916 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7b7bb4bc77-ztxsk" podStartSLOduration=2.395049242 podStartE2EDuration="5.75357762s" totalImagesPullingTime="3.358528378s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:21:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:21:29.75232977 +0000 UTC m=+278.024150166" watchObservedRunningTime="2026-11-02 06:21:29.75357762 +0000 UTC m=+278.025398012" Nov 02 06:21:34.613039 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:34.612953 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-d4659" Nov 02 06:21:35.618593 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:35.618564 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-8c429" Nov 02 06:21:36.060680 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:36.060633 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-85d774877-2mfp5"] Nov 02 06:21:36.064738 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:36.064718 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d774877-2mfp5" Nov 02 06:21:36.113570 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:36.113543 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-r4m44\"" Nov 02 06:21:36.123753 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:36.123728 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-85d774877-2mfp5"] Nov 02 06:21:36.241020 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:36.240989 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3665b4c2-f4d2-4080-8613-c42857882641-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-85d774877-2mfp5\" (UID: \"3665b4c2-f4d2-4080-8613-c42857882641\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d774877-2mfp5" Nov 02 06:21:36.241167 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:36.241042 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3665b4c2-f4d2-4080-8613-c42857882641-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-85d774877-2mfp5\" (UID: \"3665b4c2-f4d2-4080-8613-c42857882641\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d774877-2mfp5" Nov 02 06:21:36.241167 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:36.241146 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3665b4c2-f4d2-4080-8613-c42857882641-istio-token\") pod \"data-science-gateway-data-science-gateway-class-85d774877-2mfp5\" (UID: \"3665b4c2-f4d2-4080-8613-c42857882641\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d774877-2mfp5" Nov 02 06:21:36.241294 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:36.241197 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3665b4c2-f4d2-4080-8613-c42857882641-istio-data\") pod \"data-science-gateway-data-science-gateway-class-85d774877-2mfp5\" (UID: \"3665b4c2-f4d2-4080-8613-c42857882641\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d774877-2mfp5" Nov 02 06:21:36.241357 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:36.241299 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3665b4c2-f4d2-4080-8613-c42857882641-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-85d774877-2mfp5\" (UID: \"3665b4c2-f4d2-4080-8613-c42857882641\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d774877-2mfp5" Nov 02 06:21:36.241357 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:36.241341 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3665b4c2-f4d2-4080-8613-c42857882641-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-85d774877-2mfp5\" (UID: \"3665b4c2-f4d2-4080-8613-c42857882641\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d774877-2mfp5" Nov 02 06:21:36.241426 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:36.241396 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3665b4c2-f4d2-4080-8613-c42857882641-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-85d774877-2mfp5\" (UID: \"3665b4c2-f4d2-4080-8613-c42857882641\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d774877-2mfp5" Nov 02 06:21:36.241462 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:36.241438 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hv7k7\" (UniqueName: \"kubernetes.io/projected/3665b4c2-f4d2-4080-8613-c42857882641-kube-api-access-hv7k7\") pod \"data-science-gateway-data-science-gateway-class-85d774877-2mfp5\" (UID: \"3665b4c2-f4d2-4080-8613-c42857882641\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d774877-2mfp5" Nov 02 06:21:36.241505 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:36.241464 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3665b4c2-f4d2-4080-8613-c42857882641-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-85d774877-2mfp5\" (UID: \"3665b4c2-f4d2-4080-8613-c42857882641\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d774877-2mfp5" Nov 02 06:21:36.312486 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:36.312405 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-hptdf"] Nov 02 06:21:36.316267 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:36.316245 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-hptdf" Nov 02 06:21:36.342125 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:36.342097 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3665b4c2-f4d2-4080-8613-c42857882641-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-85d774877-2mfp5\" (UID: \"3665b4c2-f4d2-4080-8613-c42857882641\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d774877-2mfp5" Nov 02 06:21:36.342247 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:36.342144 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3665b4c2-f4d2-4080-8613-c42857882641-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-85d774877-2mfp5\" (UID: \"3665b4c2-f4d2-4080-8613-c42857882641\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d774877-2mfp5" Nov 02 06:21:36.342247 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:36.342165 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3665b4c2-f4d2-4080-8613-c42857882641-istio-token\") pod \"data-science-gateway-data-science-gateway-class-85d774877-2mfp5\" (UID: \"3665b4c2-f4d2-4080-8613-c42857882641\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d774877-2mfp5" Nov 02 06:21:36.342247 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:36.342202 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3665b4c2-f4d2-4080-8613-c42857882641-istio-data\") pod \"data-science-gateway-data-science-gateway-class-85d774877-2mfp5\" (UID: \"3665b4c2-f4d2-4080-8613-c42857882641\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d774877-2mfp5" Nov 02 06:21:36.342247 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:36.342238 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3665b4c2-f4d2-4080-8613-c42857882641-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-85d774877-2mfp5\" (UID: \"3665b4c2-f4d2-4080-8613-c42857882641\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d774877-2mfp5" Nov 02 06:21:36.342430 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:36.342266 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3665b4c2-f4d2-4080-8613-c42857882641-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-85d774877-2mfp5\" (UID: \"3665b4c2-f4d2-4080-8613-c42857882641\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d774877-2mfp5" Nov 02 06:21:36.342430 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:36.342294 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3665b4c2-f4d2-4080-8613-c42857882641-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-85d774877-2mfp5\" (UID: \"3665b4c2-f4d2-4080-8613-c42857882641\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d774877-2mfp5" Nov 02 06:21:36.342430 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:36.342318 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hv7k7\" (UniqueName: \"kubernetes.io/projected/3665b4c2-f4d2-4080-8613-c42857882641-kube-api-access-hv7k7\") pod \"data-science-gateway-data-science-gateway-class-85d774877-2mfp5\" (UID: \"3665b4c2-f4d2-4080-8613-c42857882641\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d774877-2mfp5" Nov 02 06:21:36.342575 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:36.342411 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3665b4c2-f4d2-4080-8613-c42857882641-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-85d774877-2mfp5\" (UID: \"3665b4c2-f4d2-4080-8613-c42857882641\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d774877-2mfp5" Nov 02 06:21:36.342575 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:36.342504 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3665b4c2-f4d2-4080-8613-c42857882641-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-85d774877-2mfp5\" (UID: \"3665b4c2-f4d2-4080-8613-c42857882641\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d774877-2mfp5" Nov 02 06:21:36.342682 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:36.342660 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3665b4c2-f4d2-4080-8613-c42857882641-istio-data\") pod \"data-science-gateway-data-science-gateway-class-85d774877-2mfp5\" (UID: \"3665b4c2-f4d2-4080-8613-c42857882641\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d774877-2mfp5" Nov 02 06:21:36.342776 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:36.342754 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3665b4c2-f4d2-4080-8613-c42857882641-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-85d774877-2mfp5\" (UID: \"3665b4c2-f4d2-4080-8613-c42857882641\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d774877-2mfp5" Nov 02 06:21:36.342831 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:36.342779 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3665b4c2-f4d2-4080-8613-c42857882641-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-85d774877-2mfp5\" (UID: \"3665b4c2-f4d2-4080-8613-c42857882641\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d774877-2mfp5" Nov 02 06:21:36.343070 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:36.343053 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3665b4c2-f4d2-4080-8613-c42857882641-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-85d774877-2mfp5\" (UID: \"3665b4c2-f4d2-4080-8613-c42857882641\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d774877-2mfp5" Nov 02 06:21:36.344547 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:36.344526 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3665b4c2-f4d2-4080-8613-c42857882641-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-85d774877-2mfp5\" (UID: \"3665b4c2-f4d2-4080-8613-c42857882641\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d774877-2mfp5" Nov 02 06:21:36.344758 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:36.344743 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3665b4c2-f4d2-4080-8613-c42857882641-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-85d774877-2mfp5\" (UID: \"3665b4c2-f4d2-4080-8613-c42857882641\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d774877-2mfp5" Nov 02 06:21:36.376874 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:36.376851 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-hptdf"] Nov 02 06:21:36.397674 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:36.397645 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3665b4c2-f4d2-4080-8613-c42857882641-istio-token\") pod \"data-science-gateway-data-science-gateway-class-85d774877-2mfp5\" (UID: \"3665b4c2-f4d2-4080-8613-c42857882641\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d774877-2mfp5" Nov 02 06:21:36.397843 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:36.397822 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hv7k7\" (UniqueName: \"kubernetes.io/projected/3665b4c2-f4d2-4080-8613-c42857882641-kube-api-access-hv7k7\") pod \"data-science-gateway-data-science-gateway-class-85d774877-2mfp5\" (UID: \"3665b4c2-f4d2-4080-8613-c42857882641\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d774877-2mfp5" Nov 02 06:21:36.443305 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:36.443280 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4nmd\" (UniqueName: \"kubernetes.io/projected/0a731aff-55a4-45a9-8ce5-77dbcb2e7a04-kube-api-access-c4nmd\") pod \"kuadrant-operator-controller-manager-678d87766f-hptdf\" (UID: \"0a731aff-55a4-45a9-8ce5-77dbcb2e7a04\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-hptdf" Nov 02 06:21:36.443476 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:36.443353 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0a731aff-55a4-45a9-8ce5-77dbcb2e7a04-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-hptdf\" (UID: \"0a731aff-55a4-45a9-8ce5-77dbcb2e7a04\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-hptdf" Nov 02 06:21:36.544796 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:36.544762 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0a731aff-55a4-45a9-8ce5-77dbcb2e7a04-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-hptdf\" (UID: \"0a731aff-55a4-45a9-8ce5-77dbcb2e7a04\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-hptdf" Nov 02 06:21:36.545002 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:36.544829 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4nmd\" (UniqueName: \"kubernetes.io/projected/0a731aff-55a4-45a9-8ce5-77dbcb2e7a04-kube-api-access-c4nmd\") pod \"kuadrant-operator-controller-manager-678d87766f-hptdf\" (UID: \"0a731aff-55a4-45a9-8ce5-77dbcb2e7a04\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-hptdf" Nov 02 06:21:36.545233 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:36.545211 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0a731aff-55a4-45a9-8ce5-77dbcb2e7a04-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-hptdf\" (UID: \"0a731aff-55a4-45a9-8ce5-77dbcb2e7a04\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-hptdf" Nov 02 06:21:36.600961 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:36.600885 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4nmd\" (UniqueName: \"kubernetes.io/projected/0a731aff-55a4-45a9-8ce5-77dbcb2e7a04-kube-api-access-c4nmd\") pod \"kuadrant-operator-controller-manager-678d87766f-hptdf\" (UID: \"0a731aff-55a4-45a9-8ce5-77dbcb2e7a04\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-hptdf" Nov 02 06:21:36.628711 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:36.628676 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-hptdf" Nov 02 06:21:36.679356 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:36.678073 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d774877-2mfp5" Nov 02 06:21:37.106120 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:37.106084 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-85d774877-2mfp5"] Nov 02 06:21:37.107702 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:37.107654 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-hptdf"] Nov 02 06:21:37.112029 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:21:37.112002 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a731aff_55a4_45a9_8ce5_77dbcb2e7a04.slice/crio-59ca90844c556b0f945c58b930fdc80e8a7387ef83a551af501da68f5427b758 WatchSource:0}: Error finding container 59ca90844c556b0f945c58b930fdc80e8a7387ef83a551af501da68f5427b758: Status 404 returned error can't find the container with id 59ca90844c556b0f945c58b930fdc80e8a7387ef83a551af501da68f5427b758 Nov 02 06:21:37.112800 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:21:37.112779 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3665b4c2_f4d2_4080_8613_c42857882641.slice/crio-47770e5290f8655cf079524ef3209ae21d134a7e4a9349eb1636a7ccd898a6b5 WatchSource:0}: Error finding container 47770e5290f8655cf079524ef3209ae21d134a7e4a9349eb1636a7ccd898a6b5: Status 404 returned error can't find the container with id 47770e5290f8655cf079524ef3209ae21d134a7e4a9349eb1636a7ccd898a6b5 Nov 02 06:21:37.115220 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:37.115195 5916 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 06:21:37.115292 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:37.115252 5916 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 06:21:37.115292 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:37.115280 5916 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 06:21:37.691799 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:37.691760 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d774877-2mfp5" event={"ID":"3665b4c2-f4d2-4080-8613-c42857882641","Type":"ContainerStarted","Data":"19ac604fa6222a81149f74b695cfd1fd70e2fb114f385393749c2d2b884bddc3"} Nov 02 06:21:37.692306 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:37.691825 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d774877-2mfp5" event={"ID":"3665b4c2-f4d2-4080-8613-c42857882641","Type":"ContainerStarted","Data":"47770e5290f8655cf079524ef3209ae21d134a7e4a9349eb1636a7ccd898a6b5"} Nov 02 06:21:37.693618 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:37.693585 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-hptdf" event={"ID":"0a731aff-55a4-45a9-8ce5-77dbcb2e7a04","Type":"ContainerStarted","Data":"6ffd49c73292e1de14463f6d339f9a9a5b03ec2ec468c726fed0e0dd326f4a24"} Nov 02 06:21:37.693733 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:37.693642 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-hptdf" event={"ID":"0a731aff-55a4-45a9-8ce5-77dbcb2e7a04","Type":"ContainerStarted","Data":"59ca90844c556b0f945c58b930fdc80e8a7387ef83a551af501da68f5427b758"} Nov 02 06:21:37.693733 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:37.693672 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-hptdf" Nov 02 06:21:37.828002 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:37.827950 5916 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d774877-2mfp5" podStartSLOduration=1.827935597 podStartE2EDuration="1.827935597s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:21:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:21:37.826630414 +0000 UTC m=+286.098450817" watchObservedRunningTime="2026-11-02 06:21:37.827935597 +0000 UTC m=+286.099755984" Nov 02 06:21:37.942743 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:37.942624 5916 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-hptdf" podStartSLOduration=1.9426020849999999 podStartE2EDuration="1.942602085s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:21:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:21:37.940638456 +0000 UTC m=+286.212458851" watchObservedRunningTime="2026-11-02 06:21:37.942602085 +0000 UTC m=+286.214422480" Nov 02 06:21:38.678453 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:38.678413 5916 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d774877-2mfp5" Nov 02 06:21:38.683190 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:38.683161 5916 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d774877-2mfp5" Nov 02 06:21:38.699141 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:38.699113 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d774877-2mfp5" Nov 02 06:21:38.700062 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:38.700042 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d774877-2mfp5" Nov 02 06:21:39.648780 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:39.648745 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-lg9rv" Nov 02 06:21:40.653641 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:40.653610 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-ztxsk" Nov 02 06:21:45.867121 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:45.867087 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-jv5mm"] Nov 02 06:21:45.875716 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:45.875688 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-jv5mm" Nov 02 06:21:45.923471 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:45.923446 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-jv5mm"] Nov 02 06:21:45.936410 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:45.936376 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6d00e7f8-2ef8-418f-958e-95bc33c8aa24-bound-sa-token\") pod \"cert-manager-597b95cb84-jv5mm\" (UID: \"6d00e7f8-2ef8-418f-958e-95bc33c8aa24\") " pod="cert-manager/cert-manager-597b95cb84-jv5mm" Nov 02 06:21:45.936536 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:45.936492 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xc94\" (UniqueName: \"kubernetes.io/projected/6d00e7f8-2ef8-418f-958e-95bc33c8aa24-kube-api-access-5xc94\") pod \"cert-manager-597b95cb84-jv5mm\" (UID: \"6d00e7f8-2ef8-418f-958e-95bc33c8aa24\") " pod="cert-manager/cert-manager-597b95cb84-jv5mm" Nov 02 06:21:45.967482 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:45.967458 5916 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Nov 02 06:21:45.969660 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:45.969644 5916 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Nov 02 06:21:45.969660 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:45.969644 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-nqr62\"" Nov 02 06:21:46.037441 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:46.037410 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xc94\" (UniqueName: \"kubernetes.io/projected/6d00e7f8-2ef8-418f-958e-95bc33c8aa24-kube-api-access-5xc94\") pod \"cert-manager-597b95cb84-jv5mm\" (UID: \"6d00e7f8-2ef8-418f-958e-95bc33c8aa24\") " pod="cert-manager/cert-manager-597b95cb84-jv5mm" Nov 02 06:21:46.037597 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:46.037507 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6d00e7f8-2ef8-418f-958e-95bc33c8aa24-bound-sa-token\") pod \"cert-manager-597b95cb84-jv5mm\" (UID: \"6d00e7f8-2ef8-418f-958e-95bc33c8aa24\") " pod="cert-manager/cert-manager-597b95cb84-jv5mm" Nov 02 06:21:46.090354 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:46.090321 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6d00e7f8-2ef8-418f-958e-95bc33c8aa24-bound-sa-token\") pod \"cert-manager-597b95cb84-jv5mm\" (UID: \"6d00e7f8-2ef8-418f-958e-95bc33c8aa24\") " pod="cert-manager/cert-manager-597b95cb84-jv5mm" Nov 02 06:21:46.090520 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:46.090500 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xc94\" (UniqueName: \"kubernetes.io/projected/6d00e7f8-2ef8-418f-958e-95bc33c8aa24-kube-api-access-5xc94\") pod \"cert-manager-597b95cb84-jv5mm\" (UID: \"6d00e7f8-2ef8-418f-958e-95bc33c8aa24\") " pod="cert-manager/cert-manager-597b95cb84-jv5mm" Nov 02 06:21:46.197591 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:46.197559 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-jv5mm" Nov 02 06:21:46.562822 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:46.562741 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-jv5mm"] Nov 02 06:21:46.568863 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:21:46.568837 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6d00e7f8_2ef8_418f_958e_95bc33c8aa24.slice/crio-26642c6c1229be2d8499e55d090abe0090e05a6d3ec5145401542f09a1647572 WatchSource:0}: Error finding container 26642c6c1229be2d8499e55d090abe0090e05a6d3ec5145401542f09a1647572: Status 404 returned error can't find the container with id 26642c6c1229be2d8499e55d090abe0090e05a6d3ec5145401542f09a1647572 Nov 02 06:21:46.740581 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:46.740543 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-jv5mm" event={"ID":"6d00e7f8-2ef8-418f-958e-95bc33c8aa24","Type":"ContainerStarted","Data":"26642c6c1229be2d8499e55d090abe0090e05a6d3ec5145401542f09a1647572"} Nov 02 06:21:48.701255 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:48.701221 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-hptdf" Nov 02 06:21:48.890967 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:48.890934 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-8c429"] Nov 02 06:21:48.891156 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:48.891130 5916 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-8c429" podUID="8eea0eab-e7ff-4a11-a7e7-9b25f7830221" containerName="manager" containerID="cri-o://15feed1b417a13df68a040080728d23213b5e2ad1849fa6a56a90047663fb321" gracePeriod=10 Nov 02 06:21:49.758230 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:49.758192 5916 generic.go:356] "Generic (PLEG): container finished" podID="8eea0eab-e7ff-4a11-a7e7-9b25f7830221" containerID="15feed1b417a13df68a040080728d23213b5e2ad1849fa6a56a90047663fb321" exitCode=0 Nov 02 06:21:49.758663 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:49.758268 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-8c429" event={"ID":"8eea0eab-e7ff-4a11-a7e7-9b25f7830221","Type":"ContainerDied","Data":"15feed1b417a13df68a040080728d23213b5e2ad1849fa6a56a90047663fb321"} Nov 02 06:21:49.758663 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:49.758356 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-8c429" event={"ID":"8eea0eab-e7ff-4a11-a7e7-9b25f7830221","Type":"ContainerDied","Data":"91fb958cbfd914c763d9ab85d9d20bda1ea356f8da1a4e12205546a008ca2894"} Nov 02 06:21:49.758663 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:49.758373 5916 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="91fb958cbfd914c763d9ab85d9d20bda1ea356f8da1a4e12205546a008ca2894" Nov 02 06:21:49.770040 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:49.770014 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-8c429" Nov 02 06:21:49.874381 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:49.874283 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8eea0eab-e7ff-4a11-a7e7-9b25f7830221-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8eea0eab-e7ff-4a11-a7e7-9b25f7830221-extensions-socket-volume\") pod \"8eea0eab-e7ff-4a11-a7e7-9b25f7830221\" (UID: \"8eea0eab-e7ff-4a11-a7e7-9b25f7830221\") " Nov 02 06:21:49.874381 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:49.874373 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8eea0eab-e7ff-4a11-a7e7-9b25f7830221-kube-api-access-7pwkw\" (UniqueName: \"kubernetes.io/projected/8eea0eab-e7ff-4a11-a7e7-9b25f7830221-kube-api-access-7pwkw\") pod \"8eea0eab-e7ff-4a11-a7e7-9b25f7830221\" (UID: \"8eea0eab-e7ff-4a11-a7e7-9b25f7830221\") " Nov 02 06:21:49.874832 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:49.874806 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8eea0eab-e7ff-4a11-a7e7-9b25f7830221-extensions-socket-volume" pod "8eea0eab-e7ff-4a11-a7e7-9b25f7830221" (UID: "8eea0eab-e7ff-4a11-a7e7-9b25f7830221"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:21:49.877385 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:49.877355 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8eea0eab-e7ff-4a11-a7e7-9b25f7830221-kube-api-access-7pwkw" pod "8eea0eab-e7ff-4a11-a7e7-9b25f7830221" (UID: "8eea0eab-e7ff-4a11-a7e7-9b25f7830221"). InnerVolumeSpecName "kube-api-access-7pwkw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:21:49.899095 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:21:49.899051 5916 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p98cq" containerName="extractor" Nov 02 06:21:49.975475 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:49.975442 5916 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8eea0eab-e7ff-4a11-a7e7-9b25f7830221-extensions-socket-volume\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:21:49.975475 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:49.975472 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7pwkw\" (UniqueName: \"kubernetes.io/projected/8eea0eab-e7ff-4a11-a7e7-9b25f7830221-kube-api-access-7pwkw\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:21:50.764884 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:50.764849 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-8c429" Nov 02 06:21:50.964967 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:50.964934 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-8c429"] Nov 02 06:21:51.010851 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:51.010821 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-8c429"] Nov 02 06:21:51.771868 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:51.771828 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-jv5mm" event={"ID":"6d00e7f8-2ef8-418f-958e-95bc33c8aa24","Type":"ContainerStarted","Data":"9252dd2512fe4d807fdd3972e6611e15415b61db316ff4ae4b0415217959715e"} Nov 02 06:21:51.840696 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:51.840658 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8eea0eab-e7ff-4a11-a7e7-9b25f7830221" path="/var/lib/kubelet/pods/8eea0eab-e7ff-4a11-a7e7-9b25f7830221/volumes" Nov 02 06:21:51.877581 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:51.877552 5916 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-csfhg_17dee72b-2f47-4016-8ffe-3a89934e66ee/console-operator/2.log" Nov 02 06:21:51.878166 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:51.878144 5916 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-csfhg_17dee72b-2f47-4016-8ffe-3a89934e66ee/console-operator/2.log" Nov 02 06:21:56.840572 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:56.840524 5916 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-jv5mm" podStartSLOduration=7.545411967 podStartE2EDuration="11.840510694s" totalImagesPullingTime="4.295098727s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:21:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:21:51.876885535 +0000 UTC m=+300.148705925" watchObservedRunningTime="2026-11-02 06:21:56.840510694 +0000 UTC m=+305.112331085" Nov 02 06:21:56.843352 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:56.843318 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-nv9mk"] Nov 02 06:21:56.844250 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:56.844226 5916 cpu_manager.go:414] "containerMap: removing container" podUID="8eea0eab-e7ff-4a11-a7e7-9b25f7830221" containerName="manager" Nov 02 06:21:56.844370 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:56.844253 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8eea0eab-e7ff-4a11-a7e7-9b25f7830221" containerName="manager" Nov 02 06:21:56.844436 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:56.844424 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="8eea0eab-e7ff-4a11-a7e7-9b25f7830221" containerName="manager" Nov 02 06:21:56.880653 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:56.880623 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-nv9mk" Nov 02 06:21:56.898370 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:56.898344 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-nv9mk"] Nov 02 06:21:56.937008 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:56.936976 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szl8t\" (UniqueName: \"kubernetes.io/projected/01649370-f637-4f7e-820f-a2af5fc068c6-kube-api-access-szl8t\") pod \"authorino-57f7f8659f-nv9mk\" (UID: \"01649370-f637-4f7e-820f-a2af5fc068c6\") " pod="kuadrant-system/authorino-57f7f8659f-nv9mk" Nov 02 06:21:56.985963 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:56.985932 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-x84wk\"" Nov 02 06:21:57.038318 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:57.038286 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szl8t\" (UniqueName: \"kubernetes.io/projected/01649370-f637-4f7e-820f-a2af5fc068c6-kube-api-access-szl8t\") pod \"authorino-57f7f8659f-nv9mk\" (UID: \"01649370-f637-4f7e-820f-a2af5fc068c6\") " pod="kuadrant-system/authorino-57f7f8659f-nv9mk" Nov 02 06:21:57.093540 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:57.093474 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-szl8t\" (UniqueName: \"kubernetes.io/projected/01649370-f637-4f7e-820f-a2af5fc068c6-kube-api-access-szl8t\") pod \"authorino-57f7f8659f-nv9mk\" (UID: \"01649370-f637-4f7e-820f-a2af5fc068c6\") " pod="kuadrant-system/authorino-57f7f8659f-nv9mk" Nov 02 06:21:57.207151 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:57.207116 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-nv9mk" Nov 02 06:21:57.591806 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:57.591767 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-nv9mk"] Nov 02 06:21:57.597797 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:21:57.597770 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod01649370_f637_4f7e_820f_a2af5fc068c6.slice/crio-e464d91bcd7166618cd129b7ac8d8192f5aca0f3d5ca97b7bf33636edc0e77f6 WatchSource:0}: Error finding container e464d91bcd7166618cd129b7ac8d8192f5aca0f3d5ca97b7bf33636edc0e77f6: Status 404 returned error can't find the container with id e464d91bcd7166618cd129b7ac8d8192f5aca0f3d5ca97b7bf33636edc0e77f6 Nov 02 06:21:57.803896 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:21:57.803860 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-nv9mk" event={"ID":"01649370-f637-4f7e-820f-a2af5fc068c6","Type":"ContainerStarted","Data":"e464d91bcd7166618cd129b7ac8d8192f5aca0f3d5ca97b7bf33636edc0e77f6"} Nov 02 06:22:01.828708 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:22:01.828673 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-nv9mk" event={"ID":"01649370-f637-4f7e-820f-a2af5fc068c6","Type":"ContainerStarted","Data":"b45d24b6ec858dd01c72515667642bfcf4dbee25c8f185b244f8ab6b960a3574"} Nov 02 06:22:01.936971 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:22:01.936916 5916 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-nv9mk" podStartSLOduration=2.684551628 podStartE2EDuration="5.936890125s" totalImagesPullingTime="3.252338497s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:21:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:22:01.934528183 +0000 UTC m=+310.206348579" watchObservedRunningTime="2026-11-02 06:22:01.936890125 +0000 UTC m=+310.208710498" Nov 02 06:22:13.917397 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:22:13.917355 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-nv9mk"] Nov 02 06:22:13.917856 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:22:13.917644 5916 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-nv9mk" podUID="01649370-f637-4f7e-820f-a2af5fc068c6" containerName="authorino" containerID="cri-o://b45d24b6ec858dd01c72515667642bfcf4dbee25c8f185b244f8ab6b960a3574" gracePeriod=30 Nov 02 06:22:14.164171 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:22:14.164144 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-nv9mk" Nov 02 06:22:14.292471 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:22:14.292382 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01649370-f637-4f7e-820f-a2af5fc068c6-kube-api-access-szl8t\" (UniqueName: \"kubernetes.io/projected/01649370-f637-4f7e-820f-a2af5fc068c6-kube-api-access-szl8t\") pod \"01649370-f637-4f7e-820f-a2af5fc068c6\" (UID: \"01649370-f637-4f7e-820f-a2af5fc068c6\") " Nov 02 06:22:14.294510 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:22:14.294475 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01649370-f637-4f7e-820f-a2af5fc068c6-kube-api-access-szl8t" pod "01649370-f637-4f7e-820f-a2af5fc068c6" (UID: "01649370-f637-4f7e-820f-a2af5fc068c6"). InnerVolumeSpecName "kube-api-access-szl8t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:22:14.393962 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:22:14.393929 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szl8t\" (UniqueName: \"kubernetes.io/projected/01649370-f637-4f7e-820f-a2af5fc068c6-kube-api-access-szl8t\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:22:14.895312 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:22:14.895277 5916 generic.go:356] "Generic (PLEG): container finished" podID="01649370-f637-4f7e-820f-a2af5fc068c6" containerID="b45d24b6ec858dd01c72515667642bfcf4dbee25c8f185b244f8ab6b960a3574" exitCode=0 Nov 02 06:22:14.895509 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:22:14.895323 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-nv9mk" Nov 02 06:22:14.895509 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:22:14.895370 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-nv9mk" event={"ID":"01649370-f637-4f7e-820f-a2af5fc068c6","Type":"ContainerDied","Data":"b45d24b6ec858dd01c72515667642bfcf4dbee25c8f185b244f8ab6b960a3574"} Nov 02 06:22:14.895509 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:22:14.895420 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-nv9mk" event={"ID":"01649370-f637-4f7e-820f-a2af5fc068c6","Type":"ContainerDied","Data":"e464d91bcd7166618cd129b7ac8d8192f5aca0f3d5ca97b7bf33636edc0e77f6"} Nov 02 06:22:14.895509 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:22:14.895445 5916 scope.go:122] "RemoveContainer" containerID="b45d24b6ec858dd01c72515667642bfcf4dbee25c8f185b244f8ab6b960a3574" Nov 02 06:22:14.904895 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:22:14.904878 5916 scope.go:122] "RemoveContainer" containerID="b45d24b6ec858dd01c72515667642bfcf4dbee25c8f185b244f8ab6b960a3574" Nov 02 06:22:14.905165 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:22:14.905149 5916 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b45d24b6ec858dd01c72515667642bfcf4dbee25c8f185b244f8ab6b960a3574\": container with ID starting with b45d24b6ec858dd01c72515667642bfcf4dbee25c8f185b244f8ab6b960a3574 not found: ID does not exist" containerID="b45d24b6ec858dd01c72515667642bfcf4dbee25c8f185b244f8ab6b960a3574" Nov 02 06:22:14.905205 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:22:14.905174 5916 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b45d24b6ec858dd01c72515667642bfcf4dbee25c8f185b244f8ab6b960a3574"} err="failed to get container status \"b45d24b6ec858dd01c72515667642bfcf4dbee25c8f185b244f8ab6b960a3574\": rpc error: code = NotFound desc = could not find container \"b45d24b6ec858dd01c72515667642bfcf4dbee25c8f185b244f8ab6b960a3574\": container with ID starting with b45d24b6ec858dd01c72515667642bfcf4dbee25c8f185b244f8ab6b960a3574 not found: ID does not exist" Nov 02 06:22:15.050706 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:22:15.050662 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-nv9mk"] Nov 02 06:22:15.052531 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:22:15.052506 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-nv9mk"] Nov 02 06:22:15.840300 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:22:15.840225 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01649370-f637-4f7e-820f-a2af5fc068c6" path="/var/lib/kubelet/pods/01649370-f637-4f7e-820f-a2af5fc068c6/volumes" Nov 02 06:22:19.892975 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:22:19.892935 5916 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p98cq" containerName="extractor" Nov 02 06:22:34.747875 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:22:34.747838 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-579cf7c6d-vv9bd"] Nov 02 06:22:34.748411 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:22:34.748258 5916 cpu_manager.go:414] "containerMap: removing container" podUID="01649370-f637-4f7e-820f-a2af5fc068c6" containerName="authorino" Nov 02 06:22:34.748411 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:22:34.748271 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01649370-f637-4f7e-820f-a2af5fc068c6" containerName="authorino" Nov 02 06:22:34.748411 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:22:34.748349 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="01649370-f637-4f7e-820f-a2af5fc068c6" containerName="authorino" Nov 02 06:22:34.751580 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:22:34.751559 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-vv9bd" Nov 02 06:22:34.766136 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:22:34.766111 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70c7215c-d8bb-4bbd-9085-a1e15c099e6a-metrics-certs\") pod \"maas-controller-579cf7c6d-vv9bd\" (UID: \"70c7215c-d8bb-4bbd-9085-a1e15c099e6a\") " pod="opendatahub/maas-controller-579cf7c6d-vv9bd" Nov 02 06:22:34.766295 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:22:34.766162 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/70c7215c-d8bb-4bbd-9085-a1e15c099e6a-webhook-cert\") pod \"maas-controller-579cf7c6d-vv9bd\" (UID: \"70c7215c-d8bb-4bbd-9085-a1e15c099e6a\") " pod="opendatahub/maas-controller-579cf7c6d-vv9bd" Nov 02 06:22:34.766295 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:22:34.766244 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjpn6\" (UniqueName: \"kubernetes.io/projected/70c7215c-d8bb-4bbd-9085-a1e15c099e6a-kube-api-access-jjpn6\") pod \"maas-controller-579cf7c6d-vv9bd\" (UID: \"70c7215c-d8bb-4bbd-9085-a1e15c099e6a\") " pod="opendatahub/maas-controller-579cf7c6d-vv9bd" Nov 02 06:22:34.810847 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:22:34.810809 5916 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 02 06:22:34.810847 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:22:34.810833 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-prjsd\"" Nov 02 06:22:34.811119 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:22:34.810852 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 02 06:22:34.846650 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:22:34.846621 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 02 06:22:34.846828 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:22:34.846647 5916 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 02 06:22:34.846828 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:22:34.846679 5916 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 02 06:22:34.848864 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:22:34.848846 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-vv9bd"] Nov 02 06:22:34.867227 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:22:34.867195 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjpn6\" (UniqueName: \"kubernetes.io/projected/70c7215c-d8bb-4bbd-9085-a1e15c099e6a-kube-api-access-jjpn6\") pod \"maas-controller-579cf7c6d-vv9bd\" (UID: \"70c7215c-d8bb-4bbd-9085-a1e15c099e6a\") " pod="opendatahub/maas-controller-579cf7c6d-vv9bd" Nov 02 06:22:34.867379 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:22:34.867274 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70c7215c-d8bb-4bbd-9085-a1e15c099e6a-metrics-certs\") pod \"maas-controller-579cf7c6d-vv9bd\" (UID: \"70c7215c-d8bb-4bbd-9085-a1e15c099e6a\") " pod="opendatahub/maas-controller-579cf7c6d-vv9bd" Nov 02 06:22:34.867379 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:22:34.867304 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/70c7215c-d8bb-4bbd-9085-a1e15c099e6a-webhook-cert\") pod \"maas-controller-579cf7c6d-vv9bd\" (UID: \"70c7215c-d8bb-4bbd-9085-a1e15c099e6a\") " pod="opendatahub/maas-controller-579cf7c6d-vv9bd" Nov 02 06:22:34.870019 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:22:34.869994 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/70c7215c-d8bb-4bbd-9085-a1e15c099e6a-webhook-cert\") pod \"maas-controller-579cf7c6d-vv9bd\" (UID: \"70c7215c-d8bb-4bbd-9085-a1e15c099e6a\") " pod="opendatahub/maas-controller-579cf7c6d-vv9bd" Nov 02 06:22:34.870151 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:22:34.870130 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70c7215c-d8bb-4bbd-9085-a1e15c099e6a-metrics-certs\") pod \"maas-controller-579cf7c6d-vv9bd\" (UID: \"70c7215c-d8bb-4bbd-9085-a1e15c099e6a\") " pod="opendatahub/maas-controller-579cf7c6d-vv9bd" Nov 02 06:22:34.921731 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:22:34.921701 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjpn6\" (UniqueName: \"kubernetes.io/projected/70c7215c-d8bb-4bbd-9085-a1e15c099e6a-kube-api-access-jjpn6\") pod \"maas-controller-579cf7c6d-vv9bd\" (UID: \"70c7215c-d8bb-4bbd-9085-a1e15c099e6a\") " pod="opendatahub/maas-controller-579cf7c6d-vv9bd" Nov 02 06:22:35.065162 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:22:35.065076 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-vv9bd" Nov 02 06:22:35.150230 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:22:35.150197 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-59fb8fd585-xgqcq"] Nov 02 06:22:35.153972 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:22:35.153953 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-59fb8fd585-xgqcq" Nov 02 06:22:35.169361 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:22:35.169334 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hx8hp\" (UniqueName: \"kubernetes.io/projected/7c414f8b-a817-405d-aedf-6c570540d01a-kube-api-access-hx8hp\") pod \"maas-controller-59fb8fd585-xgqcq\" (UID: \"7c414f8b-a817-405d-aedf-6c570540d01a\") " pod="opendatahub/maas-controller-59fb8fd585-xgqcq" Nov 02 06:22:35.169490 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:22:35.169377 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c414f8b-a817-405d-aedf-6c570540d01a-metrics-certs\") pod \"maas-controller-59fb8fd585-xgqcq\" (UID: \"7c414f8b-a817-405d-aedf-6c570540d01a\") " pod="opendatahub/maas-controller-59fb8fd585-xgqcq" Nov 02 06:22:35.169490 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:22:35.169417 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7c414f8b-a817-405d-aedf-6c570540d01a-webhook-cert\") pod \"maas-controller-59fb8fd585-xgqcq\" (UID: \"7c414f8b-a817-405d-aedf-6c570540d01a\") " pod="opendatahub/maas-controller-59fb8fd585-xgqcq" Nov 02 06:22:35.208087 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:22:35.208045 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-59fb8fd585-xgqcq"] Nov 02 06:22:35.270056 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:22:35.270018 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hx8hp\" (UniqueName: \"kubernetes.io/projected/7c414f8b-a817-405d-aedf-6c570540d01a-kube-api-access-hx8hp\") pod \"maas-controller-59fb8fd585-xgqcq\" (UID: \"7c414f8b-a817-405d-aedf-6c570540d01a\") " pod="opendatahub/maas-controller-59fb8fd585-xgqcq" Nov 02 06:22:35.270238 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:22:35.270074 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c414f8b-a817-405d-aedf-6c570540d01a-metrics-certs\") pod \"maas-controller-59fb8fd585-xgqcq\" (UID: \"7c414f8b-a817-405d-aedf-6c570540d01a\") " pod="opendatahub/maas-controller-59fb8fd585-xgqcq" Nov 02 06:22:35.270238 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:22:35.270123 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7c414f8b-a817-405d-aedf-6c570540d01a-webhook-cert\") pod \"maas-controller-59fb8fd585-xgqcq\" (UID: \"7c414f8b-a817-405d-aedf-6c570540d01a\") " pod="opendatahub/maas-controller-59fb8fd585-xgqcq" Nov 02 06:22:35.272515 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:22:35.272489 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c414f8b-a817-405d-aedf-6c570540d01a-metrics-certs\") pod \"maas-controller-59fb8fd585-xgqcq\" (UID: \"7c414f8b-a817-405d-aedf-6c570540d01a\") " pod="opendatahub/maas-controller-59fb8fd585-xgqcq" Nov 02 06:22:35.272624 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:22:35.272530 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7c414f8b-a817-405d-aedf-6c570540d01a-webhook-cert\") pod \"maas-controller-59fb8fd585-xgqcq\" (UID: \"7c414f8b-a817-405d-aedf-6c570540d01a\") " pod="opendatahub/maas-controller-59fb8fd585-xgqcq" Nov 02 06:22:35.323691 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:22:35.323621 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hx8hp\" (UniqueName: \"kubernetes.io/projected/7c414f8b-a817-405d-aedf-6c570540d01a-kube-api-access-hx8hp\") pod \"maas-controller-59fb8fd585-xgqcq\" (UID: \"7c414f8b-a817-405d-aedf-6c570540d01a\") " pod="opendatahub/maas-controller-59fb8fd585-xgqcq" Nov 02 06:22:35.433072 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:22:35.433037 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-vv9bd"] Nov 02 06:22:35.439279 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:22:35.439244 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70c7215c_d8bb_4bbd_9085_a1e15c099e6a.slice/crio-9cc3f8e74238f564f6890d653dadb112b8314616665fafc9aeaa341416f18535 WatchSource:0}: Error finding container 9cc3f8e74238f564f6890d653dadb112b8314616665fafc9aeaa341416f18535: Status 404 returned error can't find the container with id 9cc3f8e74238f564f6890d653dadb112b8314616665fafc9aeaa341416f18535 Nov 02 06:22:35.440513 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:22:35.440495 5916 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 06:22:35.467526 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:22:35.467487 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-59fb8fd585-xgqcq" Nov 02 06:22:35.842674 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:22:35.842650 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-59fb8fd585-xgqcq"] Nov 02 06:22:35.843935 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:22:35.843887 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c414f8b_a817_405d_aedf_6c570540d01a.slice/crio-7183917691cb7747fa9902c61323b5ff3245f5661cd3e8d636d3d09eb8007281 WatchSource:0}: Error finding container 7183917691cb7747fa9902c61323b5ff3245f5661cd3e8d636d3d09eb8007281: Status 404 returned error can't find the container with id 7183917691cb7747fa9902c61323b5ff3245f5661cd3e8d636d3d09eb8007281 Nov 02 06:22:36.009085 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:22:36.009045 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-59fb8fd585-xgqcq" event={"ID":"7c414f8b-a817-405d-aedf-6c570540d01a","Type":"ContainerStarted","Data":"7183917691cb7747fa9902c61323b5ff3245f5661cd3e8d636d3d09eb8007281"} Nov 02 06:22:36.010241 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:22:36.010214 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-vv9bd" event={"ID":"70c7215c-d8bb-4bbd-9085-a1e15c099e6a","Type":"ContainerStarted","Data":"9cc3f8e74238f564f6890d653dadb112b8314616665fafc9aeaa341416f18535"} Nov 02 06:22:40.038274 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:22:40.038234 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-vv9bd" event={"ID":"70c7215c-d8bb-4bbd-9085-a1e15c099e6a","Type":"ContainerStarted","Data":"fe16f504382fa4cb19725817e93d0485026a8b66a3412ea369ac96f64f26c196"} Nov 02 06:22:40.038773 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:22:40.038308 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-579cf7c6d-vv9bd" Nov 02 06:22:40.039830 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:22:40.039804 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-59fb8fd585-xgqcq" event={"ID":"7c414f8b-a817-405d-aedf-6c570540d01a","Type":"ContainerStarted","Data":"ac0ab7db0e98409adc796e615d2492a81d94a2b8dd1158d2a2127bcb7a5a58cc"} Nov 02 06:22:40.040008 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:22:40.039956 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-59fb8fd585-xgqcq" Nov 02 06:22:40.149130 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:22:40.149060 5916 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-579cf7c6d-vv9bd" podStartSLOduration=2.537704432 podStartE2EDuration="6.149046444s" totalImagesPullingTime="3.611342012s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:22:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:22:40.146958298 +0000 UTC m=+348.418778694" watchObservedRunningTime="2026-11-02 06:22:40.149046444 +0000 UTC m=+348.420866886" Nov 02 06:22:42.730518 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:22:42.730461 5916 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-59fb8fd585-xgqcq" podStartSLOduration=4.23060561 podStartE2EDuration="7.730447693s" totalImagesPullingTime="3.499842083s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:22:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:22:40.261705224 +0000 UTC m=+348.533525630" watchObservedRunningTime="2026-11-02 06:22:42.730447693 +0000 UTC m=+351.002268081" Nov 02 06:22:42.731092 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:22:42.731074 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-68wpm"] Nov 02 06:22:42.734764 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:22:42.734742 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-68wpm" Nov 02 06:22:42.832225 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:22:42.832189 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\"" Nov 02 06:22:42.832225 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:22:42.832195 5916 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\"" Nov 02 06:22:42.832225 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:22:42.832201 5916 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\"" Nov 02 06:22:42.832519 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:22:42.832256 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-vpwk6\"" Nov 02 06:22:42.834726 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:22:42.834702 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-68wpm"] Nov 02 06:22:42.841568 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:22:42.841541 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/23e396f1-81ba-4a29-8771-4f7385b52577-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-68wpm\" (UID: \"23e396f1-81ba-4a29-8771-4f7385b52577\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-68wpm" Nov 02 06:22:42.841751 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:22:42.841597 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/23e396f1-81ba-4a29-8771-4f7385b52577-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-68wpm\" (UID: \"23e396f1-81ba-4a29-8771-4f7385b52577\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-68wpm" Nov 02 06:22:42.841751 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:22:42.841698 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rflw5\" (UniqueName: \"kubernetes.io/projected/23e396f1-81ba-4a29-8771-4f7385b52577-kube-api-access-rflw5\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-68wpm\" (UID: \"23e396f1-81ba-4a29-8771-4f7385b52577\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-68wpm" Nov 02 06:22:42.841751 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:22:42.841724 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/23e396f1-81ba-4a29-8771-4f7385b52577-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-68wpm\" (UID: \"23e396f1-81ba-4a29-8771-4f7385b52577\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-68wpm" Nov 02 06:22:42.942537 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:22:42.942468 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rflw5\" (UniqueName: \"kubernetes.io/projected/23e396f1-81ba-4a29-8771-4f7385b52577-kube-api-access-rflw5\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-68wpm\" (UID: \"23e396f1-81ba-4a29-8771-4f7385b52577\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-68wpm" Nov 02 06:22:42.942737 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:22:42.942662 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/23e396f1-81ba-4a29-8771-4f7385b52577-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-68wpm\" (UID: \"23e396f1-81ba-4a29-8771-4f7385b52577\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-68wpm" Nov 02 06:22:42.942808 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:22:42.942756 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/23e396f1-81ba-4a29-8771-4f7385b52577-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-68wpm\" (UID: \"23e396f1-81ba-4a29-8771-4f7385b52577\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-68wpm" Nov 02 06:22:42.942889 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:22:42.942863 5916 configmap.go:193] Couldn't get configMap opendatahub/usage-tenancy-proxy-source: configmap "usage-tenancy-proxy-source" not found Nov 02 06:22:42.943012 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:22:42.942997 5916 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/23e396f1-81ba-4a29-8771-4f7385b52577-proxy-source podName:23e396f1-81ba-4a29-8771-4f7385b52577 nodeName:}" failed. No retries permitted until 2026-11-02 06:22:43.44295652 +0000 UTC m=+351.714776920 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-source" (UniqueName: "kubernetes.io/configmap/23e396f1-81ba-4a29-8771-4f7385b52577-proxy-source") pod "usage-logs-tenancy-proxy-9c5d49bcd-68wpm" (UID: "23e396f1-81ba-4a29-8771-4f7385b52577") : configmap "usage-tenancy-proxy-source" not found Nov 02 06:22:42.943088 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:22:42.943028 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/23e396f1-81ba-4a29-8771-4f7385b52577-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-68wpm\" (UID: \"23e396f1-81ba-4a29-8771-4f7385b52577\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-68wpm" Nov 02 06:22:42.943659 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:22:42.943639 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/23e396f1-81ba-4a29-8771-4f7385b52577-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-68wpm\" (UID: \"23e396f1-81ba-4a29-8771-4f7385b52577\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-68wpm" Nov 02 06:22:42.945226 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:22:42.945205 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/23e396f1-81ba-4a29-8771-4f7385b52577-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-68wpm\" (UID: \"23e396f1-81ba-4a29-8771-4f7385b52577\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-68wpm" Nov 02 06:22:42.998671 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:22:42.998604 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rflw5\" (UniqueName: \"kubernetes.io/projected/23e396f1-81ba-4a29-8771-4f7385b52577-kube-api-access-rflw5\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-68wpm\" (UID: \"23e396f1-81ba-4a29-8771-4f7385b52577\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-68wpm" Nov 02 06:22:43.448537 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:22:43.448492 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/23e396f1-81ba-4a29-8771-4f7385b52577-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-68wpm\" (UID: \"23e396f1-81ba-4a29-8771-4f7385b52577\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-68wpm" Nov 02 06:22:43.449337 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:22:43.449316 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/23e396f1-81ba-4a29-8771-4f7385b52577-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-68wpm\" (UID: \"23e396f1-81ba-4a29-8771-4f7385b52577\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-68wpm" Nov 02 06:22:43.647395 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:22:43.647352 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-68wpm" Nov 02 06:22:44.018559 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:22:44.018466 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-68wpm"] Nov 02 06:22:44.024394 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:22:44.024358 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23e396f1_81ba_4a29_8771_4f7385b52577.slice/crio-1254a4111a922d503c3f41ee8c3f0f7ed328ed3b59ebae08d4621ecca65ce326 WatchSource:0}: Error finding container 1254a4111a922d503c3f41ee8c3f0f7ed328ed3b59ebae08d4621ecca65ce326: Status 404 returned error can't find the container with id 1254a4111a922d503c3f41ee8c3f0f7ed328ed3b59ebae08d4621ecca65ce326 Nov 02 06:22:44.063251 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:22:44.063216 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-68wpm" event={"ID":"23e396f1-81ba-4a29-8771-4f7385b52577","Type":"ContainerStarted","Data":"1254a4111a922d503c3f41ee8c3f0f7ed328ed3b59ebae08d4621ecca65ce326"} Nov 02 06:22:49.904606 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:22:49.904566 5916 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p98cq" containerName="extractor" Nov 02 06:22:51.049277 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:22:51.049248 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-59fb8fd585-xgqcq" Nov 02 06:22:51.049782 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:22:51.049335 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-579cf7c6d-vv9bd" Nov 02 06:22:51.063943 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:22:51.063894 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-ffx9j"] Nov 02 06:22:51.068120 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:22:51.068093 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-ffx9j" Nov 02 06:22:51.127963 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:22:51.127900 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/100eebf9-dc20-403c-bad6-cc053586aa5d-cert\") pod \"kserve-controller-manager-776cc66d47-ffx9j\" (UID: \"100eebf9-dc20-403c-bad6-cc053586aa5d\") " pod="opendatahub/kserve-controller-manager-776cc66d47-ffx9j" Nov 02 06:22:51.128155 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:22:51.128070 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79xrl\" (UniqueName: \"kubernetes.io/projected/100eebf9-dc20-403c-bad6-cc053586aa5d-kube-api-access-79xrl\") pod \"kserve-controller-manager-776cc66d47-ffx9j\" (UID: \"100eebf9-dc20-403c-bad6-cc053586aa5d\") " pod="opendatahub/kserve-controller-manager-776cc66d47-ffx9j" Nov 02 06:22:51.132688 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:22:51.132663 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-k4gsr\"" Nov 02 06:22:51.132831 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:22:51.132760 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Nov 02 06:22:51.161803 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:22:51.161772 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-ffx9j"] Nov 02 06:22:51.172447 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:22:51.172410 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-fnf5n"] Nov 02 06:22:51.178807 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:22:51.178782 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-fnf5n" Nov 02 06:22:51.199407 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:22:51.199378 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-vv9bd"] Nov 02 06:22:51.199757 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:22:51.199731 5916 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-579cf7c6d-vv9bd" podUID="70c7215c-d8bb-4bbd-9085-a1e15c099e6a" containerName="manager" containerID="cri-o://fe16f504382fa4cb19725817e93d0485026a8b66a3412ea369ac96f64f26c196" gracePeriod=30 Nov 02 06:22:51.228685 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:22:51.228644 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4hnz\" (UniqueName: \"kubernetes.io/projected/7043e9d5-e26f-426a-b480-9f9880da0962-kube-api-access-c4hnz\") pod \"llmisvc-controller-manager-869d5dcc5d-fnf5n\" (UID: \"7043e9d5-e26f-426a-b480-9f9880da0962\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-fnf5n" Nov 02 06:22:51.228938 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:22:51.228832 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Nov 02 06:22:51.229021 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:22:51.228941 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-49hgv\"" Nov 02 06:22:51.229492 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:22:51.229463 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/100eebf9-dc20-403c-bad6-cc053586aa5d-cert\") pod \"kserve-controller-manager-776cc66d47-ffx9j\" (UID: \"100eebf9-dc20-403c-bad6-cc053586aa5d\") " pod="opendatahub/kserve-controller-manager-776cc66d47-ffx9j" Nov 02 06:22:51.229587 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:22:51.229516 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7043e9d5-e26f-426a-b480-9f9880da0962-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-fnf5n\" (UID: \"7043e9d5-e26f-426a-b480-9f9880da0962\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-fnf5n" Nov 02 06:22:51.229644 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:22:51.229586 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79xrl\" (UniqueName: \"kubernetes.io/projected/100eebf9-dc20-403c-bad6-cc053586aa5d-kube-api-access-79xrl\") pod \"kserve-controller-manager-776cc66d47-ffx9j\" (UID: \"100eebf9-dc20-403c-bad6-cc053586aa5d\") " pod="opendatahub/kserve-controller-manager-776cc66d47-ffx9j" Nov 02 06:22:51.241463 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:22:51.238781 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/100eebf9-dc20-403c-bad6-cc053586aa5d-cert\") pod \"kserve-controller-manager-776cc66d47-ffx9j\" (UID: \"100eebf9-dc20-403c-bad6-cc053586aa5d\") " pod="opendatahub/kserve-controller-manager-776cc66d47-ffx9j" Nov 02 06:22:51.241463 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:22:51.239539 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-fnf5n"] Nov 02 06:22:51.287290 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:22:51.287253 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79xrl\" (UniqueName: \"kubernetes.io/projected/100eebf9-dc20-403c-bad6-cc053586aa5d-kube-api-access-79xrl\") pod \"kserve-controller-manager-776cc66d47-ffx9j\" (UID: \"100eebf9-dc20-403c-bad6-cc053586aa5d\") " pod="opendatahub/kserve-controller-manager-776cc66d47-ffx9j" Nov 02 06:22:51.332254 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:22:51.331222 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4hnz\" (UniqueName: \"kubernetes.io/projected/7043e9d5-e26f-426a-b480-9f9880da0962-kube-api-access-c4hnz\") pod \"llmisvc-controller-manager-869d5dcc5d-fnf5n\" (UID: \"7043e9d5-e26f-426a-b480-9f9880da0962\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-fnf5n" Nov 02 06:22:51.333084 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:22:51.332560 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7043e9d5-e26f-426a-b480-9f9880da0962-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-fnf5n\" (UID: \"7043e9d5-e26f-426a-b480-9f9880da0962\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-fnf5n" Nov 02 06:22:51.336428 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:22:51.336400 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7043e9d5-e26f-426a-b480-9f9880da0962-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-fnf5n\" (UID: \"7043e9d5-e26f-426a-b480-9f9880da0962\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-fnf5n" Nov 02 06:22:51.386753 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:22:51.386693 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-ffx9j" Nov 02 06:22:51.387681 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:22:51.387618 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4hnz\" (UniqueName: \"kubernetes.io/projected/7043e9d5-e26f-426a-b480-9f9880da0962-kube-api-access-c4hnz\") pod \"llmisvc-controller-manager-869d5dcc5d-fnf5n\" (UID: \"7043e9d5-e26f-426a-b480-9f9880da0962\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-fnf5n" Nov 02 06:22:51.500059 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:22:51.500020 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-fnf5n" Nov 02 06:22:51.616278 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:22:51.616252 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-vv9bd" Nov 02 06:22:51.736762 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:22:51.736728 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70c7215c-d8bb-4bbd-9085-a1e15c099e6a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/70c7215c-d8bb-4bbd-9085-a1e15c099e6a-metrics-certs\") pod \"70c7215c-d8bb-4bbd-9085-a1e15c099e6a\" (UID: \"70c7215c-d8bb-4bbd-9085-a1e15c099e6a\") " Nov 02 06:22:51.736960 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:22:51.736792 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70c7215c-d8bb-4bbd-9085-a1e15c099e6a-webhook-cert\" (UniqueName: \"kubernetes.io/secret/70c7215c-d8bb-4bbd-9085-a1e15c099e6a-webhook-cert\") pod \"70c7215c-d8bb-4bbd-9085-a1e15c099e6a\" (UID: \"70c7215c-d8bb-4bbd-9085-a1e15c099e6a\") " Nov 02 06:22:51.736960 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:22:51.736828 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70c7215c-d8bb-4bbd-9085-a1e15c099e6a-kube-api-access-jjpn6\" (UniqueName: \"kubernetes.io/projected/70c7215c-d8bb-4bbd-9085-a1e15c099e6a-kube-api-access-jjpn6\") pod \"70c7215c-d8bb-4bbd-9085-a1e15c099e6a\" (UID: \"70c7215c-d8bb-4bbd-9085-a1e15c099e6a\") " Nov 02 06:22:51.744299 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:22:51.744250 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70c7215c-d8bb-4bbd-9085-a1e15c099e6a-metrics-certs" pod "70c7215c-d8bb-4bbd-9085-a1e15c099e6a" (UID: "70c7215c-d8bb-4bbd-9085-a1e15c099e6a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:22:51.744454 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:22:51.744341 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70c7215c-d8bb-4bbd-9085-a1e15c099e6a-webhook-cert" pod "70c7215c-d8bb-4bbd-9085-a1e15c099e6a" (UID: "70c7215c-d8bb-4bbd-9085-a1e15c099e6a"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:22:51.746007 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:22:51.745893 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70c7215c-d8bb-4bbd-9085-a1e15c099e6a-kube-api-access-jjpn6" pod "70c7215c-d8bb-4bbd-9085-a1e15c099e6a" (UID: "70c7215c-d8bb-4bbd-9085-a1e15c099e6a"). InnerVolumeSpecName "kube-api-access-jjpn6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:22:51.776948 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:22:51.776920 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-ffx9j"] Nov 02 06:22:51.783745 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:22:51.783712 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod100eebf9_dc20_403c_bad6_cc053586aa5d.slice/crio-1aa034b067259e59e8afbba5516953dc1353d596752f29fa9b02ecd4bfe5c4dd WatchSource:0}: Error finding container 1aa034b067259e59e8afbba5516953dc1353d596752f29fa9b02ecd4bfe5c4dd: Status 404 returned error can't find the container with id 1aa034b067259e59e8afbba5516953dc1353d596752f29fa9b02ecd4bfe5c4dd Nov 02 06:22:51.838357 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:22:51.838311 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjpn6\" (UniqueName: \"kubernetes.io/projected/70c7215c-d8bb-4bbd-9085-a1e15c099e6a-kube-api-access-jjpn6\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:22:51.838357 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:22:51.838344 5916 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70c7215c-d8bb-4bbd-9085-a1e15c099e6a-metrics-certs\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:22:51.838357 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:22:51.838359 5916 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/70c7215c-d8bb-4bbd-9085-a1e15c099e6a-webhook-cert\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:22:51.896091 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:22:51.896054 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-fnf5n"] Nov 02 06:22:51.903697 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:22:51.903663 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod7043e9d5_e26f_426a_b480_9f9880da0962.slice/crio-4ac1887a1d070a13287d3881415d3f61db792bb9a8736ac96ea401b88ae39666 WatchSource:0}: Error finding container 4ac1887a1d070a13287d3881415d3f61db792bb9a8736ac96ea401b88ae39666: Status 404 returned error can't find the container with id 4ac1887a1d070a13287d3881415d3f61db792bb9a8736ac96ea401b88ae39666 Nov 02 06:22:52.115203 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:22:52.115116 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-fnf5n" event={"ID":"7043e9d5-e26f-426a-b480-9f9880da0962","Type":"ContainerStarted","Data":"4ac1887a1d070a13287d3881415d3f61db792bb9a8736ac96ea401b88ae39666"} Nov 02 06:22:52.121775 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:22:52.121736 5916 generic.go:356] "Generic (PLEG): container finished" podID="70c7215c-d8bb-4bbd-9085-a1e15c099e6a" containerID="fe16f504382fa4cb19725817e93d0485026a8b66a3412ea369ac96f64f26c196" exitCode=0 Nov 02 06:22:52.121956 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:22:52.121935 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-vv9bd" Nov 02 06:22:52.122021 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:22:52.121963 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-vv9bd" event={"ID":"70c7215c-d8bb-4bbd-9085-a1e15c099e6a","Type":"ContainerDied","Data":"fe16f504382fa4cb19725817e93d0485026a8b66a3412ea369ac96f64f26c196"} Nov 02 06:22:52.122069 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:22:52.122031 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-vv9bd" event={"ID":"70c7215c-d8bb-4bbd-9085-a1e15c099e6a","Type":"ContainerDied","Data":"9cc3f8e74238f564f6890d653dadb112b8314616665fafc9aeaa341416f18535"} Nov 02 06:22:52.122069 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:22:52.122059 5916 scope.go:122] "RemoveContainer" containerID="fe16f504382fa4cb19725817e93d0485026a8b66a3412ea369ac96f64f26c196" Nov 02 06:22:52.124531 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:22:52.124498 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-ffx9j" event={"ID":"100eebf9-dc20-403c-bad6-cc053586aa5d","Type":"ContainerStarted","Data":"1aa034b067259e59e8afbba5516953dc1353d596752f29fa9b02ecd4bfe5c4dd"} Nov 02 06:22:52.146899 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:22:52.146880 5916 scope.go:122] "RemoveContainer" containerID="fe16f504382fa4cb19725817e93d0485026a8b66a3412ea369ac96f64f26c196" Nov 02 06:22:52.147302 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:22:52.147274 5916 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe16f504382fa4cb19725817e93d0485026a8b66a3412ea369ac96f64f26c196\": container with ID starting with fe16f504382fa4cb19725817e93d0485026a8b66a3412ea369ac96f64f26c196 not found: ID does not exist" containerID="fe16f504382fa4cb19725817e93d0485026a8b66a3412ea369ac96f64f26c196" Nov 02 06:22:52.147414 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:22:52.147314 5916 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe16f504382fa4cb19725817e93d0485026a8b66a3412ea369ac96f64f26c196"} err="failed to get container status \"fe16f504382fa4cb19725817e93d0485026a8b66a3412ea369ac96f64f26c196\": rpc error: code = NotFound desc = could not find container \"fe16f504382fa4cb19725817e93d0485026a8b66a3412ea369ac96f64f26c196\": container with ID starting with fe16f504382fa4cb19725817e93d0485026a8b66a3412ea369ac96f64f26c196 not found: ID does not exist" Nov 02 06:22:52.276001 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:22:52.275967 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-vv9bd"] Nov 02 06:22:52.282041 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:22:52.282009 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-vv9bd"] Nov 02 06:22:53.846931 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:22:53.846829 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70c7215c-d8bb-4bbd-9085-a1e15c099e6a" path="/var/lib/kubelet/pods/70c7215c-d8bb-4bbd-9085-a1e15c099e6a/volumes" Nov 02 06:22:56.161010 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:22:56.160972 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-ffx9j" event={"ID":"100eebf9-dc20-403c-bad6-cc053586aa5d","Type":"ContainerStarted","Data":"488858f876e08965172b2ecc6ade736ab9c67b2c5b75158e6f74c1790f9ed2fc"} Nov 02 06:22:56.161463 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:22:56.161173 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-ffx9j" Nov 02 06:22:56.271294 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:22:56.271235 5916 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-ffx9j" podStartSLOduration=1.232965165 podStartE2EDuration="5.271217946s" totalImagesPullingTime="4.038252781s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:22:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:22:56.268888406 +0000 UTC m=+364.540708798" watchObservedRunningTime="2026-11-02 06:22:56.271217946 +0000 UTC m=+364.543038384" Nov 02 06:22:58.175342 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:22:58.175305 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-fnf5n" event={"ID":"7043e9d5-e26f-426a-b480-9f9880da0962","Type":"ContainerStarted","Data":"e830ab4199a8558bbf9853c79fe8bf7fe29a0f23a05bfaf28cc0722757fc9c3c"} Nov 02 06:22:58.175771 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:22:58.175388 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-fnf5n" Nov 02 06:22:58.282611 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:22:58.282555 5916 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-fnf5n" podStartSLOduration=1.733803044 podStartE2EDuration="7.282538723s" totalImagesPullingTime="5.548735679s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:22:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:22:58.281772506 +0000 UTC m=+366.553592900" watchObservedRunningTime="2026-11-02 06:22:58.282538723 +0000 UTC m=+366.554359117" Nov 02 06:23:00.188540 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:23:00.188503 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-68wpm" event={"ID":"23e396f1-81ba-4a29-8771-4f7385b52577","Type":"ContainerStarted","Data":"0971fded3bfa57354de271b4dfe222460464c1ad62ad9ba60c559f595d189b10"} Nov 02 06:23:00.188994 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:23:00.188663 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-68wpm" Nov 02 06:23:00.296390 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:23:00.296338 5916 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-68wpm" podStartSLOduration=2.704929946 podStartE2EDuration="18.296322224s" totalImagesPullingTime="15.591392278s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:22:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:23:00.29544986 +0000 UTC m=+368.567270255" watchObservedRunningTime="2026-11-02 06:23:00.296322224 +0000 UTC m=+368.568142609" Nov 02 06:23:11.199621 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:23:11.199579 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-68wpm" Nov 02 06:23:19.894025 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:23:19.893985 5916 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p98cq" containerName="extractor" Nov 02 06:23:27.174383 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:23:27.174343 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-ffx9j" Nov 02 06:23:28.211375 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:23:28.211333 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-55dccdb8c8-tnt9m"] Nov 02 06:23:28.211944 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:23:28.211777 5916 cpu_manager.go:414] "containerMap: removing container" podUID="70c7215c-d8bb-4bbd-9085-a1e15c099e6a" containerName="manager" Nov 02 06:23:28.211944 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:23:28.211790 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70c7215c-d8bb-4bbd-9085-a1e15c099e6a" containerName="manager" Nov 02 06:23:28.211944 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:23:28.211860 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="70c7215c-d8bb-4bbd-9085-a1e15c099e6a" containerName="manager" Nov 02 06:23:28.265171 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:23:28.265138 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-55dccdb8c8-tnt9m" Nov 02 06:23:28.267435 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:23:28.267405 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-55dccdb8c8-tnt9m"] Nov 02 06:23:28.297474 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:23:28.297438 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdf22\" (UniqueName: \"kubernetes.io/projected/18e22bc3-85bf-4a30-9dee-9941b6ee9230-kube-api-access-tdf22\") pod \"authorino-55dccdb8c8-tnt9m\" (UID: \"18e22bc3-85bf-4a30-9dee-9941b6ee9230\") " pod="kuadrant-system/authorino-55dccdb8c8-tnt9m" Nov 02 06:23:28.313806 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:23:28.313770 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-x84wk\"" Nov 02 06:23:28.398043 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:23:28.398011 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdf22\" (UniqueName: \"kubernetes.io/projected/18e22bc3-85bf-4a30-9dee-9941b6ee9230-kube-api-access-tdf22\") pod \"authorino-55dccdb8c8-tnt9m\" (UID: \"18e22bc3-85bf-4a30-9dee-9941b6ee9230\") " pod="kuadrant-system/authorino-55dccdb8c8-tnt9m" Nov 02 06:23:28.451150 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:23:28.451126 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdf22\" (UniqueName: \"kubernetes.io/projected/18e22bc3-85bf-4a30-9dee-9941b6ee9230-kube-api-access-tdf22\") pod \"authorino-55dccdb8c8-tnt9m\" (UID: \"18e22bc3-85bf-4a30-9dee-9941b6ee9230\") " pod="kuadrant-system/authorino-55dccdb8c8-tnt9m" Nov 02 06:23:28.580517 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:23:28.580480 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-55dccdb8c8-tnt9m" Nov 02 06:23:28.681060 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:23:28.681026 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-55dccdb8c8-tnt9m"] Nov 02 06:23:28.707868 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:23:28.707801 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5f4b77b644-jbqtb"] Nov 02 06:23:28.712720 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:23:28.712696 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5f4b77b644-jbqtb" Nov 02 06:23:28.763416 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:23:28.763377 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Nov 02 06:23:28.765550 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:23:28.765522 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5f4b77b644-jbqtb"] Nov 02 06:23:28.802423 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:23:28.802335 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9fd1afa1-0f94-4c7d-a80d-ca653be2ba71-tls-cert\") pod \"authorino-5f4b77b644-jbqtb\" (UID: \"9fd1afa1-0f94-4c7d-a80d-ca653be2ba71\") " pod="kuadrant-system/authorino-5f4b77b644-jbqtb" Nov 02 06:23:28.802582 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:23:28.802479 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmk2n\" (UniqueName: \"kubernetes.io/projected/9fd1afa1-0f94-4c7d-a80d-ca653be2ba71-kube-api-access-vmk2n\") pod \"authorino-5f4b77b644-jbqtb\" (UID: \"9fd1afa1-0f94-4c7d-a80d-ca653be2ba71\") " pod="kuadrant-system/authorino-5f4b77b644-jbqtb" Nov 02 06:23:28.903294 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:23:28.903256 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9fd1afa1-0f94-4c7d-a80d-ca653be2ba71-tls-cert\") pod \"authorino-5f4b77b644-jbqtb\" (UID: \"9fd1afa1-0f94-4c7d-a80d-ca653be2ba71\") " pod="kuadrant-system/authorino-5f4b77b644-jbqtb" Nov 02 06:23:28.903473 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:23:28.903353 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmk2n\" (UniqueName: \"kubernetes.io/projected/9fd1afa1-0f94-4c7d-a80d-ca653be2ba71-kube-api-access-vmk2n\") pod \"authorino-5f4b77b644-jbqtb\" (UID: \"9fd1afa1-0f94-4c7d-a80d-ca653be2ba71\") " pod="kuadrant-system/authorino-5f4b77b644-jbqtb" Nov 02 06:23:28.905668 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:23:28.905639 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9fd1afa1-0f94-4c7d-a80d-ca653be2ba71-tls-cert\") pod \"authorino-5f4b77b644-jbqtb\" (UID: \"9fd1afa1-0f94-4c7d-a80d-ca653be2ba71\") " pod="kuadrant-system/authorino-5f4b77b644-jbqtb" Nov 02 06:23:28.945143 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:23:28.945115 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-55dccdb8c8-tnt9m"] Nov 02 06:23:28.951599 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:23:28.951570 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod18e22bc3_85bf_4a30_9dee_9941b6ee9230.slice/crio-b5719256a4815440a8408aea5167cd9938467f52d8735ab69ab1996b731af6cd WatchSource:0}: Error finding container b5719256a4815440a8408aea5167cd9938467f52d8735ab69ab1996b731af6cd: Status 404 returned error can't find the container with id b5719256a4815440a8408aea5167cd9938467f52d8735ab69ab1996b731af6cd Nov 02 06:23:28.956248 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:23:28.956224 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmk2n\" (UniqueName: \"kubernetes.io/projected/9fd1afa1-0f94-4c7d-a80d-ca653be2ba71-kube-api-access-vmk2n\") pod \"authorino-5f4b77b644-jbqtb\" (UID: \"9fd1afa1-0f94-4c7d-a80d-ca653be2ba71\") " pod="kuadrant-system/authorino-5f4b77b644-jbqtb" Nov 02 06:23:29.025526 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:23:29.025497 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5f4b77b644-jbqtb" Nov 02 06:23:29.183746 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:23:29.183708 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-fnf5n" Nov 02 06:23:29.346564 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:23:29.346470 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-55dccdb8c8-tnt9m" event={"ID":"18e22bc3-85bf-4a30-9dee-9941b6ee9230","Type":"ContainerStarted","Data":"b5719256a4815440a8408aea5167cd9938467f52d8735ab69ab1996b731af6cd"} Nov 02 06:23:29.348417 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:23:29.348390 5916 generic.go:356] "Generic (PLEG): container finished" podID="fccb2c65-8f82-4808-bb0f-7d9333d8932a" containerID="0433bc04f4a45475241fe4622439501b58a25a1db75351c76111fbbbbe571ef6" exitCode=0 Nov 02 06:23:29.348572 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:23:29.348457 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-44p69" event={"ID":"fccb2c65-8f82-4808-bb0f-7d9333d8932a","Type":"ContainerDied","Data":"0433bc04f4a45475241fe4622439501b58a25a1db75351c76111fbbbbe571ef6"} Nov 02 06:23:29.348840 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:23:29.348825 5916 scope.go:122] "RemoveContainer" containerID="0433bc04f4a45475241fe4622439501b58a25a1db75351c76111fbbbbe571ef6" Nov 02 06:23:29.350118 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:23:29.350087 5916 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 06:23:29.350210 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:23:29.350169 5916 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 06:23:29.350247 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:23:29.350216 5916 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 06:23:29.400792 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:23:29.400769 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5f4b77b644-jbqtb"] Nov 02 06:23:29.407024 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:23:29.406999 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9fd1afa1_0f94_4c7d_a80d_ca653be2ba71.slice/crio-15119b6ae80e86d1819418cc6623b858e9c42c579dfe00e33f0eb05e5578174c WatchSource:0}: Error finding container 15119b6ae80e86d1819418cc6623b858e9c42c579dfe00e33f0eb05e5578174c: Status 404 returned error can't find the container with id 15119b6ae80e86d1819418cc6623b858e9c42c579dfe00e33f0eb05e5578174c Nov 02 06:23:29.980916 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:23:29.980858 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-44p69" Nov 02 06:23:30.358372 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:23:30.358277 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5f4b77b644-jbqtb" event={"ID":"9fd1afa1-0f94-4c7d-a80d-ca653be2ba71","Type":"ContainerStarted","Data":"15119b6ae80e86d1819418cc6623b858e9c42c579dfe00e33f0eb05e5578174c"} Nov 02 06:23:30.360287 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:23:30.360252 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-44p69" event={"ID":"fccb2c65-8f82-4808-bb0f-7d9333d8932a","Type":"ContainerStarted","Data":"2cfc1c5010d443e621858a1bb05a61b188a5c1e5e85ae7718eed002e73038e0b"} Nov 02 06:23:30.735307 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:23:30.735225 5916 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-44p69" Nov 02 06:23:30.741926 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:23:30.741821 5916 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-44p69 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body= Nov 02 06:23:30.742435 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:23:30.741898 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-44p69" podUID="fccb2c65-8f82-4808-bb0f-7d9333d8932a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" Nov 02 06:23:31.380764 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:23:31.380712 5916 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-55dccdb8c8-tnt9m" podUID="18e22bc3-85bf-4a30-9dee-9941b6ee9230" containerName="authorino" containerID="cri-o://42a49378e193b992edf4606d62e4fa8d07d867e96b4ced16262560560842bdd7" gracePeriod=30 Nov 02 06:23:31.380764 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:23:31.380656 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-55dccdb8c8-tnt9m" event={"ID":"18e22bc3-85bf-4a30-9dee-9941b6ee9230","Type":"ContainerStarted","Data":"42a49378e193b992edf4606d62e4fa8d07d867e96b4ced16262560560842bdd7"} Nov 02 06:23:31.383947 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:23:31.383902 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5f4b77b644-jbqtb" event={"ID":"9fd1afa1-0f94-4c7d-a80d-ca653be2ba71","Type":"ContainerStarted","Data":"5f55b3ce6b27f9a29b79d90fbd56c79534c780407513a68ff4e069b1ae0e6964"} Nov 02 06:23:31.487929 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:23:31.487867 5916 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-55dccdb8c8-tnt9m" podStartSLOduration=1.958721577 podStartE2EDuration="3.487853566s" totalImagesPullingTime="1.529131989s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:23:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:23:31.487010321 +0000 UTC m=+399.758830714" watchObservedRunningTime="2026-11-02 06:23:31.487853566 +0000 UTC m=+399.759673959" Nov 02 06:23:31.601014 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:23:31.600938 5916 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-5f4b77b644-jbqtb" podStartSLOduration=2.112210324 podStartE2EDuration="3.600899338s" totalImagesPullingTime="1.488689014s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:23:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:23:31.598306308 +0000 UTC m=+399.870126704" watchObservedRunningTime="2026-11-02 06:23:31.600899338 +0000 UTC m=+399.872719733" Nov 02 06:23:31.698621 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:23:31.698591 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-55dccdb8c8-tnt9m" Nov 02 06:23:31.749942 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:23:31.749110 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18e22bc3-85bf-4a30-9dee-9941b6ee9230-kube-api-access-tdf22\" (UniqueName: \"kubernetes.io/projected/18e22bc3-85bf-4a30-9dee-9941b6ee9230-kube-api-access-tdf22\") pod \"18e22bc3-85bf-4a30-9dee-9941b6ee9230\" (UID: \"18e22bc3-85bf-4a30-9dee-9941b6ee9230\") " Nov 02 06:23:31.750163 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:23:31.750137 5916 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-44p69 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body= Nov 02 06:23:31.750261 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:23:31.750190 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-44p69" podUID="fccb2c65-8f82-4808-bb0f-7d9333d8932a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" Nov 02 06:23:31.754068 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:23:31.754035 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18e22bc3-85bf-4a30-9dee-9941b6ee9230-kube-api-access-tdf22" pod "18e22bc3-85bf-4a30-9dee-9941b6ee9230" (UID: "18e22bc3-85bf-4a30-9dee-9941b6ee9230"). InnerVolumeSpecName "kube-api-access-tdf22". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:23:31.853533 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:23:31.853480 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdf22\" (UniqueName: \"kubernetes.io/projected/18e22bc3-85bf-4a30-9dee-9941b6ee9230-kube-api-access-tdf22\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:23:32.390889 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:23:32.390850 5916 generic.go:356] "Generic (PLEG): container finished" podID="18e22bc3-85bf-4a30-9dee-9941b6ee9230" containerID="42a49378e193b992edf4606d62e4fa8d07d867e96b4ced16262560560842bdd7" exitCode=0 Nov 02 06:23:32.391346 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:23:32.390901 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-55dccdb8c8-tnt9m" Nov 02 06:23:32.391346 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:23:32.390940 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-55dccdb8c8-tnt9m" event={"ID":"18e22bc3-85bf-4a30-9dee-9941b6ee9230","Type":"ContainerDied","Data":"42a49378e193b992edf4606d62e4fa8d07d867e96b4ced16262560560842bdd7"} Nov 02 06:23:32.391346 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:23:32.390993 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-55dccdb8c8-tnt9m" event={"ID":"18e22bc3-85bf-4a30-9dee-9941b6ee9230","Type":"ContainerDied","Data":"b5719256a4815440a8408aea5167cd9938467f52d8735ab69ab1996b731af6cd"} Nov 02 06:23:32.391346 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:23:32.391015 5916 scope.go:122] "RemoveContainer" containerID="42a49378e193b992edf4606d62e4fa8d07d867e96b4ced16262560560842bdd7" Nov 02 06:23:32.400105 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:23:32.400088 5916 scope.go:122] "RemoveContainer" containerID="42a49378e193b992edf4606d62e4fa8d07d867e96b4ced16262560560842bdd7" Nov 02 06:23:32.400373 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:23:32.400353 5916 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42a49378e193b992edf4606d62e4fa8d07d867e96b4ced16262560560842bdd7\": container with ID starting with 42a49378e193b992edf4606d62e4fa8d07d867e96b4ced16262560560842bdd7 not found: ID does not exist" containerID="42a49378e193b992edf4606d62e4fa8d07d867e96b4ced16262560560842bdd7" Nov 02 06:23:32.400419 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:23:32.400382 5916 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42a49378e193b992edf4606d62e4fa8d07d867e96b4ced16262560560842bdd7"} err="failed to get container status \"42a49378e193b992edf4606d62e4fa8d07d867e96b4ced16262560560842bdd7\": rpc error: code = NotFound desc = could not find container \"42a49378e193b992edf4606d62e4fa8d07d867e96b4ced16262560560842bdd7\": container with ID starting with 42a49378e193b992edf4606d62e4fa8d07d867e96b4ced16262560560842bdd7 not found: ID does not exist" Nov 02 06:23:32.553057 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:23:32.553011 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-55dccdb8c8-tnt9m"] Nov 02 06:23:32.554293 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:23:32.554257 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-55dccdb8c8-tnt9m"] Nov 02 06:23:32.745854 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:23:32.745330 5916 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-44p69 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body= Nov 02 06:23:32.745854 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:23:32.745398 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-44p69" podUID="fccb2c65-8f82-4808-bb0f-7d9333d8932a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" Nov 02 06:23:33.750088 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:23:33.750050 5916 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-44p69 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body= Nov 02 06:23:33.750606 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:23:33.750128 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-44p69" podUID="fccb2c65-8f82-4808-bb0f-7d9333d8932a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" Nov 02 06:23:33.853625 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:23:33.853580 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18e22bc3-85bf-4a30-9dee-9941b6ee9230" path="/var/lib/kubelet/pods/18e22bc3-85bf-4a30-9dee-9941b6ee9230/volumes" Nov 02 06:23:34.750071 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:23:34.750023 5916 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-44p69 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body= Nov 02 06:23:34.750275 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:23:34.750089 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-44p69" podUID="fccb2c65-8f82-4808-bb0f-7d9333d8932a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" Nov 02 06:23:35.735098 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:23:35.735068 5916 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-44p69" Nov 02 06:23:35.735494 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:23:35.735476 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-44p69" Nov 02 06:23:35.736087 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:23:35.736070 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-44p69" Nov 02 06:23:49.892442 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:23:49.892393 5916 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p98cq" containerName="extractor" Nov 02 06:24:12.938512 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:24:12.938468 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-cnkkh"] Nov 02 06:24:12.938916 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:24:12.938874 5916 cpu_manager.go:414] "containerMap: removing container" podUID="18e22bc3-85bf-4a30-9dee-9941b6ee9230" containerName="authorino" Nov 02 06:24:12.938916 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:24:12.938886 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18e22bc3-85bf-4a30-9dee-9941b6ee9230" containerName="authorino" Nov 02 06:24:12.939004 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:24:12.938981 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="18e22bc3-85bf-4a30-9dee-9941b6ee9230" containerName="authorino" Nov 02 06:24:12.942421 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:24:12.942398 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-cnkkh" Nov 02 06:24:12.995865 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:24:12.995835 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-cnkkh"] Nov 02 06:24:13.118424 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:24:13.118395 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2e6b0bf4-af81-408e-9e73-5480cf91f065-cert\") pod \"llmisvc-controller-manager-6466c554d8-cnkkh\" (UID: \"2e6b0bf4-af81-408e-9e73-5480cf91f065\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-cnkkh" Nov 02 06:24:13.118630 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:24:13.118555 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqgjm\" (UniqueName: \"kubernetes.io/projected/2e6b0bf4-af81-408e-9e73-5480cf91f065-kube-api-access-hqgjm\") pod \"llmisvc-controller-manager-6466c554d8-cnkkh\" (UID: \"2e6b0bf4-af81-408e-9e73-5480cf91f065\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-cnkkh" Nov 02 06:24:13.219790 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:24:13.219699 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2e6b0bf4-af81-408e-9e73-5480cf91f065-cert\") pod \"llmisvc-controller-manager-6466c554d8-cnkkh\" (UID: \"2e6b0bf4-af81-408e-9e73-5480cf91f065\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-cnkkh" Nov 02 06:24:13.219968 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:24:13.219872 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqgjm\" (UniqueName: \"kubernetes.io/projected/2e6b0bf4-af81-408e-9e73-5480cf91f065-kube-api-access-hqgjm\") pod \"llmisvc-controller-manager-6466c554d8-cnkkh\" (UID: \"2e6b0bf4-af81-408e-9e73-5480cf91f065\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-cnkkh" Nov 02 06:24:13.222193 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:24:13.222173 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2e6b0bf4-af81-408e-9e73-5480cf91f065-cert\") pod \"llmisvc-controller-manager-6466c554d8-cnkkh\" (UID: \"2e6b0bf4-af81-408e-9e73-5480cf91f065\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-cnkkh" Nov 02 06:24:13.273235 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:24:13.273207 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqgjm\" (UniqueName: \"kubernetes.io/projected/2e6b0bf4-af81-408e-9e73-5480cf91f065-kube-api-access-hqgjm\") pod \"llmisvc-controller-manager-6466c554d8-cnkkh\" (UID: \"2e6b0bf4-af81-408e-9e73-5480cf91f065\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-cnkkh" Nov 02 06:24:13.555217 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:24:13.555131 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-cnkkh" Nov 02 06:24:13.922963 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:24:13.922926 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-cnkkh"] Nov 02 06:24:13.929593 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:24:13.929563 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e6b0bf4_af81_408e_9e73_5480cf91f065.slice/crio-a7851ba855fa4f55f07d55573706da238059f4423c021f65f7b8d9b7d24328d5 WatchSource:0}: Error finding container a7851ba855fa4f55f07d55573706da238059f4423c021f65f7b8d9b7d24328d5: Status 404 returned error can't find the container with id a7851ba855fa4f55f07d55573706da238059f4423c021f65f7b8d9b7d24328d5 Nov 02 06:24:14.623234 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:24:14.623192 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-cnkkh" event={"ID":"2e6b0bf4-af81-408e-9e73-5480cf91f065","Type":"ContainerStarted","Data":"a7851ba855fa4f55f07d55573706da238059f4423c021f65f7b8d9b7d24328d5"} Nov 02 06:24:15.631174 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:24:15.631138 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-cnkkh" event={"ID":"2e6b0bf4-af81-408e-9e73-5480cf91f065","Type":"ContainerStarted","Data":"e8f5c6fd434133d57ec63f3ec07f6441f9b81d6b187346182203d0257f9eb0c5"} Nov 02 06:24:15.631572 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:24:15.631271 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-cnkkh" Nov 02 06:24:15.738367 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:24:15.738312 5916 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-cnkkh" podStartSLOduration=2.938343012 podStartE2EDuration="3.738297265s" totalImagesPullingTime="799.954253ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:24:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:24:15.736773433 +0000 UTC m=+444.008593853" watchObservedRunningTime="2026-11-02 06:24:15.738297265 +0000 UTC m=+444.010117671" Nov 02 06:24:19.893588 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:24:19.893552 5916 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p98cq" containerName="extractor" Nov 02 06:24:46.638997 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:24:46.638923 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-cnkkh" Nov 02 06:24:46.793401 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:24:46.793364 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-fnf5n"] Nov 02 06:24:46.793648 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:24:46.793603 5916 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-fnf5n" podUID="7043e9d5-e26f-426a-b480-9f9880da0962" containerName="manager" containerID="cri-o://e830ab4199a8558bbf9853c79fe8bf7fe29a0f23a05bfaf28cc0722757fc9c3c" gracePeriod=30 Nov 02 06:24:47.040174 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:24:47.040149 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-fnf5n" Nov 02 06:24:47.120635 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:24:47.120608 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7043e9d5-e26f-426a-b480-9f9880da0962-cert\" (UniqueName: \"kubernetes.io/secret/7043e9d5-e26f-426a-b480-9f9880da0962-cert\") pod \"7043e9d5-e26f-426a-b480-9f9880da0962\" (UID: \"7043e9d5-e26f-426a-b480-9f9880da0962\") " Nov 02 06:24:47.120796 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:24:47.120647 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7043e9d5-e26f-426a-b480-9f9880da0962-kube-api-access-c4hnz\" (UniqueName: \"kubernetes.io/projected/7043e9d5-e26f-426a-b480-9f9880da0962-kube-api-access-c4hnz\") pod \"7043e9d5-e26f-426a-b480-9f9880da0962\" (UID: \"7043e9d5-e26f-426a-b480-9f9880da0962\") " Nov 02 06:24:47.122745 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:24:47.122720 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7043e9d5-e26f-426a-b480-9f9880da0962-cert" pod "7043e9d5-e26f-426a-b480-9f9880da0962" (UID: "7043e9d5-e26f-426a-b480-9f9880da0962"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:24:47.122831 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:24:47.122744 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7043e9d5-e26f-426a-b480-9f9880da0962-kube-api-access-c4hnz" pod "7043e9d5-e26f-426a-b480-9f9880da0962" (UID: "7043e9d5-e26f-426a-b480-9f9880da0962"). InnerVolumeSpecName "kube-api-access-c4hnz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:24:47.222129 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:24:47.222042 5916 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7043e9d5-e26f-426a-b480-9f9880da0962-cert\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:24:47.222129 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:24:47.222077 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4hnz\" (UniqueName: \"kubernetes.io/projected/7043e9d5-e26f-426a-b480-9f9880da0962-kube-api-access-c4hnz\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:24:47.806461 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:24:47.806424 5916 generic.go:356] "Generic (PLEG): container finished" podID="7043e9d5-e26f-426a-b480-9f9880da0962" containerID="e830ab4199a8558bbf9853c79fe8bf7fe29a0f23a05bfaf28cc0722757fc9c3c" exitCode=0 Nov 02 06:24:47.806937 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:24:47.806481 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-fnf5n" Nov 02 06:24:47.806937 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:24:47.806516 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-fnf5n" event={"ID":"7043e9d5-e26f-426a-b480-9f9880da0962","Type":"ContainerDied","Data":"e830ab4199a8558bbf9853c79fe8bf7fe29a0f23a05bfaf28cc0722757fc9c3c"} Nov 02 06:24:47.806937 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:24:47.806599 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-fnf5n" event={"ID":"7043e9d5-e26f-426a-b480-9f9880da0962","Type":"ContainerDied","Data":"4ac1887a1d070a13287d3881415d3f61db792bb9a8736ac96ea401b88ae39666"} Nov 02 06:24:47.806937 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:24:47.806624 5916 scope.go:122] "RemoveContainer" containerID="e830ab4199a8558bbf9853c79fe8bf7fe29a0f23a05bfaf28cc0722757fc9c3c" Nov 02 06:24:47.818961 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:24:47.818825 5916 scope.go:122] "RemoveContainer" containerID="e830ab4199a8558bbf9853c79fe8bf7fe29a0f23a05bfaf28cc0722757fc9c3c" Nov 02 06:24:47.819359 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:24:47.819335 5916 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e830ab4199a8558bbf9853c79fe8bf7fe29a0f23a05bfaf28cc0722757fc9c3c\": container with ID starting with e830ab4199a8558bbf9853c79fe8bf7fe29a0f23a05bfaf28cc0722757fc9c3c not found: ID does not exist" containerID="e830ab4199a8558bbf9853c79fe8bf7fe29a0f23a05bfaf28cc0722757fc9c3c" Nov 02 06:24:47.819421 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:24:47.819380 5916 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e830ab4199a8558bbf9853c79fe8bf7fe29a0f23a05bfaf28cc0722757fc9c3c"} err="failed to get container status \"e830ab4199a8558bbf9853c79fe8bf7fe29a0f23a05bfaf28cc0722757fc9c3c\": rpc error: code = NotFound desc = could not find container \"e830ab4199a8558bbf9853c79fe8bf7fe29a0f23a05bfaf28cc0722757fc9c3c\": container with ID starting with e830ab4199a8558bbf9853c79fe8bf7fe29a0f23a05bfaf28cc0722757fc9c3c not found: ID does not exist" Nov 02 06:24:47.958731 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:24:47.958699 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-fnf5n"] Nov 02 06:24:48.004445 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:24:48.004417 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-fnf5n"] Nov 02 06:24:49.840321 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:24:49.840292 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7043e9d5-e26f-426a-b480-9f9880da0962" path="/var/lib/kubelet/pods/7043e9d5-e26f-426a-b480-9f9880da0962/volumes" Nov 02 06:24:49.893149 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:24:49.893108 5916 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p98cq" containerName="extractor" Nov 02 06:25:19.893996 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:25:19.893950 5916 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p98cq" containerName="extractor" Nov 02 06:25:49.893546 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:25:49.893503 5916 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p98cq" containerName="extractor" Nov 02 06:26:02.592255 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:26:02.592176 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7cc5b4bb47-5fkgj"] Nov 02 06:26:02.592696 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:26:02.592656 5916 cpu_manager.go:414] "containerMap: removing container" podUID="7043e9d5-e26f-426a-b480-9f9880da0962" containerName="manager" Nov 02 06:26:02.592696 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:26:02.592671 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7043e9d5-e26f-426a-b480-9f9880da0962" containerName="manager" Nov 02 06:26:02.592774 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:26:02.592766 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="7043e9d5-e26f-426a-b480-9f9880da0962" containerName="manager" Nov 02 06:26:02.596024 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:26:02.596004 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7cc5b4bb47-5fkgj" Nov 02 06:26:02.609241 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:26:02.609216 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49mnc\" (UniqueName: \"kubernetes.io/projected/f1400f25-cbd3-4a40-9d3c-abf294704a13-kube-api-access-49mnc\") pod \"authorino-7cc5b4bb47-5fkgj\" (UID: \"f1400f25-cbd3-4a40-9d3c-abf294704a13\") " pod="kuadrant-system/authorino-7cc5b4bb47-5fkgj" Nov 02 06:26:02.609383 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:26:02.609362 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f1400f25-cbd3-4a40-9d3c-abf294704a13-tls-cert\") pod \"authorino-7cc5b4bb47-5fkgj\" (UID: \"f1400f25-cbd3-4a40-9d3c-abf294704a13\") " pod="kuadrant-system/authorino-7cc5b4bb47-5fkgj" Nov 02 06:26:02.656494 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:26:02.656456 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7cc5b4bb47-5fkgj"] Nov 02 06:26:02.710387 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:26:02.710354 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49mnc\" (UniqueName: \"kubernetes.io/projected/f1400f25-cbd3-4a40-9d3c-abf294704a13-kube-api-access-49mnc\") pod \"authorino-7cc5b4bb47-5fkgj\" (UID: \"f1400f25-cbd3-4a40-9d3c-abf294704a13\") " pod="kuadrant-system/authorino-7cc5b4bb47-5fkgj" Nov 02 06:26:02.710566 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:26:02.710446 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f1400f25-cbd3-4a40-9d3c-abf294704a13-tls-cert\") pod \"authorino-7cc5b4bb47-5fkgj\" (UID: \"f1400f25-cbd3-4a40-9d3c-abf294704a13\") " pod="kuadrant-system/authorino-7cc5b4bb47-5fkgj" Nov 02 06:26:02.713087 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:26:02.713060 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f1400f25-cbd3-4a40-9d3c-abf294704a13-tls-cert\") pod \"authorino-7cc5b4bb47-5fkgj\" (UID: \"f1400f25-cbd3-4a40-9d3c-abf294704a13\") " pod="kuadrant-system/authorino-7cc5b4bb47-5fkgj" Nov 02 06:26:02.764712 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:26:02.764667 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49mnc\" (UniqueName: \"kubernetes.io/projected/f1400f25-cbd3-4a40-9d3c-abf294704a13-kube-api-access-49mnc\") pod \"authorino-7cc5b4bb47-5fkgj\" (UID: \"f1400f25-cbd3-4a40-9d3c-abf294704a13\") " pod="kuadrant-system/authorino-7cc5b4bb47-5fkgj" Nov 02 06:26:02.908324 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:26:02.908291 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7cc5b4bb47-5fkgj" Nov 02 06:26:03.272514 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:26:03.272431 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7cc5b4bb47-5fkgj"] Nov 02 06:26:03.278974 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:26:03.278941 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf1400f25_cbd3_4a40_9d3c_abf294704a13.slice/crio-7c70a41ffbe01dc2654a23c233e1e835018ffdc35aef792ee0a12f44915d9a5d WatchSource:0}: Error finding container 7c70a41ffbe01dc2654a23c233e1e835018ffdc35aef792ee0a12f44915d9a5d: Status 404 returned error can't find the container with id 7c70a41ffbe01dc2654a23c233e1e835018ffdc35aef792ee0a12f44915d9a5d Nov 02 06:26:04.254360 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:26:04.254322 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7cc5b4bb47-5fkgj" event={"ID":"f1400f25-cbd3-4a40-9d3c-abf294704a13","Type":"ContainerStarted","Data":"7c70a41ffbe01dc2654a23c233e1e835018ffdc35aef792ee0a12f44915d9a5d"} Nov 02 06:26:05.261739 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:26:05.261645 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7cc5b4bb47-5fkgj" event={"ID":"f1400f25-cbd3-4a40-9d3c-abf294704a13","Type":"ContainerStarted","Data":"d6772d90b79049e9ac0dec384d16383c38b27180ed7c3fdbc111d51d44a2b2e3"} Nov 02 06:26:05.371823 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:26:05.371767 5916 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-7cc5b4bb47-5fkgj" podStartSLOduration=1.669895013 podStartE2EDuration="3.371751402s" totalImagesPullingTime="1.701856389s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:26:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:26:05.369333654 +0000 UTC m=+553.641154047" watchObservedRunningTime="2026-11-02 06:26:05.371751402 +0000 UTC m=+553.643571826" Nov 02 06:26:05.418812 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:26:05.418774 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5f4b77b644-jbqtb"] Nov 02 06:26:05.419057 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:26:05.419028 5916 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-5f4b77b644-jbqtb" podUID="9fd1afa1-0f94-4c7d-a80d-ca653be2ba71" containerName="authorino" containerID="cri-o://5f55b3ce6b27f9a29b79d90fbd56c79534c780407513a68ff4e069b1ae0e6964" gracePeriod=30 Nov 02 06:26:05.663595 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:26:05.663568 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5f4b77b644-jbqtb" Nov 02 06:26:05.730354 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:26:05.730321 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9fd1afa1-0f94-4c7d-a80d-ca653be2ba71-tls-cert\" (UniqueName: \"kubernetes.io/secret/9fd1afa1-0f94-4c7d-a80d-ca653be2ba71-tls-cert\") pod \"9fd1afa1-0f94-4c7d-a80d-ca653be2ba71\" (UID: \"9fd1afa1-0f94-4c7d-a80d-ca653be2ba71\") " Nov 02 06:26:05.730540 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:26:05.730375 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fd1afa1-0f94-4c7d-a80d-ca653be2ba71-kube-api-access-vmk2n\" (UniqueName: \"kubernetes.io/projected/9fd1afa1-0f94-4c7d-a80d-ca653be2ba71-kube-api-access-vmk2n\") pod \"9fd1afa1-0f94-4c7d-a80d-ca653be2ba71\" (UID: \"9fd1afa1-0f94-4c7d-a80d-ca653be2ba71\") " Nov 02 06:26:05.732477 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:26:05.732448 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fd1afa1-0f94-4c7d-a80d-ca653be2ba71-kube-api-access-vmk2n" pod "9fd1afa1-0f94-4c7d-a80d-ca653be2ba71" (UID: "9fd1afa1-0f94-4c7d-a80d-ca653be2ba71"). InnerVolumeSpecName "kube-api-access-vmk2n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:26:05.743492 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:26:05.743464 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fd1afa1-0f94-4c7d-a80d-ca653be2ba71-tls-cert" pod "9fd1afa1-0f94-4c7d-a80d-ca653be2ba71" (UID: "9fd1afa1-0f94-4c7d-a80d-ca653be2ba71"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:26:05.831316 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:26:05.831224 5916 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9fd1afa1-0f94-4c7d-a80d-ca653be2ba71-tls-cert\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:26:05.831316 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:26:05.831253 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmk2n\" (UniqueName: \"kubernetes.io/projected/9fd1afa1-0f94-4c7d-a80d-ca653be2ba71-kube-api-access-vmk2n\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:26:06.268640 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:26:06.268600 5916 generic.go:356] "Generic (PLEG): container finished" podID="9fd1afa1-0f94-4c7d-a80d-ca653be2ba71" containerID="5f55b3ce6b27f9a29b79d90fbd56c79534c780407513a68ff4e069b1ae0e6964" exitCode=0 Nov 02 06:26:06.269179 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:26:06.268669 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5f4b77b644-jbqtb" Nov 02 06:26:06.269496 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:26:06.269459 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5f4b77b644-jbqtb" event={"ID":"9fd1afa1-0f94-4c7d-a80d-ca653be2ba71","Type":"ContainerDied","Data":"5f55b3ce6b27f9a29b79d90fbd56c79534c780407513a68ff4e069b1ae0e6964"} Nov 02 06:26:06.269615 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:26:06.269521 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5f4b77b644-jbqtb" event={"ID":"9fd1afa1-0f94-4c7d-a80d-ca653be2ba71","Type":"ContainerDied","Data":"15119b6ae80e86d1819418cc6623b858e9c42c579dfe00e33f0eb05e5578174c"} Nov 02 06:26:06.269615 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:26:06.269539 5916 scope.go:122] "RemoveContainer" containerID="5f55b3ce6b27f9a29b79d90fbd56c79534c780407513a68ff4e069b1ae0e6964" Nov 02 06:26:06.278993 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:26:06.278970 5916 scope.go:122] "RemoveContainer" containerID="5f55b3ce6b27f9a29b79d90fbd56c79534c780407513a68ff4e069b1ae0e6964" Nov 02 06:26:06.279268 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:26:06.279245 5916 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f55b3ce6b27f9a29b79d90fbd56c79534c780407513a68ff4e069b1ae0e6964\": container with ID starting with 5f55b3ce6b27f9a29b79d90fbd56c79534c780407513a68ff4e069b1ae0e6964 not found: ID does not exist" containerID="5f55b3ce6b27f9a29b79d90fbd56c79534c780407513a68ff4e069b1ae0e6964" Nov 02 06:26:06.279349 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:26:06.279283 5916 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f55b3ce6b27f9a29b79d90fbd56c79534c780407513a68ff4e069b1ae0e6964"} err="failed to get container status \"5f55b3ce6b27f9a29b79d90fbd56c79534c780407513a68ff4e069b1ae0e6964\": rpc error: code = NotFound desc = could not find container \"5f55b3ce6b27f9a29b79d90fbd56c79534c780407513a68ff4e069b1ae0e6964\": container with ID starting with 5f55b3ce6b27f9a29b79d90fbd56c79534c780407513a68ff4e069b1ae0e6964 not found: ID does not exist" Nov 02 06:26:06.422070 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:26:06.422011 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5f4b77b644-jbqtb"] Nov 02 06:26:06.425200 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:26:06.425174 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5f4b77b644-jbqtb"] Nov 02 06:26:07.840767 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:26:07.840734 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9fd1afa1-0f94-4c7d-a80d-ca653be2ba71" path="/var/lib/kubelet/pods/9fd1afa1-0f94-4c7d-a80d-ca653be2ba71/volumes" Nov 02 06:26:19.892766 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:26:19.892730 5916 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p98cq" containerName="extractor" Nov 02 06:26:49.893706 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:26:49.893667 5916 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p98cq" containerName="extractor" Nov 02 06:26:51.918478 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:26:51.918442 5916 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-csfhg_17dee72b-2f47-4016-8ffe-3a89934e66ee/console-operator/2.log" Nov 02 06:26:51.924459 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:26:51.924434 5916 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-csfhg_17dee72b-2f47-4016-8ffe-3a89934e66ee/console-operator/2.log" Nov 02 06:27:13.273690 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:13.273650 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-rvhhk"] Nov 02 06:27:13.274585 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:13.274101 5916 cpu_manager.go:414] "containerMap: removing container" podUID="9fd1afa1-0f94-4c7d-a80d-ca653be2ba71" containerName="authorino" Nov 02 06:27:13.274585 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:13.274114 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fd1afa1-0f94-4c7d-a80d-ca653be2ba71" containerName="authorino" Nov 02 06:27:13.274585 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:13.274180 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fd1afa1-0f94-4c7d-a80d-ca653be2ba71" containerName="authorino" Nov 02 06:27:13.277330 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:13.277307 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-rvhhk" Nov 02 06:27:13.326422 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:13.326388 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-6ff85e-openshift-default-dockercfg-f2wx5\"" Nov 02 06:27:13.375765 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:13.375734 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-rvhhk"] Nov 02 06:27:13.417860 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:13.417822 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d74d644-1bbf-4522-8075-43b1f476f656-istio-envoy\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-rvhhk\" (UID: \"8d74d644-1bbf-4522-8075-43b1f476f656\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-rvhhk" Nov 02 06:27:13.418077 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:13.417873 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d74d644-1bbf-4522-8075-43b1f476f656-istiod-ca-cert\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-rvhhk\" (UID: \"8d74d644-1bbf-4522-8075-43b1f476f656\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-rvhhk" Nov 02 06:27:13.418077 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:13.417963 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d74d644-1bbf-4522-8075-43b1f476f656-istio-token\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-rvhhk\" (UID: \"8d74d644-1bbf-4522-8075-43b1f476f656\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-rvhhk" Nov 02 06:27:13.418077 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:13.418001 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d74d644-1bbf-4522-8075-43b1f476f656-workload-socket\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-rvhhk\" (UID: \"8d74d644-1bbf-4522-8075-43b1f476f656\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-rvhhk" Nov 02 06:27:13.418077 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:13.418030 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d74d644-1bbf-4522-8075-43b1f476f656-workload-certs\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-rvhhk\" (UID: \"8d74d644-1bbf-4522-8075-43b1f476f656\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-rvhhk" Nov 02 06:27:13.418283 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:13.418137 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d74d644-1bbf-4522-8075-43b1f476f656-istio-data\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-rvhhk\" (UID: \"8d74d644-1bbf-4522-8075-43b1f476f656\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-rvhhk" Nov 02 06:27:13.418283 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:13.418183 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlj2p\" (UniqueName: \"kubernetes.io/projected/8d74d644-1bbf-4522-8075-43b1f476f656-kube-api-access-mlj2p\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-rvhhk\" (UID: \"8d74d644-1bbf-4522-8075-43b1f476f656\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-rvhhk" Nov 02 06:27:13.418283 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:13.418252 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d74d644-1bbf-4522-8075-43b1f476f656-istio-podinfo\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-rvhhk\" (UID: \"8d74d644-1bbf-4522-8075-43b1f476f656\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-rvhhk" Nov 02 06:27:13.418457 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:13.418302 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d74d644-1bbf-4522-8075-43b1f476f656-credential-socket\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-rvhhk\" (UID: \"8d74d644-1bbf-4522-8075-43b1f476f656\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-rvhhk" Nov 02 06:27:13.519366 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:13.519329 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d74d644-1bbf-4522-8075-43b1f476f656-istio-envoy\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-rvhhk\" (UID: \"8d74d644-1bbf-4522-8075-43b1f476f656\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-rvhhk" Nov 02 06:27:13.519366 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:13.519374 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d74d644-1bbf-4522-8075-43b1f476f656-istiod-ca-cert\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-rvhhk\" (UID: \"8d74d644-1bbf-4522-8075-43b1f476f656\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-rvhhk" Nov 02 06:27:13.519635 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:13.519403 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d74d644-1bbf-4522-8075-43b1f476f656-istio-token\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-rvhhk\" (UID: \"8d74d644-1bbf-4522-8075-43b1f476f656\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-rvhhk" Nov 02 06:27:13.519635 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:13.519434 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d74d644-1bbf-4522-8075-43b1f476f656-workload-socket\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-rvhhk\" (UID: \"8d74d644-1bbf-4522-8075-43b1f476f656\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-rvhhk" Nov 02 06:27:13.519635 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:13.519475 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d74d644-1bbf-4522-8075-43b1f476f656-workload-certs\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-rvhhk\" (UID: \"8d74d644-1bbf-4522-8075-43b1f476f656\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-rvhhk" Nov 02 06:27:13.519635 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:13.519518 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d74d644-1bbf-4522-8075-43b1f476f656-istio-data\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-rvhhk\" (UID: \"8d74d644-1bbf-4522-8075-43b1f476f656\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-rvhhk" Nov 02 06:27:13.519635 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:13.519588 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlj2p\" (UniqueName: \"kubernetes.io/projected/8d74d644-1bbf-4522-8075-43b1f476f656-kube-api-access-mlj2p\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-rvhhk\" (UID: \"8d74d644-1bbf-4522-8075-43b1f476f656\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-rvhhk" Nov 02 06:27:13.519635 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:13.519617 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d74d644-1bbf-4522-8075-43b1f476f656-istio-podinfo\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-rvhhk\" (UID: \"8d74d644-1bbf-4522-8075-43b1f476f656\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-rvhhk" Nov 02 06:27:13.519964 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:13.519653 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d74d644-1bbf-4522-8075-43b1f476f656-credential-socket\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-rvhhk\" (UID: \"8d74d644-1bbf-4522-8075-43b1f476f656\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-rvhhk" Nov 02 06:27:13.519964 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:13.519941 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d74d644-1bbf-4522-8075-43b1f476f656-workload-socket\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-rvhhk\" (UID: \"8d74d644-1bbf-4522-8075-43b1f476f656\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-rvhhk" Nov 02 06:27:13.520225 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:13.520197 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d74d644-1bbf-4522-8075-43b1f476f656-credential-socket\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-rvhhk\" (UID: \"8d74d644-1bbf-4522-8075-43b1f476f656\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-rvhhk" Nov 02 06:27:13.520330 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:13.520253 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d74d644-1bbf-4522-8075-43b1f476f656-workload-certs\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-rvhhk\" (UID: \"8d74d644-1bbf-4522-8075-43b1f476f656\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-rvhhk" Nov 02 06:27:13.520330 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:13.520270 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d74d644-1bbf-4522-8075-43b1f476f656-istio-data\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-rvhhk\" (UID: \"8d74d644-1bbf-4522-8075-43b1f476f656\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-rvhhk" Nov 02 06:27:13.520330 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:13.520286 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d74d644-1bbf-4522-8075-43b1f476f656-istiod-ca-cert\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-rvhhk\" (UID: \"8d74d644-1bbf-4522-8075-43b1f476f656\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-rvhhk" Nov 02 06:27:13.522096 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:13.522066 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d74d644-1bbf-4522-8075-43b1f476f656-istio-envoy\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-rvhhk\" (UID: \"8d74d644-1bbf-4522-8075-43b1f476f656\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-rvhhk" Nov 02 06:27:13.522265 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:13.522249 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d74d644-1bbf-4522-8075-43b1f476f656-istio-podinfo\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-rvhhk\" (UID: \"8d74d644-1bbf-4522-8075-43b1f476f656\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-rvhhk" Nov 02 06:27:13.572438 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:13.572368 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d74d644-1bbf-4522-8075-43b1f476f656-istio-token\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-rvhhk\" (UID: \"8d74d644-1bbf-4522-8075-43b1f476f656\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-rvhhk" Nov 02 06:27:13.572562 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:13.572490 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlj2p\" (UniqueName: \"kubernetes.io/projected/8d74d644-1bbf-4522-8075-43b1f476f656-kube-api-access-mlj2p\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-rvhhk\" (UID: \"8d74d644-1bbf-4522-8075-43b1f476f656\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-rvhhk" Nov 02 06:27:13.590985 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:13.590961 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-rvhhk" Nov 02 06:27:14.014543 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:14.014507 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-rvhhk"] Nov 02 06:27:14.061021 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:27:14.060978 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d74d644_1bbf_4522_8075_43b1f476f656.slice/crio-cce13f840ee4dfcbc5871eb317067e269c8f9da3127842317467be21cf1a084d WatchSource:0}: Error finding container cce13f840ee4dfcbc5871eb317067e269c8f9da3127842317467be21cf1a084d: Status 404 returned error can't find the container with id cce13f840ee4dfcbc5871eb317067e269c8f9da3127842317467be21cf1a084d Nov 02 06:27:14.063220 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:14.063185 5916 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 06:27:14.063315 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:14.063258 5916 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 06:27:14.063315 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:14.063308 5916 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 06:27:14.680203 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:14.680160 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-rvhhk" event={"ID":"8d74d644-1bbf-4522-8075-43b1f476f656","Type":"ContainerStarted","Data":"86a483e176dbc71802a13babaadd925ebc4b9c64710630e13f7af6c9079394e2"} Nov 02 06:27:14.680627 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:14.680227 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-rvhhk" event={"ID":"8d74d644-1bbf-4522-8075-43b1f476f656","Type":"ContainerStarted","Data":"cce13f840ee4dfcbc5871eb317067e269c8f9da3127842317467be21cf1a084d"} Nov 02 06:27:14.835231 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:14.835182 5916 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-rvhhk" podStartSLOduration=1.835168625 podStartE2EDuration="1.835168625s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:27:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:27:14.834868684 +0000 UTC m=+623.106689079" watchObservedRunningTime="2026-11-02 06:27:14.835168625 +0000 UTC m=+623.106989019" Nov 02 06:27:15.591555 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:15.591515 5916 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-rvhhk" Nov 02 06:27:15.597037 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:15.597013 5916 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-rvhhk" Nov 02 06:27:15.686624 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:15.686595 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-rvhhk" Nov 02 06:27:15.687677 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:15.687653 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-rvhhk" Nov 02 06:27:19.895398 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:27:19.895355 5916 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p98cq" containerName="extractor" Nov 02 06:27:24.981493 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:24.981452 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-581608-666ff49d8c-2466x"] Nov 02 06:27:24.986444 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:24.986413 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-581608-666ff49d8c-2466x" Nov 02 06:27:25.025216 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:25.025179 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9b7j\" (UniqueName: \"kubernetes.io/projected/63b6b2e6-166d-4571-b148-6c504875697c-kube-api-access-m9b7j\") pod \"payload-pre-processing-e2e-worker-w2-581608-666ff49d8c-2466x\" (UID: \"63b6b2e6-166d-4571-b148-6c504875697c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-581608-666ff49d8c-2466x" Nov 02 06:27:25.025425 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:25.025399 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63b6b2e6-166d-4571-b148-6c504875697c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-581608-666ff49d8c-2466x\" (UID: \"63b6b2e6-166d-4571-b148-6c504875697c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-581608-666ff49d8c-2466x" Nov 02 06:27:25.081675 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:25.081643 5916 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-581608\"" Nov 02 06:27:25.081845 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:25.081720 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-581608-dockercfg-j8dbg\"" Nov 02 06:27:25.083968 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:25.083945 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-581608-666ff49d8c-2466x"] Nov 02 06:27:25.126949 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:25.126878 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63b6b2e6-166d-4571-b148-6c504875697c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-581608-666ff49d8c-2466x\" (UID: \"63b6b2e6-166d-4571-b148-6c504875697c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-581608-666ff49d8c-2466x" Nov 02 06:27:25.127157 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:25.126957 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9b7j\" (UniqueName: \"kubernetes.io/projected/63b6b2e6-166d-4571-b148-6c504875697c-kube-api-access-m9b7j\") pod \"payload-pre-processing-e2e-worker-w2-581608-666ff49d8c-2466x\" (UID: \"63b6b2e6-166d-4571-b148-6c504875697c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-581608-666ff49d8c-2466x" Nov 02 06:27:25.128332 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:25.128300 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63b6b2e6-166d-4571-b148-6c504875697c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-581608-666ff49d8c-2466x\" (UID: \"63b6b2e6-166d-4571-b148-6c504875697c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-581608-666ff49d8c-2466x" Nov 02 06:27:25.183873 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:25.183838 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9b7j\" (UniqueName: \"kubernetes.io/projected/63b6b2e6-166d-4571-b148-6c504875697c-kube-api-access-m9b7j\") pod \"payload-pre-processing-e2e-worker-w2-581608-666ff49d8c-2466x\" (UID: \"63b6b2e6-166d-4571-b148-6c504875697c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-581608-666ff49d8c-2466x" Nov 02 06:27:25.300526 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:25.300432 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-581608-666ff49d8c-2466x" Nov 02 06:27:25.673310 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:25.673273 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-581608-666ff49d8c-2466x"] Nov 02 06:27:25.742569 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:25.742535 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-581608-666ff49d8c-2466x" event={"ID":"63b6b2e6-166d-4571-b148-6c504875697c","Type":"ContainerStarted","Data":"159cb26a898a0d52f3f15a745f7a82cd824c789121cb80c462a98e3d187bd449"} Nov 02 06:27:28.767567 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:28.767527 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-581608-666ff49d8c-2466x" event={"ID":"63b6b2e6-166d-4571-b148-6c504875697c","Type":"ContainerStarted","Data":"e8049b156c9b93ea0f393748b64b6b2c992316b3b9b5488443e7c1ba0d70eb59"} Nov 02 06:27:28.767972 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:28.767744 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-581608-666ff49d8c-2466x" Nov 02 06:27:28.874368 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:28.874318 5916 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-581608-666ff49d8c-2466x" podStartSLOduration=1.889398227 podStartE2EDuration="4.874304437s" totalImagesPullingTime="2.98490621s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:27:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:27:28.872358474 +0000 UTC m=+637.144178867" watchObservedRunningTime="2026-11-02 06:27:28.874304437 +0000 UTC m=+637.146124827" Nov 02 06:27:29.447349 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:29.447312 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-680b7b-775c58cc67-6d6rn"] Nov 02 06:27:29.451192 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:29.451168 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-680b7b-775c58cc67-6d6rn" Nov 02 06:27:29.507302 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:29.507264 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-680b7b-775c58cc67-6d6rn"] Nov 02 06:27:29.547427 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:29.547397 5916 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-680b7b\"" Nov 02 06:27:29.547595 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:29.547467 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-680b7b-dockercfg-48fkp\"" Nov 02 06:27:29.571848 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:29.571811 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxd4m\" (UniqueName: \"kubernetes.io/projected/90130906-592a-4243-9697-6daa79939293-kube-api-access-rxd4m\") pod \"payload-pre-processing-e2e-worker-w4-680b7b-775c58cc67-6d6rn\" (UID: \"90130906-592a-4243-9697-6daa79939293\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-680b7b-775c58cc67-6d6rn" Nov 02 06:27:29.572002 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:29.571868 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90130906-592a-4243-9697-6daa79939293-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-680b7b-775c58cc67-6d6rn\" (UID: \"90130906-592a-4243-9697-6daa79939293\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-680b7b-775c58cc67-6d6rn" Nov 02 06:27:29.612404 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:29.612373 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-680b7b-867b5cf97c-s84v9"] Nov 02 06:27:29.616273 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:29.616253 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-680b7b-867b5cf97c-s84v9" Nov 02 06:27:29.668699 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:29.668670 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-680b7b-867b5cf97c-s84v9"] Nov 02 06:27:29.673090 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:29.673062 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvq2d\" (UniqueName: \"kubernetes.io/projected/144a4e28-24fb-467e-8e2a-2e26736c9589-kube-api-access-gvq2d\") pod \"payload-processing-e2e-worker-w4-680b7b-867b5cf97c-s84v9\" (UID: \"144a4e28-24fb-467e-8e2a-2e26736c9589\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-680b7b-867b5cf97c-s84v9" Nov 02 06:27:29.673245 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:29.673116 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxd4m\" (UniqueName: \"kubernetes.io/projected/90130906-592a-4243-9697-6daa79939293-kube-api-access-rxd4m\") pod \"payload-pre-processing-e2e-worker-w4-680b7b-775c58cc67-6d6rn\" (UID: \"90130906-592a-4243-9697-6daa79939293\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-680b7b-775c58cc67-6d6rn" Nov 02 06:27:29.673245 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:29.673214 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90130906-592a-4243-9697-6daa79939293-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-680b7b-775c58cc67-6d6rn\" (UID: \"90130906-592a-4243-9697-6daa79939293\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-680b7b-775c58cc67-6d6rn" Nov 02 06:27:29.673362 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:29.673332 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/144a4e28-24fb-467e-8e2a-2e26736c9589-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-680b7b-867b5cf97c-s84v9\" (UID: \"144a4e28-24fb-467e-8e2a-2e26736c9589\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-680b7b-867b5cf97c-s84v9" Nov 02 06:27:29.674185 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:29.674165 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90130906-592a-4243-9697-6daa79939293-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-680b7b-775c58cc67-6d6rn\" (UID: \"90130906-592a-4243-9697-6daa79939293\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-680b7b-775c58cc67-6d6rn" Nov 02 06:27:29.726281 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:29.726206 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxd4m\" (UniqueName: \"kubernetes.io/projected/90130906-592a-4243-9697-6daa79939293-kube-api-access-rxd4m\") pod \"payload-pre-processing-e2e-worker-w4-680b7b-775c58cc67-6d6rn\" (UID: \"90130906-592a-4243-9697-6daa79939293\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-680b7b-775c58cc67-6d6rn" Nov 02 06:27:29.763986 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:29.763953 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-680b7b-775c58cc67-6d6rn" Nov 02 06:27:29.773865 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:29.773833 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvq2d\" (UniqueName: \"kubernetes.io/projected/144a4e28-24fb-467e-8e2a-2e26736c9589-kube-api-access-gvq2d\") pod \"payload-processing-e2e-worker-w4-680b7b-867b5cf97c-s84v9\" (UID: \"144a4e28-24fb-467e-8e2a-2e26736c9589\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-680b7b-867b5cf97c-s84v9" Nov 02 06:27:29.774276 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:29.773994 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/144a4e28-24fb-467e-8e2a-2e26736c9589-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-680b7b-867b5cf97c-s84v9\" (UID: \"144a4e28-24fb-467e-8e2a-2e26736c9589\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-680b7b-867b5cf97c-s84v9" Nov 02 06:27:29.774612 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:29.774593 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/144a4e28-24fb-467e-8e2a-2e26736c9589-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-680b7b-867b5cf97c-s84v9\" (UID: \"144a4e28-24fb-467e-8e2a-2e26736c9589\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-680b7b-867b5cf97c-s84v9" Nov 02 06:27:29.828372 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:29.828312 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvq2d\" (UniqueName: \"kubernetes.io/projected/144a4e28-24fb-467e-8e2a-2e26736c9589-kube-api-access-gvq2d\") pod \"payload-processing-e2e-worker-w4-680b7b-867b5cf97c-s84v9\" (UID: \"144a4e28-24fb-467e-8e2a-2e26736c9589\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-680b7b-867b5cf97c-s84v9" Nov 02 06:27:29.929549 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:29.929518 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-680b7b-867b5cf97c-s84v9" Nov 02 06:27:30.141220 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:30.141185 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-680b7b-775c58cc67-6d6rn"] Nov 02 06:27:30.145669 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:27:30.145640 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90130906_592a_4243_9697_6daa79939293.slice/crio-f2dacff3ffb0c1b7be64a47061bae6aa76cec87270047e1459d0c51527a53992 WatchSource:0}: Error finding container f2dacff3ffb0c1b7be64a47061bae6aa76cec87270047e1459d0c51527a53992: Status 404 returned error can't find the container with id f2dacff3ffb0c1b7be64a47061bae6aa76cec87270047e1459d0c51527a53992 Nov 02 06:27:30.304898 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:30.304815 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-680b7b-867b5cf97c-s84v9"] Nov 02 06:27:30.309987 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:27:30.309959 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod144a4e28_24fb_467e_8e2a_2e26736c9589.slice/crio-57a6d8128a397763aedffd824fcc9b55dcf0089590725411792d143b24254ee6 WatchSource:0}: Error finding container 57a6d8128a397763aedffd824fcc9b55dcf0089590725411792d143b24254ee6: Status 404 returned error can't find the container with id 57a6d8128a397763aedffd824fcc9b55dcf0089590725411792d143b24254ee6 Nov 02 06:27:30.781255 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:30.781211 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-680b7b-867b5cf97c-s84v9" event={"ID":"144a4e28-24fb-467e-8e2a-2e26736c9589","Type":"ContainerStarted","Data":"7121fef955574f68cbc728df2965193b959aad9917544b2c19f134ba0cbd6ea3"} Nov 02 06:27:30.781760 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:30.781280 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-680b7b-867b5cf97c-s84v9" event={"ID":"144a4e28-24fb-467e-8e2a-2e26736c9589","Type":"ContainerStarted","Data":"57a6d8128a397763aedffd824fcc9b55dcf0089590725411792d143b24254ee6"} Nov 02 06:27:30.781760 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:30.781306 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-680b7b-867b5cf97c-s84v9" Nov 02 06:27:30.782798 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:30.782771 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-680b7b-775c58cc67-6d6rn" event={"ID":"90130906-592a-4243-9697-6daa79939293","Type":"ContainerStarted","Data":"7dbfa1661aa93f68b1c6bd6df01d8439677c9f686228ff490143ba51750c882d"} Nov 02 06:27:30.782925 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:30.782821 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-680b7b-775c58cc67-6d6rn" event={"ID":"90130906-592a-4243-9697-6daa79939293","Type":"ContainerStarted","Data":"f2dacff3ffb0c1b7be64a47061bae6aa76cec87270047e1459d0c51527a53992"} Nov 02 06:27:30.782925 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:30.782849 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-680b7b-775c58cc67-6d6rn" Nov 02 06:27:30.889125 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:30.889022 5916 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-680b7b-867b5cf97c-s84v9" podStartSLOduration=1.889007507 podStartE2EDuration="1.889007507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:27:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:27:30.887455856 +0000 UTC m=+639.159276251" watchObservedRunningTime="2026-11-02 06:27:30.889007507 +0000 UTC m=+639.160827901" Nov 02 06:27:30.994032 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:30.993977 5916 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-680b7b-775c58cc67-6d6rn" podStartSLOduration=1.993958778 podStartE2EDuration="1.993958778s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:27:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:27:30.992657707 +0000 UTC m=+639.264478101" watchObservedRunningTime="2026-11-02 06:27:30.993958778 +0000 UTC m=+639.265779173" Nov 02 06:27:31.751582 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:31.751548 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-dcf9c3-694d5779d7-cwfqq"] Nov 02 06:27:31.755498 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:31.755477 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-dcf9c3-694d5779d7-cwfqq" Nov 02 06:27:31.794100 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:31.794069 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4w7l\" (UniqueName: \"kubernetes.io/projected/8205f559-100b-4fff-8e3c-4dea453db8e6-kube-api-access-b4w7l\") pod \"payload-pre-processing-e2e-worker-w5-dcf9c3-694d5779d7-cwfqq\" (UID: \"8205f559-100b-4fff-8e3c-4dea453db8e6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-dcf9c3-694d5779d7-cwfqq" Nov 02 06:27:31.794483 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:31.794116 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8205f559-100b-4fff-8e3c-4dea453db8e6-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-dcf9c3-694d5779d7-cwfqq\" (UID: \"8205f559-100b-4fff-8e3c-4dea453db8e6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-dcf9c3-694d5779d7-cwfqq" Nov 02 06:27:31.809215 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:31.808745 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-dcf9c3-694d5779d7-cwfqq"] Nov 02 06:27:31.850935 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:31.850868 5916 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-dcf9c3\"" Nov 02 06:27:31.850935 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:31.850893 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-dcf9c3-dockercfg-ggn74\"" Nov 02 06:27:31.895064 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:31.895032 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4w7l\" (UniqueName: \"kubernetes.io/projected/8205f559-100b-4fff-8e3c-4dea453db8e6-kube-api-access-b4w7l\") pod \"payload-pre-processing-e2e-worker-w5-dcf9c3-694d5779d7-cwfqq\" (UID: \"8205f559-100b-4fff-8e3c-4dea453db8e6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-dcf9c3-694d5779d7-cwfqq" Nov 02 06:27:31.895277 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:31.895105 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8205f559-100b-4fff-8e3c-4dea453db8e6-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-dcf9c3-694d5779d7-cwfqq\" (UID: \"8205f559-100b-4fff-8e3c-4dea453db8e6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-dcf9c3-694d5779d7-cwfqq" Nov 02 06:27:31.895898 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:31.895870 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8205f559-100b-4fff-8e3c-4dea453db8e6-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-dcf9c3-694d5779d7-cwfqq\" (UID: \"8205f559-100b-4fff-8e3c-4dea453db8e6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-dcf9c3-694d5779d7-cwfqq" Nov 02 06:27:31.918505 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:31.918477 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-dcf9c3-66797968df-l4z8j"] Nov 02 06:27:31.922230 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:31.922204 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-dcf9c3-66797968df-l4z8j" Nov 02 06:27:31.948384 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:31.948349 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4w7l\" (UniqueName: \"kubernetes.io/projected/8205f559-100b-4fff-8e3c-4dea453db8e6-kube-api-access-b4w7l\") pod \"payload-pre-processing-e2e-worker-w5-dcf9c3-694d5779d7-cwfqq\" (UID: \"8205f559-100b-4fff-8e3c-4dea453db8e6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-dcf9c3-694d5779d7-cwfqq" Nov 02 06:27:31.975790 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:31.975736 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-dcf9c3-66797968df-l4z8j"] Nov 02 06:27:31.995980 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:31.995947 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4h96p\" (UniqueName: \"kubernetes.io/projected/9b7d318a-026f-4857-904d-20713b54266c-kube-api-access-4h96p\") pod \"payload-processing-e2e-worker-w5-dcf9c3-66797968df-l4z8j\" (UID: \"9b7d318a-026f-4857-904d-20713b54266c\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-dcf9c3-66797968df-l4z8j" Nov 02 06:27:31.996182 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:31.996009 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b7d318a-026f-4857-904d-20713b54266c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-dcf9c3-66797968df-l4z8j\" (UID: \"9b7d318a-026f-4857-904d-20713b54266c\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-dcf9c3-66797968df-l4z8j" Nov 02 06:27:32.070021 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:32.069939 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-dcf9c3-694d5779d7-cwfqq" Nov 02 06:27:32.098086 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:32.098050 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4h96p\" (UniqueName: \"kubernetes.io/projected/9b7d318a-026f-4857-904d-20713b54266c-kube-api-access-4h96p\") pod \"payload-processing-e2e-worker-w5-dcf9c3-66797968df-l4z8j\" (UID: \"9b7d318a-026f-4857-904d-20713b54266c\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-dcf9c3-66797968df-l4z8j" Nov 02 06:27:32.098248 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:32.098141 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b7d318a-026f-4857-904d-20713b54266c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-dcf9c3-66797968df-l4z8j\" (UID: \"9b7d318a-026f-4857-904d-20713b54266c\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-dcf9c3-66797968df-l4z8j" Nov 02 06:27:32.098935 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:32.098894 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b7d318a-026f-4857-904d-20713b54266c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-dcf9c3-66797968df-l4z8j\" (UID: \"9b7d318a-026f-4857-904d-20713b54266c\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-dcf9c3-66797968df-l4z8j" Nov 02 06:27:32.152064 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:32.152031 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4h96p\" (UniqueName: \"kubernetes.io/projected/9b7d318a-026f-4857-904d-20713b54266c-kube-api-access-4h96p\") pod \"payload-processing-e2e-worker-w5-dcf9c3-66797968df-l4z8j\" (UID: \"9b7d318a-026f-4857-904d-20713b54266c\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-dcf9c3-66797968df-l4z8j" Nov 02 06:27:32.235726 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:32.235691 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-dcf9c3-66797968df-l4z8j" Nov 02 06:27:32.448323 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:32.448283 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-dcf9c3-694d5779d7-cwfqq"] Nov 02 06:27:32.453708 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:27:32.453682 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8205f559_100b_4fff_8e3c_4dea453db8e6.slice/crio-f19ffdcc787bef41e9e5539e31a4f96bc3df921ec505a19204f8392d62cc48e5 WatchSource:0}: Error finding container f19ffdcc787bef41e9e5539e31a4f96bc3df921ec505a19204f8392d62cc48e5: Status 404 returned error can't find the container with id f19ffdcc787bef41e9e5539e31a4f96bc3df921ec505a19204f8392d62cc48e5 Nov 02 06:27:32.605250 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:32.605219 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-dcf9c3-66797968df-l4z8j"] Nov 02 06:27:32.610989 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:27:32.610955 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b7d318a_026f_4857_904d_20713b54266c.slice/crio-59553fcaf56dcad33d026924ae1790eb183853ae099d7d13470ed683c1783ec1 WatchSource:0}: Error finding container 59553fcaf56dcad33d026924ae1790eb183853ae099d7d13470ed683c1783ec1: Status 404 returned error can't find the container with id 59553fcaf56dcad33d026924ae1790eb183853ae099d7d13470ed683c1783ec1 Nov 02 06:27:32.795995 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:32.795873 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-dcf9c3-66797968df-l4z8j" event={"ID":"9b7d318a-026f-4857-904d-20713b54266c","Type":"ContainerStarted","Data":"7355631c7872a7c15867a0737444c6536a06decaeec309a9b83b4967a3d81ce9"} Nov 02 06:27:32.795995 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:32.795951 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-dcf9c3-66797968df-l4z8j" event={"ID":"9b7d318a-026f-4857-904d-20713b54266c","Type":"ContainerStarted","Data":"59553fcaf56dcad33d026924ae1790eb183853ae099d7d13470ed683c1783ec1"} Nov 02 06:27:32.795995 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:32.795976 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-dcf9c3-66797968df-l4z8j" Nov 02 06:27:32.797525 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:32.797501 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-dcf9c3-694d5779d7-cwfqq" event={"ID":"8205f559-100b-4fff-8e3c-4dea453db8e6","Type":"ContainerStarted","Data":"fc7f5391da15d863a20fe22c17206e8ac5c704b3011130ef09451f6dc0d915ea"} Nov 02 06:27:32.797639 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:32.797541 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-dcf9c3-694d5779d7-cwfqq" event={"ID":"8205f559-100b-4fff-8e3c-4dea453db8e6","Type":"ContainerStarted","Data":"f19ffdcc787bef41e9e5539e31a4f96bc3df921ec505a19204f8392d62cc48e5"} Nov 02 06:27:32.797688 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:32.797634 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-dcf9c3-694d5779d7-cwfqq" Nov 02 06:27:32.903051 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:32.902999 5916 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w5-dcf9c3-66797968df-l4z8j" podStartSLOduration=1.902983597 podStartE2EDuration="1.902983597s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:27:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:27:32.901173821 +0000 UTC m=+641.172994214" watchObservedRunningTime="2026-11-02 06:27:32.902983597 +0000 UTC m=+641.174804021" Nov 02 06:27:33.006720 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:33.006616 5916 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-dcf9c3-694d5779d7-cwfqq" podStartSLOduration=2.006598142 podStartE2EDuration="2.006598142s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:27:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:27:33.005639078 +0000 UTC m=+641.277459473" watchObservedRunningTime="2026-11-02 06:27:33.006598142 +0000 UTC m=+641.278418536" Nov 02 06:27:39.776948 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:39.776919 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-581608-666ff49d8c-2466x" Nov 02 06:27:41.791774 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:41.791745 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-680b7b-775c58cc67-6d6rn" Nov 02 06:27:41.792274 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:41.791809 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-680b7b-867b5cf97c-s84v9" Nov 02 06:27:43.148762 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:43.148719 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-6ff85e-7fd4b5f85f-4wcpn"] Nov 02 06:27:43.153169 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:43.153145 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-6ff85e-7fd4b5f85f-4wcpn" Nov 02 06:27:43.205499 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:43.205468 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-6ff85e-7fd4b5f85f-4wcpn"] Nov 02 06:27:43.214404 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:43.214373 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcnd2\" (UniqueName: \"kubernetes.io/projected/771371d6-10a2-4679-8a9e-3ca2e695f4db-kube-api-access-zcnd2\") pod \"payload-pre-processing-e2e-worker-w6-6ff85e-7fd4b5f85f-4wcpn\" (UID: \"771371d6-10a2-4679-8a9e-3ca2e695f4db\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-6ff85e-7fd4b5f85f-4wcpn" Nov 02 06:27:43.214744 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:43.214719 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/771371d6-10a2-4679-8a9e-3ca2e695f4db-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-6ff85e-7fd4b5f85f-4wcpn\" (UID: \"771371d6-10a2-4679-8a9e-3ca2e695f4db\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-6ff85e-7fd4b5f85f-4wcpn" Nov 02 06:27:43.249004 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:43.248974 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-6ff85e-dockercfg-xwnr2\"" Nov 02 06:27:43.249004 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:43.248985 5916 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-6ff85e\"" Nov 02 06:27:43.305185 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:43.305156 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-6ff85e-7c6f4dfd87-c5m8z"] Nov 02 06:27:43.309122 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:43.309101 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-6ff85e-7c6f4dfd87-c5m8z" Nov 02 06:27:43.315493 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:43.315468 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcnd2\" (UniqueName: \"kubernetes.io/projected/771371d6-10a2-4679-8a9e-3ca2e695f4db-kube-api-access-zcnd2\") pod \"payload-pre-processing-e2e-worker-w6-6ff85e-7fd4b5f85f-4wcpn\" (UID: \"771371d6-10a2-4679-8a9e-3ca2e695f4db\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-6ff85e-7fd4b5f85f-4wcpn" Nov 02 06:27:43.315636 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:43.315550 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/771371d6-10a2-4679-8a9e-3ca2e695f4db-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-6ff85e-7fd4b5f85f-4wcpn\" (UID: \"771371d6-10a2-4679-8a9e-3ca2e695f4db\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-6ff85e-7fd4b5f85f-4wcpn" Nov 02 06:27:43.316355 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:43.316330 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/771371d6-10a2-4679-8a9e-3ca2e695f4db-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-6ff85e-7fd4b5f85f-4wcpn\" (UID: \"771371d6-10a2-4679-8a9e-3ca2e695f4db\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-6ff85e-7fd4b5f85f-4wcpn" Nov 02 06:27:43.362877 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:43.362846 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-6ff85e-7c6f4dfd87-c5m8z"] Nov 02 06:27:43.369492 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:43.369471 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcnd2\" (UniqueName: \"kubernetes.io/projected/771371d6-10a2-4679-8a9e-3ca2e695f4db-kube-api-access-zcnd2\") pod \"payload-pre-processing-e2e-worker-w6-6ff85e-7fd4b5f85f-4wcpn\" (UID: \"771371d6-10a2-4679-8a9e-3ca2e695f4db\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-6ff85e-7fd4b5f85f-4wcpn" Nov 02 06:27:43.417045 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:43.416968 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9282\" (UniqueName: \"kubernetes.io/projected/ee6a9440-ee89-4c29-aa2b-eb34c10935bf-kube-api-access-l9282\") pod \"payload-processing-e2e-worker-w6-6ff85e-7c6f4dfd87-c5m8z\" (UID: \"ee6a9440-ee89-4c29-aa2b-eb34c10935bf\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-6ff85e-7c6f4dfd87-c5m8z" Nov 02 06:27:43.417045 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:43.417021 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee6a9440-ee89-4c29-aa2b-eb34c10935bf-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-6ff85e-7c6f4dfd87-c5m8z\" (UID: \"ee6a9440-ee89-4c29-aa2b-eb34c10935bf\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-6ff85e-7c6f4dfd87-c5m8z" Nov 02 06:27:43.467031 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:43.467005 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-6ff85e-7fd4b5f85f-4wcpn" Nov 02 06:27:43.518479 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:43.518437 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9282\" (UniqueName: \"kubernetes.io/projected/ee6a9440-ee89-4c29-aa2b-eb34c10935bf-kube-api-access-l9282\") pod \"payload-processing-e2e-worker-w6-6ff85e-7c6f4dfd87-c5m8z\" (UID: \"ee6a9440-ee89-4c29-aa2b-eb34c10935bf\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-6ff85e-7c6f4dfd87-c5m8z" Nov 02 06:27:43.518624 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:43.518506 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee6a9440-ee89-4c29-aa2b-eb34c10935bf-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-6ff85e-7c6f4dfd87-c5m8z\" (UID: \"ee6a9440-ee89-4c29-aa2b-eb34c10935bf\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-6ff85e-7c6f4dfd87-c5m8z" Nov 02 06:27:43.519395 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:43.519363 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee6a9440-ee89-4c29-aa2b-eb34c10935bf-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-6ff85e-7c6f4dfd87-c5m8z\" (UID: \"ee6a9440-ee89-4c29-aa2b-eb34c10935bf\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-6ff85e-7c6f4dfd87-c5m8z" Nov 02 06:27:43.572195 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:43.572165 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9282\" (UniqueName: \"kubernetes.io/projected/ee6a9440-ee89-4c29-aa2b-eb34c10935bf-kube-api-access-l9282\") pod \"payload-processing-e2e-worker-w6-6ff85e-7c6f4dfd87-c5m8z\" (UID: \"ee6a9440-ee89-4c29-aa2b-eb34c10935bf\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-6ff85e-7c6f4dfd87-c5m8z" Nov 02 06:27:43.622198 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:43.622162 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-6ff85e-7c6f4dfd87-c5m8z" Nov 02 06:27:43.808834 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:43.808806 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-dcf9c3-66797968df-l4z8j" Nov 02 06:27:43.809024 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:43.808878 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-dcf9c3-694d5779d7-cwfqq" Nov 02 06:27:43.842181 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:43.842154 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-6ff85e-7fd4b5f85f-4wcpn"] Nov 02 06:27:43.842737 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:27:43.842714 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod771371d6_10a2_4679_8a9e_3ca2e695f4db.slice/crio-7c392b9efd24406d1261beda11cc9b1b23a73b14639b027458b0a42e9dcf2e42 WatchSource:0}: Error finding container 7c392b9efd24406d1261beda11cc9b1b23a73b14639b027458b0a42e9dcf2e42: Status 404 returned error can't find the container with id 7c392b9efd24406d1261beda11cc9b1b23a73b14639b027458b0a42e9dcf2e42 Nov 02 06:27:43.869044 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:43.869020 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-6ff85e-7fd4b5f85f-4wcpn" event={"ID":"771371d6-10a2-4679-8a9e-3ca2e695f4db","Type":"ContainerStarted","Data":"7c392b9efd24406d1261beda11cc9b1b23a73b14639b027458b0a42e9dcf2e42"} Nov 02 06:27:43.990761 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:43.990678 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-6ff85e-7c6f4dfd87-c5m8z"] Nov 02 06:27:43.996964 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:27:43.996938 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee6a9440_ee89_4c29_aa2b_eb34c10935bf.slice/crio-807b3c30a9be1e105bd2fad3a7dec2b764111187b3d3364cc1ae2f003f127547 WatchSource:0}: Error finding container 807b3c30a9be1e105bd2fad3a7dec2b764111187b3d3364cc1ae2f003f127547: Status 404 returned error can't find the container with id 807b3c30a9be1e105bd2fad3a7dec2b764111187b3d3364cc1ae2f003f127547 Nov 02 06:27:44.877854 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:44.877815 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-6ff85e-7fd4b5f85f-4wcpn" event={"ID":"771371d6-10a2-4679-8a9e-3ca2e695f4db","Type":"ContainerStarted","Data":"bc5ad09cad1229e5799b0ec72209b54e2a2e835508faa0b019607cb64356d848"} Nov 02 06:27:44.878337 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:44.877876 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-6ff85e-7fd4b5f85f-4wcpn" Nov 02 06:27:44.879565 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:44.879543 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-6ff85e-7c6f4dfd87-c5m8z" event={"ID":"ee6a9440-ee89-4c29-aa2b-eb34c10935bf","Type":"ContainerStarted","Data":"aa49170d5fb55818a96f24ed50fe303d658eb084693107e35c5e6d9c0eded4d4"} Nov 02 06:27:44.879661 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:44.879581 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-6ff85e-7c6f4dfd87-c5m8z" event={"ID":"ee6a9440-ee89-4c29-aa2b-eb34c10935bf","Type":"ContainerStarted","Data":"807b3c30a9be1e105bd2fad3a7dec2b764111187b3d3364cc1ae2f003f127547"} Nov 02 06:27:44.879661 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:44.879618 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-6ff85e-7c6f4dfd87-c5m8z" Nov 02 06:27:44.986762 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:44.986712 5916 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-6ff85e-7fd4b5f85f-4wcpn" podStartSLOduration=1.986697165 podStartE2EDuration="1.986697165s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:27:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:27:44.984936172 +0000 UTC m=+653.256756563" watchObservedRunningTime="2026-11-02 06:27:44.986697165 +0000 UTC m=+653.258517558" Nov 02 06:27:45.093206 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:45.093148 5916 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w6-6ff85e-7c6f4dfd87-c5m8z" podStartSLOduration=2.093132904 podStartE2EDuration="2.093132904s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:27:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:27:45.091282887 +0000 UTC m=+653.363103282" watchObservedRunningTime="2026-11-02 06:27:45.093132904 +0000 UTC m=+653.364953343" Nov 02 06:27:49.893577 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:27:49.893541 5916 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p98cq" containerName="extractor" Nov 02 06:27:51.930281 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:51.930248 5916 scope.go:122] "RemoveContainer" containerID="15feed1b417a13df68a040080728d23213b5e2ad1849fa6a56a90047663fb321" Nov 02 06:27:55.888195 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:55.888167 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-6ff85e-7c6f4dfd87-c5m8z" Nov 02 06:27:55.888732 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:55.888232 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-6ff85e-7fd4b5f85f-4wcpn" Nov 02 06:27:56.525417 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:56.525383 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-fb1398-54468d6bf5-wpgxd"] Nov 02 06:27:56.530720 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:56.530699 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-fb1398-54468d6bf5-wpgxd" Nov 02 06:27:56.583383 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:56.583358 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-fb1398-54468d6bf5-wpgxd"] Nov 02 06:27:56.625811 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:56.625789 5916 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 06:27:56.626028 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:56.626013 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-fb1398-serving-cert\"" Nov 02 06:27:56.626143 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:56.626124 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-gm8x6\"" Nov 02 06:27:56.626209 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:56.626128 5916 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 06:27:56.626296 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:56.626281 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 02 06:27:56.641207 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:56.641176 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tcmm\" (UniqueName: \"kubernetes.io/projected/50326ab7-9415-4017-ab7a-a6b04c2d222f-kube-api-access-8tcmm\") pod \"maas-api-e2e-worker-w3-fb1398-54468d6bf5-wpgxd\" (UID: \"50326ab7-9415-4017-ab7a-a6b04c2d222f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-fb1398-54468d6bf5-wpgxd" Nov 02 06:27:56.641331 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:56.641238 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50326ab7-9415-4017-ab7a-a6b04c2d222f-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-fb1398-54468d6bf5-wpgxd\" (UID: \"50326ab7-9415-4017-ab7a-a6b04c2d222f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-fb1398-54468d6bf5-wpgxd" Nov 02 06:27:56.641393 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:56.641347 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50326ab7-9415-4017-ab7a-a6b04c2d222f-maas-api-tls\") pod \"maas-api-e2e-worker-w3-fb1398-54468d6bf5-wpgxd\" (UID: \"50326ab7-9415-4017-ab7a-a6b04c2d222f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-fb1398-54468d6bf5-wpgxd" Nov 02 06:27:56.641393 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:56.641369 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50326ab7-9415-4017-ab7a-a6b04c2d222f-metrics-certs\") pod \"maas-api-e2e-worker-w3-fb1398-54468d6bf5-wpgxd\" (UID: \"50326ab7-9415-4017-ab7a-a6b04c2d222f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-fb1398-54468d6bf5-wpgxd" Nov 02 06:27:56.742687 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:56.742650 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50326ab7-9415-4017-ab7a-a6b04c2d222f-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-fb1398-54468d6bf5-wpgxd\" (UID: \"50326ab7-9415-4017-ab7a-a6b04c2d222f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-fb1398-54468d6bf5-wpgxd" Nov 02 06:27:56.742832 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:56.742721 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50326ab7-9415-4017-ab7a-a6b04c2d222f-maas-api-tls\") pod \"maas-api-e2e-worker-w3-fb1398-54468d6bf5-wpgxd\" (UID: \"50326ab7-9415-4017-ab7a-a6b04c2d222f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-fb1398-54468d6bf5-wpgxd" Nov 02 06:27:56.742832 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:56.742746 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50326ab7-9415-4017-ab7a-a6b04c2d222f-metrics-certs\") pod \"maas-api-e2e-worker-w3-fb1398-54468d6bf5-wpgxd\" (UID: \"50326ab7-9415-4017-ab7a-a6b04c2d222f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-fb1398-54468d6bf5-wpgxd" Nov 02 06:27:56.742832 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:56.742800 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tcmm\" (UniqueName: \"kubernetes.io/projected/50326ab7-9415-4017-ab7a-a6b04c2d222f-kube-api-access-8tcmm\") pod \"maas-api-e2e-worker-w3-fb1398-54468d6bf5-wpgxd\" (UID: \"50326ab7-9415-4017-ab7a-a6b04c2d222f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-fb1398-54468d6bf5-wpgxd" Nov 02 06:27:56.742988 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:27:56.742879 5916 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w3-fb1398-serving-cert: secret "maas-api-e2e-worker-w3-fb1398-serving-cert" not found Nov 02 06:27:56.742988 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:27:56.742980 5916 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50326ab7-9415-4017-ab7a-a6b04c2d222f-maas-api-tls podName:50326ab7-9415-4017-ab7a-a6b04c2d222f nodeName:}" failed. No retries permitted until 2026-11-02 06:27:57.24296335 +0000 UTC m=+665.514783734 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/50326ab7-9415-4017-ab7a-a6b04c2d222f-maas-api-tls") pod "maas-api-e2e-worker-w3-fb1398-54468d6bf5-wpgxd" (UID: "50326ab7-9415-4017-ab7a-a6b04c2d222f") : secret "maas-api-e2e-worker-w3-fb1398-serving-cert" not found Nov 02 06:27:56.743436 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:56.743418 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50326ab7-9415-4017-ab7a-a6b04c2d222f-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-fb1398-54468d6bf5-wpgxd\" (UID: \"50326ab7-9415-4017-ab7a-a6b04c2d222f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-fb1398-54468d6bf5-wpgxd" Nov 02 06:27:56.745320 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:56.745299 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50326ab7-9415-4017-ab7a-a6b04c2d222f-metrics-certs\") pod \"maas-api-e2e-worker-w3-fb1398-54468d6bf5-wpgxd\" (UID: \"50326ab7-9415-4017-ab7a-a6b04c2d222f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-fb1398-54468d6bf5-wpgxd" Nov 02 06:27:56.798207 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:56.798131 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tcmm\" (UniqueName: \"kubernetes.io/projected/50326ab7-9415-4017-ab7a-a6b04c2d222f-kube-api-access-8tcmm\") pod \"maas-api-e2e-worker-w3-fb1398-54468d6bf5-wpgxd\" (UID: \"50326ab7-9415-4017-ab7a-a6b04c2d222f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-fb1398-54468d6bf5-wpgxd" Nov 02 06:27:57.248072 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:57.248035 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50326ab7-9415-4017-ab7a-a6b04c2d222f-maas-api-tls\") pod \"maas-api-e2e-worker-w3-fb1398-54468d6bf5-wpgxd\" (UID: \"50326ab7-9415-4017-ab7a-a6b04c2d222f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-fb1398-54468d6bf5-wpgxd" Nov 02 06:27:57.250419 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:57.250395 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50326ab7-9415-4017-ab7a-a6b04c2d222f-maas-api-tls\") pod \"maas-api-e2e-worker-w3-fb1398-54468d6bf5-wpgxd\" (UID: \"50326ab7-9415-4017-ab7a-a6b04c2d222f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-fb1398-54468d6bf5-wpgxd" Nov 02 06:27:57.444113 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:57.444069 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-fb1398-54468d6bf5-wpgxd" Nov 02 06:27:57.828690 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:57.828606 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-fb1398-54468d6bf5-wpgxd"] Nov 02 06:27:57.835055 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:27:57.835015 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50326ab7_9415_4017_ab7a_a6b04c2d222f.slice/crio-b07844dbf8226522c69852688bf384879b4142e8a8a1b06935126ad08cdc4198 WatchSource:0}: Error finding container b07844dbf8226522c69852688bf384879b4142e8a8a1b06935126ad08cdc4198: Status 404 returned error can't find the container with id b07844dbf8226522c69852688bf384879b4142e8a8a1b06935126ad08cdc4198 Nov 02 06:27:57.836617 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:57.836595 5916 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 06:27:57.958082 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:57.958042 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-fb1398-54468d6bf5-wpgxd" event={"ID":"50326ab7-9415-4017-ab7a-a6b04c2d222f","Type":"ContainerStarted","Data":"b07844dbf8226522c69852688bf384879b4142e8a8a1b06935126ad08cdc4198"} Nov 02 06:27:59.741091 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:59.741054 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-581608/simulator-w2-581608-kserve-68f68d8559-xkcwp"] Nov 02 06:27:59.745692 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:59.745671 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-581608/simulator-w2-581608-kserve-68f68d8559-xkcwp" Nov 02 06:27:59.817116 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:59.817090 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-581608/simulator-w2-581608-kserve-68f68d8559-xkcwp"] Nov 02 06:27:59.842504 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:59.842467 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-581608\"/\"default-dockercfg-k692v\"" Nov 02 06:27:59.842659 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:59.842637 5916 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-581608\"/\"kube-root-ca.crt\"" Nov 02 06:27:59.842739 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:59.842680 5916 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-581608\"/\"openshift-service-ca.crt\"" Nov 02 06:27:59.842739 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:59.842682 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-581608\"/\"simulator-w2-581608-kserve-self-signed-certs\"" Nov 02 06:27:59.875718 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:59.875685 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/529e25e6-4479-49d6-85c7-85dad6c2193e-dshm\") pod \"simulator-w2-581608-kserve-68f68d8559-xkcwp\" (UID: \"529e25e6-4479-49d6-85c7-85dad6c2193e\") " pod="e2e-models-e2e-worker-w2-581608/simulator-w2-581608-kserve-68f68d8559-xkcwp" Nov 02 06:27:59.875867 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:59.875784 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/529e25e6-4479-49d6-85c7-85dad6c2193e-tmp-dir\") pod \"simulator-w2-581608-kserve-68f68d8559-xkcwp\" (UID: \"529e25e6-4479-49d6-85c7-85dad6c2193e\") " pod="e2e-models-e2e-worker-w2-581608/simulator-w2-581608-kserve-68f68d8559-xkcwp" Nov 02 06:27:59.875939 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:59.875893 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/529e25e6-4479-49d6-85c7-85dad6c2193e-tls-certs\") pod \"simulator-w2-581608-kserve-68f68d8559-xkcwp\" (UID: \"529e25e6-4479-49d6-85c7-85dad6c2193e\") " pod="e2e-models-e2e-worker-w2-581608/simulator-w2-581608-kserve-68f68d8559-xkcwp" Nov 02 06:27:59.875993 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:59.875957 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/529e25e6-4479-49d6-85c7-85dad6c2193e-model-cache\") pod \"simulator-w2-581608-kserve-68f68d8559-xkcwp\" (UID: \"529e25e6-4479-49d6-85c7-85dad6c2193e\") " pod="e2e-models-e2e-worker-w2-581608/simulator-w2-581608-kserve-68f68d8559-xkcwp" Nov 02 06:27:59.876049 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:59.876029 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8f7k7\" (UniqueName: \"kubernetes.io/projected/529e25e6-4479-49d6-85c7-85dad6c2193e-kube-api-access-8f7k7\") pod \"simulator-w2-581608-kserve-68f68d8559-xkcwp\" (UID: \"529e25e6-4479-49d6-85c7-85dad6c2193e\") " pod="e2e-models-e2e-worker-w2-581608/simulator-w2-581608-kserve-68f68d8559-xkcwp" Nov 02 06:27:59.876115 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:59.876099 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/529e25e6-4479-49d6-85c7-85dad6c2193e-home\") pod \"simulator-w2-581608-kserve-68f68d8559-xkcwp\" (UID: \"529e25e6-4479-49d6-85c7-85dad6c2193e\") " pod="e2e-models-e2e-worker-w2-581608/simulator-w2-581608-kserve-68f68d8559-xkcwp" Nov 02 06:27:59.973628 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:59.973590 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-fb1398-54468d6bf5-wpgxd" event={"ID":"50326ab7-9415-4017-ab7a-a6b04c2d222f","Type":"ContainerStarted","Data":"2bc46a298a74341a8dbca8e89f8991eec247a202535d2d1a649b3454e14aba2f"} Nov 02 06:27:59.973804 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:59.973672 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-fb1398-54468d6bf5-wpgxd" Nov 02 06:27:59.977223 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:59.977201 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/529e25e6-4479-49d6-85c7-85dad6c2193e-tls-certs\") pod \"simulator-w2-581608-kserve-68f68d8559-xkcwp\" (UID: \"529e25e6-4479-49d6-85c7-85dad6c2193e\") " pod="e2e-models-e2e-worker-w2-581608/simulator-w2-581608-kserve-68f68d8559-xkcwp" Nov 02 06:27:59.977276 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:59.977230 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/529e25e6-4479-49d6-85c7-85dad6c2193e-model-cache\") pod \"simulator-w2-581608-kserve-68f68d8559-xkcwp\" (UID: \"529e25e6-4479-49d6-85c7-85dad6c2193e\") " pod="e2e-models-e2e-worker-w2-581608/simulator-w2-581608-kserve-68f68d8559-xkcwp" Nov 02 06:27:59.977276 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:59.977247 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8f7k7\" (UniqueName: \"kubernetes.io/projected/529e25e6-4479-49d6-85c7-85dad6c2193e-kube-api-access-8f7k7\") pod \"simulator-w2-581608-kserve-68f68d8559-xkcwp\" (UID: \"529e25e6-4479-49d6-85c7-85dad6c2193e\") " pod="e2e-models-e2e-worker-w2-581608/simulator-w2-581608-kserve-68f68d8559-xkcwp" Nov 02 06:27:59.977359 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:59.977350 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/529e25e6-4479-49d6-85c7-85dad6c2193e-home\") pod \"simulator-w2-581608-kserve-68f68d8559-xkcwp\" (UID: \"529e25e6-4479-49d6-85c7-85dad6c2193e\") " pod="e2e-models-e2e-worker-w2-581608/simulator-w2-581608-kserve-68f68d8559-xkcwp" Nov 02 06:27:59.977403 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:59.977388 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/529e25e6-4479-49d6-85c7-85dad6c2193e-dshm\") pod \"simulator-w2-581608-kserve-68f68d8559-xkcwp\" (UID: \"529e25e6-4479-49d6-85c7-85dad6c2193e\") " pod="e2e-models-e2e-worker-w2-581608/simulator-w2-581608-kserve-68f68d8559-xkcwp" Nov 02 06:27:59.977473 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:59.977458 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/529e25e6-4479-49d6-85c7-85dad6c2193e-tmp-dir\") pod \"simulator-w2-581608-kserve-68f68d8559-xkcwp\" (UID: \"529e25e6-4479-49d6-85c7-85dad6c2193e\") " pod="e2e-models-e2e-worker-w2-581608/simulator-w2-581608-kserve-68f68d8559-xkcwp" Nov 02 06:27:59.977687 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:59.977663 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/529e25e6-4479-49d6-85c7-85dad6c2193e-model-cache\") pod \"simulator-w2-581608-kserve-68f68d8559-xkcwp\" (UID: \"529e25e6-4479-49d6-85c7-85dad6c2193e\") " pod="e2e-models-e2e-worker-w2-581608/simulator-w2-581608-kserve-68f68d8559-xkcwp" Nov 02 06:27:59.977749 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:59.977707 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/529e25e6-4479-49d6-85c7-85dad6c2193e-home\") pod \"simulator-w2-581608-kserve-68f68d8559-xkcwp\" (UID: \"529e25e6-4479-49d6-85c7-85dad6c2193e\") " pod="e2e-models-e2e-worker-w2-581608/simulator-w2-581608-kserve-68f68d8559-xkcwp" Nov 02 06:27:59.977875 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:59.977858 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/529e25e6-4479-49d6-85c7-85dad6c2193e-tmp-dir\") pod \"simulator-w2-581608-kserve-68f68d8559-xkcwp\" (UID: \"529e25e6-4479-49d6-85c7-85dad6c2193e\") " pod="e2e-models-e2e-worker-w2-581608/simulator-w2-581608-kserve-68f68d8559-xkcwp" Nov 02 06:27:59.979652 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:59.979633 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/529e25e6-4479-49d6-85c7-85dad6c2193e-dshm\") pod \"simulator-w2-581608-kserve-68f68d8559-xkcwp\" (UID: \"529e25e6-4479-49d6-85c7-85dad6c2193e\") " pod="e2e-models-e2e-worker-w2-581608/simulator-w2-581608-kserve-68f68d8559-xkcwp" Nov 02 06:27:59.979831 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:27:59.979804 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/529e25e6-4479-49d6-85c7-85dad6c2193e-tls-certs\") pod \"simulator-w2-581608-kserve-68f68d8559-xkcwp\" (UID: \"529e25e6-4479-49d6-85c7-85dad6c2193e\") " pod="e2e-models-e2e-worker-w2-581608/simulator-w2-581608-kserve-68f68d8559-xkcwp" Nov 02 06:28:00.030775 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:28:00.030711 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8f7k7\" (UniqueName: \"kubernetes.io/projected/529e25e6-4479-49d6-85c7-85dad6c2193e-kube-api-access-8f7k7\") pod \"simulator-w2-581608-kserve-68f68d8559-xkcwp\" (UID: \"529e25e6-4479-49d6-85c7-85dad6c2193e\") " pod="e2e-models-e2e-worker-w2-581608/simulator-w2-581608-kserve-68f68d8559-xkcwp" Nov 02 06:28:00.057591 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:28:00.057560 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-581608/simulator-w2-581608-kserve-68f68d8559-xkcwp" Nov 02 06:28:00.083719 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:28:00.083670 5916 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-fb1398-54468d6bf5-wpgxd" podStartSLOduration=2.131046366 podStartE2EDuration="4.083654467s" totalImagesPullingTime="1.952608101s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:27:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:28:00.080617104 +0000 UTC m=+668.352437497" watchObservedRunningTime="2026-11-02 06:28:00.083654467 +0000 UTC m=+668.355474920" Nov 02 06:28:00.433003 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:28:00.432970 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-581608/simulator-w2-581608-kserve-68f68d8559-xkcwp"] Nov 02 06:28:00.438699 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:28:00.438669 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod529e25e6_4479_49d6_85c7_85dad6c2193e.slice/crio-acf644edf4c4c5afb4e1e3a3e11bf43b0f7963851c69e157638e261aad1f1465 WatchSource:0}: Error finding container acf644edf4c4c5afb4e1e3a3e11bf43b0f7963851c69e157638e261aad1f1465: Status 404 returned error can't find the container with id acf644edf4c4c5afb4e1e3a3e11bf43b0f7963851c69e157638e261aad1f1465 Nov 02 06:28:00.980234 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:28:00.980197 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-581608/simulator-w2-581608-kserve-68f68d8559-xkcwp" event={"ID":"529e25e6-4479-49d6-85c7-85dad6c2193e","Type":"ContainerStarted","Data":"acf644edf4c4c5afb4e1e3a3e11bf43b0f7963851c69e157638e261aad1f1465"} Nov 02 06:28:02.996591 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:28:02.996543 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-581608/simulator-w2-581608-kserve-68f68d8559-xkcwp" event={"ID":"529e25e6-4479-49d6-85c7-85dad6c2193e","Type":"ContainerStarted","Data":"2b2274da40934012bf54016614c8a6ed76dd176cfb0e64478c63aade679a591c"} Nov 02 06:28:03.058406 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:28:03.058368 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-581608/simulator-w2-581608-kserve-68f68d8559-xkcwp" Nov 02 06:28:03.105073 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:28:03.105016 5916 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-581608/simulator-w2-581608-kserve-68f68d8559-xkcwp" podStartSLOduration=2.299822229 podStartE2EDuration="4.105001854s" totalImagesPullingTime="1.805179625s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:27:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:28:03.103337413 +0000 UTC m=+671.375157807" watchObservedRunningTime="2026-11-02 06:28:03.105001854 +0000 UTC m=+671.376822252" Nov 02 06:28:05.986574 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:28:05.986536 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-fb1398-54468d6bf5-wpgxd" Nov 02 06:28:10.058101 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:28:10.058068 5916 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-581608/simulator-w2-581608-kserve-68f68d8559-xkcwp" Nov 02 06:28:10.070147 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:28:10.070119 5916 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-581608/simulator-w2-581608-kserve-68f68d8559-xkcwp" Nov 02 06:28:11.057334 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:28:11.057303 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-581608/simulator-w2-581608-kserve-68f68d8559-xkcwp" Nov 02 06:28:19.897284 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:28:19.897235 5916 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p98cq" containerName="extractor" Nov 02 06:28:22.123135 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:28:22.123068 5916 generic.go:356] "Generic (PLEG): container finished" podID="8d74d644-1bbf-4522-8075-43b1f476f656" containerID="86a483e176dbc71802a13babaadd925ebc4b9c64710630e13f7af6c9079394e2" exitCode=0 Nov 02 06:28:22.123542 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:28:22.123144 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-rvhhk" event={"ID":"8d74d644-1bbf-4522-8075-43b1f476f656","Type":"ContainerDied","Data":"86a483e176dbc71802a13babaadd925ebc4b9c64710630e13f7af6c9079394e2"} Nov 02 06:28:22.123542 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:28:22.123503 5916 scope.go:122] "RemoveContainer" containerID="86a483e176dbc71802a13babaadd925ebc4b9c64710630e13f7af6c9079394e2" Nov 02 06:28:22.124956 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:28:22.124898 5916 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 06:28:22.125050 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:28:22.125013 5916 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 06:28:22.125090 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:28:22.125061 5916 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 06:28:23.140325 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:28:23.140288 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-rvhhk" event={"ID":"8d74d644-1bbf-4522-8075-43b1f476f656","Type":"ContainerStarted","Data":"1d0cc66708f23899d1f02f3b1561f95c902e0fda996233051880a0383f05bc04"} Nov 02 06:28:23.592223 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:28:23.592142 5916 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-rvhhk" Nov 02 06:28:23.592390 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:28:23.592329 5916 patch_prober.go:29] interesting pod/e2e-worker-w6-6ff85e-openshift-default-698585659c-rvhhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Nov 02 06:28:23.592439 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:28:23.592387 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-rvhhk" podUID="8d74d644-1bbf-4522-8075-43b1f476f656" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Nov 02 06:28:24.591704 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:28:24.591665 5916 patch_prober.go:29] interesting pod/e2e-worker-w6-6ff85e-openshift-default-698585659c-rvhhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Nov 02 06:28:24.592147 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:28:24.591734 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-rvhhk" podUID="8d74d644-1bbf-4522-8075-43b1f476f656" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Nov 02 06:28:25.592320 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:28:25.592280 5916 patch_prober.go:29] interesting pod/e2e-worker-w6-6ff85e-openshift-default-698585659c-rvhhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Nov 02 06:28:25.592737 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:28:25.592343 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-rvhhk" podUID="8d74d644-1bbf-4522-8075-43b1f476f656" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Nov 02 06:28:26.051931 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:28:26.051233 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-680b7b/simulator-w4-680b7b-kserve-6c4f574c56-p5xtc"] Nov 02 06:28:26.058208 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:28:26.058183 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-680b7b/simulator-w4-680b7b-kserve-6c4f574c56-p5xtc" Nov 02 06:28:26.103325 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:28:26.103290 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-680b7b/simulator-w4-680b7b-kserve-6c4f574c56-p5xtc"] Nov 02 06:28:26.150331 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:28:26.150167 5916 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-680b7b\"/\"openshift-service-ca.crt\"" Nov 02 06:28:26.150331 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:28:26.150241 5916 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-680b7b\"/\"kube-root-ca.crt\"" Nov 02 06:28:26.153120 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:28:26.153100 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-680b7b\"/\"simulator-w4-680b7b-kserve-self-signed-certs\"" Nov 02 06:28:26.153247 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:28:26.153119 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-680b7b\"/\"default-dockercfg-8rnrh\"" Nov 02 06:28:26.190839 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:28:26.190807 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be8a1039-5934-4885-93fe-070575f936ec-dshm\") pod \"simulator-w4-680b7b-kserve-6c4f574c56-p5xtc\" (UID: \"be8a1039-5934-4885-93fe-070575f936ec\") " pod="e2e-models-e2e-worker-w4-680b7b/simulator-w4-680b7b-kserve-6c4f574c56-p5xtc" Nov 02 06:28:26.190839 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:28:26.190842 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvvxp\" (UniqueName: \"kubernetes.io/projected/be8a1039-5934-4885-93fe-070575f936ec-kube-api-access-kvvxp\") pod \"simulator-w4-680b7b-kserve-6c4f574c56-p5xtc\" (UID: \"be8a1039-5934-4885-93fe-070575f936ec\") " pod="e2e-models-e2e-worker-w4-680b7b/simulator-w4-680b7b-kserve-6c4f574c56-p5xtc" Nov 02 06:28:26.191083 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:28:26.190870 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be8a1039-5934-4885-93fe-070575f936ec-tmp-dir\") pod \"simulator-w4-680b7b-kserve-6c4f574c56-p5xtc\" (UID: \"be8a1039-5934-4885-93fe-070575f936ec\") " pod="e2e-models-e2e-worker-w4-680b7b/simulator-w4-680b7b-kserve-6c4f574c56-p5xtc" Nov 02 06:28:26.191083 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:28:26.190956 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be8a1039-5934-4885-93fe-070575f936ec-model-cache\") pod \"simulator-w4-680b7b-kserve-6c4f574c56-p5xtc\" (UID: \"be8a1039-5934-4885-93fe-070575f936ec\") " pod="e2e-models-e2e-worker-w4-680b7b/simulator-w4-680b7b-kserve-6c4f574c56-p5xtc" Nov 02 06:28:26.191083 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:28:26.191001 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be8a1039-5934-4885-93fe-070575f936ec-tls-certs\") pod \"simulator-w4-680b7b-kserve-6c4f574c56-p5xtc\" (UID: \"be8a1039-5934-4885-93fe-070575f936ec\") " pod="e2e-models-e2e-worker-w4-680b7b/simulator-w4-680b7b-kserve-6c4f574c56-p5xtc" Nov 02 06:28:26.191200 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:28:26.191119 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be8a1039-5934-4885-93fe-070575f936ec-home\") pod \"simulator-w4-680b7b-kserve-6c4f574c56-p5xtc\" (UID: \"be8a1039-5934-4885-93fe-070575f936ec\") " pod="e2e-models-e2e-worker-w4-680b7b/simulator-w4-680b7b-kserve-6c4f574c56-p5xtc" Nov 02 06:28:26.291765 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:28:26.291728 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be8a1039-5934-4885-93fe-070575f936ec-dshm\") pod \"simulator-w4-680b7b-kserve-6c4f574c56-p5xtc\" (UID: \"be8a1039-5934-4885-93fe-070575f936ec\") " pod="e2e-models-e2e-worker-w4-680b7b/simulator-w4-680b7b-kserve-6c4f574c56-p5xtc" Nov 02 06:28:26.291765 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:28:26.291764 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvvxp\" (UniqueName: \"kubernetes.io/projected/be8a1039-5934-4885-93fe-070575f936ec-kube-api-access-kvvxp\") pod \"simulator-w4-680b7b-kserve-6c4f574c56-p5xtc\" (UID: \"be8a1039-5934-4885-93fe-070575f936ec\") " pod="e2e-models-e2e-worker-w4-680b7b/simulator-w4-680b7b-kserve-6c4f574c56-p5xtc" Nov 02 06:28:26.292083 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:28:26.291787 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be8a1039-5934-4885-93fe-070575f936ec-tmp-dir\") pod \"simulator-w4-680b7b-kserve-6c4f574c56-p5xtc\" (UID: \"be8a1039-5934-4885-93fe-070575f936ec\") " pod="e2e-models-e2e-worker-w4-680b7b/simulator-w4-680b7b-kserve-6c4f574c56-p5xtc" Nov 02 06:28:26.292083 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:28:26.291809 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be8a1039-5934-4885-93fe-070575f936ec-model-cache\") pod \"simulator-w4-680b7b-kserve-6c4f574c56-p5xtc\" (UID: \"be8a1039-5934-4885-93fe-070575f936ec\") " pod="e2e-models-e2e-worker-w4-680b7b/simulator-w4-680b7b-kserve-6c4f574c56-p5xtc" Nov 02 06:28:26.292083 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:28:26.291833 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be8a1039-5934-4885-93fe-070575f936ec-tls-certs\") pod \"simulator-w4-680b7b-kserve-6c4f574c56-p5xtc\" (UID: \"be8a1039-5934-4885-93fe-070575f936ec\") " pod="e2e-models-e2e-worker-w4-680b7b/simulator-w4-680b7b-kserve-6c4f574c56-p5xtc" Nov 02 06:28:26.292083 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:28:26.291861 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be8a1039-5934-4885-93fe-070575f936ec-home\") pod \"simulator-w4-680b7b-kserve-6c4f574c56-p5xtc\" (UID: \"be8a1039-5934-4885-93fe-070575f936ec\") " pod="e2e-models-e2e-worker-w4-680b7b/simulator-w4-680b7b-kserve-6c4f574c56-p5xtc" Nov 02 06:28:26.292392 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:28:26.292369 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be8a1039-5934-4885-93fe-070575f936ec-tmp-dir\") pod \"simulator-w4-680b7b-kserve-6c4f574c56-p5xtc\" (UID: \"be8a1039-5934-4885-93fe-070575f936ec\") " pod="e2e-models-e2e-worker-w4-680b7b/simulator-w4-680b7b-kserve-6c4f574c56-p5xtc" Nov 02 06:28:26.292480 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:28:26.292415 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be8a1039-5934-4885-93fe-070575f936ec-home\") pod \"simulator-w4-680b7b-kserve-6c4f574c56-p5xtc\" (UID: \"be8a1039-5934-4885-93fe-070575f936ec\") " pod="e2e-models-e2e-worker-w4-680b7b/simulator-w4-680b7b-kserve-6c4f574c56-p5xtc" Nov 02 06:28:26.292480 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:28:26.292419 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be8a1039-5934-4885-93fe-070575f936ec-model-cache\") pod \"simulator-w4-680b7b-kserve-6c4f574c56-p5xtc\" (UID: \"be8a1039-5934-4885-93fe-070575f936ec\") " pod="e2e-models-e2e-worker-w4-680b7b/simulator-w4-680b7b-kserve-6c4f574c56-p5xtc" Nov 02 06:28:26.294156 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:28:26.294129 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be8a1039-5934-4885-93fe-070575f936ec-dshm\") pod \"simulator-w4-680b7b-kserve-6c4f574c56-p5xtc\" (UID: \"be8a1039-5934-4885-93fe-070575f936ec\") " pod="e2e-models-e2e-worker-w4-680b7b/simulator-w4-680b7b-kserve-6c4f574c56-p5xtc" Nov 02 06:28:26.294370 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:28:26.294353 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be8a1039-5934-4885-93fe-070575f936ec-tls-certs\") pod \"simulator-w4-680b7b-kserve-6c4f574c56-p5xtc\" (UID: \"be8a1039-5934-4885-93fe-070575f936ec\") " pod="e2e-models-e2e-worker-w4-680b7b/simulator-w4-680b7b-kserve-6c4f574c56-p5xtc" Nov 02 06:28:26.350974 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:28:26.350763 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvvxp\" (UniqueName: \"kubernetes.io/projected/be8a1039-5934-4885-93fe-070575f936ec-kube-api-access-kvvxp\") pod \"simulator-w4-680b7b-kserve-6c4f574c56-p5xtc\" (UID: \"be8a1039-5934-4885-93fe-070575f936ec\") " pod="e2e-models-e2e-worker-w4-680b7b/simulator-w4-680b7b-kserve-6c4f574c56-p5xtc" Nov 02 06:28:26.371281 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:28:26.371245 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-680b7b/simulator-w4-680b7b-kserve-6c4f574c56-p5xtc" Nov 02 06:28:26.591768 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:28:26.591737 5916 patch_prober.go:29] interesting pod/e2e-worker-w6-6ff85e-openshift-default-698585659c-rvhhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Nov 02 06:28:26.591955 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:28:26.591788 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-rvhhk" podUID="8d74d644-1bbf-4522-8075-43b1f476f656" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Nov 02 06:28:26.759040 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:28:26.759000 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-680b7b/simulator-w4-680b7b-kserve-6c4f574c56-p5xtc"] Nov 02 06:28:26.764250 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:28:26.764219 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbe8a1039_5934_4885_93fe_070575f936ec.slice/crio-afbcb7dee987643a452dc61651bc8586babebf5bd24118a68a970ee48c611716 WatchSource:0}: Error finding container afbcb7dee987643a452dc61651bc8586babebf5bd24118a68a970ee48c611716: Status 404 returned error can't find the container with id afbcb7dee987643a452dc61651bc8586babebf5bd24118a68a970ee48c611716 Nov 02 06:28:27.176398 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:28:27.176358 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-680b7b/simulator-w4-680b7b-kserve-6c4f574c56-p5xtc" event={"ID":"be8a1039-5934-4885-93fe-070575f936ec","Type":"ContainerStarted","Data":"a928abd80efbbfa38f816d3f018ed4f0c9a479d962794cd394ccba05a4b2fa80"} Nov 02 06:28:27.176398 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:28:27.176405 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-680b7b/simulator-w4-680b7b-kserve-6c4f574c56-p5xtc" event={"ID":"be8a1039-5934-4885-93fe-070575f936ec","Type":"ContainerStarted","Data":"afbcb7dee987643a452dc61651bc8586babebf5bd24118a68a970ee48c611716"} Nov 02 06:28:27.288685 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:28:27.288631 5916 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-680b7b/simulator-w4-680b7b-kserve-6c4f574c56-p5xtc" podStartSLOduration=1.2886154140000001 podStartE2EDuration="1.288615414s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:28:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:28:27.286811338 +0000 UTC m=+695.558631731" watchObservedRunningTime="2026-11-02 06:28:27.288615414 +0000 UTC m=+695.560435812" Nov 02 06:28:27.372441 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:28:27.372397 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-680b7b/simulator-w4-680b7b-kserve-6c4f574c56-p5xtc" Nov 02 06:28:27.591963 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:28:27.591857 5916 patch_prober.go:29] interesting pod/e2e-worker-w6-6ff85e-openshift-default-698585659c-rvhhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Nov 02 06:28:27.591963 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:28:27.591931 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-rvhhk" podUID="8d74d644-1bbf-4522-8075-43b1f476f656" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Nov 02 06:28:28.596266 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:28:28.596232 5916 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-rvhhk" Nov 02 06:28:28.596685 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:28:28.596536 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-rvhhk" Nov 02 06:28:28.597349 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:28:28.597325 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-rvhhk" Nov 02 06:28:36.372155 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:28:36.372116 5916 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-680b7b/simulator-w4-680b7b-kserve-6c4f574c56-p5xtc" Nov 02 06:28:36.384379 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:28:36.384356 5916 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-680b7b/simulator-w4-680b7b-kserve-6c4f574c56-p5xtc" Nov 02 06:28:36.541806 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:28:36.541777 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-cdfce9/simulator-w1-cdfce9-kserve-7ffd68f78-5vwl2"] Nov 02 06:28:36.545990 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:28:36.545965 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-cdfce9/simulator-w1-cdfce9-kserve-7ffd68f78-5vwl2" Nov 02 06:28:36.593495 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:28:36.593466 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvwlv\" (UniqueName: \"kubernetes.io/projected/b8b219aa-6a59-4d31-96f6-8d50ab910aca-kube-api-access-zvwlv\") pod \"simulator-w1-cdfce9-kserve-7ffd68f78-5vwl2\" (UID: \"b8b219aa-6a59-4d31-96f6-8d50ab910aca\") " pod="e2e-models-e2e-worker-w1-cdfce9/simulator-w1-cdfce9-kserve-7ffd68f78-5vwl2" Nov 02 06:28:36.593671 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:28:36.593532 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8b219aa-6a59-4d31-96f6-8d50ab910aca-tmp-dir\") pod \"simulator-w1-cdfce9-kserve-7ffd68f78-5vwl2\" (UID: \"b8b219aa-6a59-4d31-96f6-8d50ab910aca\") " pod="e2e-models-e2e-worker-w1-cdfce9/simulator-w1-cdfce9-kserve-7ffd68f78-5vwl2" Nov 02 06:28:36.593671 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:28:36.593616 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8b219aa-6a59-4d31-96f6-8d50ab910aca-model-cache\") pod \"simulator-w1-cdfce9-kserve-7ffd68f78-5vwl2\" (UID: \"b8b219aa-6a59-4d31-96f6-8d50ab910aca\") " pod="e2e-models-e2e-worker-w1-cdfce9/simulator-w1-cdfce9-kserve-7ffd68f78-5vwl2" Nov 02 06:28:36.593671 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:28:36.593661 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8b219aa-6a59-4d31-96f6-8d50ab910aca-home\") pod \"simulator-w1-cdfce9-kserve-7ffd68f78-5vwl2\" (UID: \"b8b219aa-6a59-4d31-96f6-8d50ab910aca\") " pod="e2e-models-e2e-worker-w1-cdfce9/simulator-w1-cdfce9-kserve-7ffd68f78-5vwl2" Nov 02 06:28:36.593788 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:28:36.593765 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8b219aa-6a59-4d31-96f6-8d50ab910aca-dshm\") pod \"simulator-w1-cdfce9-kserve-7ffd68f78-5vwl2\" (UID: \"b8b219aa-6a59-4d31-96f6-8d50ab910aca\") " pod="e2e-models-e2e-worker-w1-cdfce9/simulator-w1-cdfce9-kserve-7ffd68f78-5vwl2" Nov 02 06:28:36.593788 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:28:36.593782 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8b219aa-6a59-4d31-96f6-8d50ab910aca-tls-certs\") pod \"simulator-w1-cdfce9-kserve-7ffd68f78-5vwl2\" (UID: \"b8b219aa-6a59-4d31-96f6-8d50ab910aca\") " pod="e2e-models-e2e-worker-w1-cdfce9/simulator-w1-cdfce9-kserve-7ffd68f78-5vwl2" Nov 02 06:28:36.597317 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:28:36.597294 5916 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-cdfce9\"/\"openshift-service-ca.crt\"" Nov 02 06:28:36.597317 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:28:36.597311 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-cdfce9\"/\"default-dockercfg-28q6p\"" Nov 02 06:28:36.597499 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:28:36.597320 5916 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-cdfce9\"/\"kube-root-ca.crt\"" Nov 02 06:28:36.597499 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:28:36.597343 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-cdfce9\"/\"simulator-w1-cdfce9-kserve-self-signed-certs\"" Nov 02 06:28:36.600691 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:28:36.600664 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-cdfce9/simulator-w1-cdfce9-kserve-7ffd68f78-5vwl2"] Nov 02 06:28:36.694582 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:28:36.694549 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8b219aa-6a59-4d31-96f6-8d50ab910aca-tmp-dir\") pod \"simulator-w1-cdfce9-kserve-7ffd68f78-5vwl2\" (UID: \"b8b219aa-6a59-4d31-96f6-8d50ab910aca\") " pod="e2e-models-e2e-worker-w1-cdfce9/simulator-w1-cdfce9-kserve-7ffd68f78-5vwl2" Nov 02 06:28:36.694582 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:28:36.694587 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8b219aa-6a59-4d31-96f6-8d50ab910aca-model-cache\") pod \"simulator-w1-cdfce9-kserve-7ffd68f78-5vwl2\" (UID: \"b8b219aa-6a59-4d31-96f6-8d50ab910aca\") " pod="e2e-models-e2e-worker-w1-cdfce9/simulator-w1-cdfce9-kserve-7ffd68f78-5vwl2" Nov 02 06:28:36.694806 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:28:36.694606 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8b219aa-6a59-4d31-96f6-8d50ab910aca-home\") pod \"simulator-w1-cdfce9-kserve-7ffd68f78-5vwl2\" (UID: \"b8b219aa-6a59-4d31-96f6-8d50ab910aca\") " pod="e2e-models-e2e-worker-w1-cdfce9/simulator-w1-cdfce9-kserve-7ffd68f78-5vwl2" Nov 02 06:28:36.694806 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:28:36.694661 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8b219aa-6a59-4d31-96f6-8d50ab910aca-dshm\") pod \"simulator-w1-cdfce9-kserve-7ffd68f78-5vwl2\" (UID: \"b8b219aa-6a59-4d31-96f6-8d50ab910aca\") " pod="e2e-models-e2e-worker-w1-cdfce9/simulator-w1-cdfce9-kserve-7ffd68f78-5vwl2" Nov 02 06:28:36.694806 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:28:36.694678 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8b219aa-6a59-4d31-96f6-8d50ab910aca-tls-certs\") pod \"simulator-w1-cdfce9-kserve-7ffd68f78-5vwl2\" (UID: \"b8b219aa-6a59-4d31-96f6-8d50ab910aca\") " pod="e2e-models-e2e-worker-w1-cdfce9/simulator-w1-cdfce9-kserve-7ffd68f78-5vwl2" Nov 02 06:28:36.694806 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:28:36.694743 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvwlv\" (UniqueName: \"kubernetes.io/projected/b8b219aa-6a59-4d31-96f6-8d50ab910aca-kube-api-access-zvwlv\") pod \"simulator-w1-cdfce9-kserve-7ffd68f78-5vwl2\" (UID: \"b8b219aa-6a59-4d31-96f6-8d50ab910aca\") " pod="e2e-models-e2e-worker-w1-cdfce9/simulator-w1-cdfce9-kserve-7ffd68f78-5vwl2" Nov 02 06:28:36.695053 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:28:36.695033 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8b219aa-6a59-4d31-96f6-8d50ab910aca-model-cache\") pod \"simulator-w1-cdfce9-kserve-7ffd68f78-5vwl2\" (UID: \"b8b219aa-6a59-4d31-96f6-8d50ab910aca\") " pod="e2e-models-e2e-worker-w1-cdfce9/simulator-w1-cdfce9-kserve-7ffd68f78-5vwl2" Nov 02 06:28:36.695091 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:28:36.695064 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8b219aa-6a59-4d31-96f6-8d50ab910aca-tmp-dir\") pod \"simulator-w1-cdfce9-kserve-7ffd68f78-5vwl2\" (UID: \"b8b219aa-6a59-4d31-96f6-8d50ab910aca\") " pod="e2e-models-e2e-worker-w1-cdfce9/simulator-w1-cdfce9-kserve-7ffd68f78-5vwl2" Nov 02 06:28:36.695128 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:28:36.695116 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8b219aa-6a59-4d31-96f6-8d50ab910aca-home\") pod \"simulator-w1-cdfce9-kserve-7ffd68f78-5vwl2\" (UID: \"b8b219aa-6a59-4d31-96f6-8d50ab910aca\") " pod="e2e-models-e2e-worker-w1-cdfce9/simulator-w1-cdfce9-kserve-7ffd68f78-5vwl2" Nov 02 06:28:36.696992 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:28:36.696961 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8b219aa-6a59-4d31-96f6-8d50ab910aca-dshm\") pod \"simulator-w1-cdfce9-kserve-7ffd68f78-5vwl2\" (UID: \"b8b219aa-6a59-4d31-96f6-8d50ab910aca\") " pod="e2e-models-e2e-worker-w1-cdfce9/simulator-w1-cdfce9-kserve-7ffd68f78-5vwl2" Nov 02 06:28:36.697285 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:28:36.697269 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8b219aa-6a59-4d31-96f6-8d50ab910aca-tls-certs\") pod \"simulator-w1-cdfce9-kserve-7ffd68f78-5vwl2\" (UID: \"b8b219aa-6a59-4d31-96f6-8d50ab910aca\") " pod="e2e-models-e2e-worker-w1-cdfce9/simulator-w1-cdfce9-kserve-7ffd68f78-5vwl2" Nov 02 06:28:36.750229 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:28:36.750197 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvwlv\" (UniqueName: \"kubernetes.io/projected/b8b219aa-6a59-4d31-96f6-8d50ab910aca-kube-api-access-zvwlv\") pod \"simulator-w1-cdfce9-kserve-7ffd68f78-5vwl2\" (UID: \"b8b219aa-6a59-4d31-96f6-8d50ab910aca\") " pod="e2e-models-e2e-worker-w1-cdfce9/simulator-w1-cdfce9-kserve-7ffd68f78-5vwl2" Nov 02 06:28:36.859992 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:28:36.859950 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-cdfce9/simulator-w1-cdfce9-kserve-7ffd68f78-5vwl2" Nov 02 06:28:37.232615 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:28:37.232585 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-cdfce9/simulator-w1-cdfce9-kserve-7ffd68f78-5vwl2"] Nov 02 06:28:37.238619 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:28:37.238586 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb8b219aa_6a59_4d31_96f6_8d50ab910aca.slice/crio-3aa4436b28ac1b05648bc671ff17e9e22d93972aa56abf7c7ba4065bcd91dd8a WatchSource:0}: Error finding container 3aa4436b28ac1b05648bc671ff17e9e22d93972aa56abf7c7ba4065bcd91dd8a: Status 404 returned error can't find the container with id 3aa4436b28ac1b05648bc671ff17e9e22d93972aa56abf7c7ba4065bcd91dd8a Nov 02 06:28:37.251031 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:28:37.251003 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-680b7b/simulator-w4-680b7b-kserve-6c4f574c56-p5xtc" Nov 02 06:28:38.247859 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:28:38.247820 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-cdfce9/simulator-w1-cdfce9-kserve-7ffd68f78-5vwl2" event={"ID":"b8b219aa-6a59-4d31-96f6-8d50ab910aca","Type":"ContainerStarted","Data":"58d6ff44bea63b532b79c7d9be036bc26050edc4e322bb88a11615a417f77dd4"} Nov 02 06:28:38.248420 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:28:38.247869 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-cdfce9/simulator-w1-cdfce9-kserve-7ffd68f78-5vwl2" event={"ID":"b8b219aa-6a59-4d31-96f6-8d50ab910aca","Type":"ContainerStarted","Data":"3aa4436b28ac1b05648bc671ff17e9e22d93972aa56abf7c7ba4065bcd91dd8a"} Nov 02 06:28:38.357080 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:28:38.357031 5916 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-cdfce9/simulator-w1-cdfce9-kserve-7ffd68f78-5vwl2" podStartSLOduration=2.35701439 podStartE2EDuration="2.35701439s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:28:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:28:38.35488809 +0000 UTC m=+706.626708496" watchObservedRunningTime="2026-11-02 06:28:38.35701439 +0000 UTC m=+706.628834784" Nov 02 06:28:38.860200 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:28:38.860166 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-cdfce9/simulator-w1-cdfce9-kserve-7ffd68f78-5vwl2" Nov 02 06:28:46.860589 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:28:46.860555 5916 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-cdfce9/simulator-w1-cdfce9-kserve-7ffd68f78-5vwl2" Nov 02 06:28:46.872827 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:28:46.872800 5916 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-cdfce9/simulator-w1-cdfce9-kserve-7ffd68f78-5vwl2" Nov 02 06:28:47.315929 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:28:47.315882 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-cdfce9/simulator-w1-cdfce9-kserve-7ffd68f78-5vwl2" Nov 02 06:28:49.893215 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:28:49.893181 5916 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p98cq" containerName="extractor" Nov 02 06:29:07.232305 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:29:07.232271 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-bdcd80/simulator-w0-bdcd80-kserve-5749c847cb-9ktcr"] Nov 02 06:29:07.236260 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:29:07.236235 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-bdcd80/simulator-w0-bdcd80-kserve-5749c847cb-9ktcr" Nov 02 06:29:07.285276 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:29:07.285242 5916 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-bdcd80\"/\"kube-root-ca.crt\"" Nov 02 06:29:07.287930 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:29:07.287891 5916 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-bdcd80\"/\"openshift-service-ca.crt\"" Nov 02 06:29:07.334027 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:29:07.333993 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-bdcd80\"/\"simulator-w0-bdcd80-kserve-self-signed-certs\"" Nov 02 06:29:07.334197 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:29:07.334053 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-bdcd80/simulator-w0-bdcd80-kserve-5749c847cb-9ktcr"] Nov 02 06:29:07.334252 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:29:07.334197 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-bdcd80\"/\"default-dockercfg-6lvdv\"" Nov 02 06:29:07.426189 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:29:07.426156 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6tc7\" (UniqueName: \"kubernetes.io/projected/2043ff0a-8a71-4b71-9750-beb566a12bad-kube-api-access-n6tc7\") pod \"simulator-w0-bdcd80-kserve-5749c847cb-9ktcr\" (UID: \"2043ff0a-8a71-4b71-9750-beb566a12bad\") " pod="e2e-models-e2e-worker-w0-bdcd80/simulator-w0-bdcd80-kserve-5749c847cb-9ktcr" Nov 02 06:29:07.426367 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:29:07.426222 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2043ff0a-8a71-4b71-9750-beb566a12bad-model-cache\") pod \"simulator-w0-bdcd80-kserve-5749c847cb-9ktcr\" (UID: \"2043ff0a-8a71-4b71-9750-beb566a12bad\") " pod="e2e-models-e2e-worker-w0-bdcd80/simulator-w0-bdcd80-kserve-5749c847cb-9ktcr" Nov 02 06:29:07.426367 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:29:07.426331 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2043ff0a-8a71-4b71-9750-beb566a12bad-home\") pod \"simulator-w0-bdcd80-kserve-5749c847cb-9ktcr\" (UID: \"2043ff0a-8a71-4b71-9750-beb566a12bad\") " pod="e2e-models-e2e-worker-w0-bdcd80/simulator-w0-bdcd80-kserve-5749c847cb-9ktcr" Nov 02 06:29:07.426460 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:29:07.426378 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2043ff0a-8a71-4b71-9750-beb566a12bad-dshm\") pod \"simulator-w0-bdcd80-kserve-5749c847cb-9ktcr\" (UID: \"2043ff0a-8a71-4b71-9750-beb566a12bad\") " pod="e2e-models-e2e-worker-w0-bdcd80/simulator-w0-bdcd80-kserve-5749c847cb-9ktcr" Nov 02 06:29:07.426460 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:29:07.426398 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2043ff0a-8a71-4b71-9750-beb566a12bad-tmp-dir\") pod \"simulator-w0-bdcd80-kserve-5749c847cb-9ktcr\" (UID: \"2043ff0a-8a71-4b71-9750-beb566a12bad\") " pod="e2e-models-e2e-worker-w0-bdcd80/simulator-w0-bdcd80-kserve-5749c847cb-9ktcr" Nov 02 06:29:07.426546 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:29:07.426521 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2043ff0a-8a71-4b71-9750-beb566a12bad-tls-certs\") pod \"simulator-w0-bdcd80-kserve-5749c847cb-9ktcr\" (UID: \"2043ff0a-8a71-4b71-9750-beb566a12bad\") " pod="e2e-models-e2e-worker-w0-bdcd80/simulator-w0-bdcd80-kserve-5749c847cb-9ktcr" Nov 02 06:29:07.527453 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:29:07.527357 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6tc7\" (UniqueName: \"kubernetes.io/projected/2043ff0a-8a71-4b71-9750-beb566a12bad-kube-api-access-n6tc7\") pod \"simulator-w0-bdcd80-kserve-5749c847cb-9ktcr\" (UID: \"2043ff0a-8a71-4b71-9750-beb566a12bad\") " pod="e2e-models-e2e-worker-w0-bdcd80/simulator-w0-bdcd80-kserve-5749c847cb-9ktcr" Nov 02 06:29:07.527453 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:29:07.527428 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2043ff0a-8a71-4b71-9750-beb566a12bad-model-cache\") pod \"simulator-w0-bdcd80-kserve-5749c847cb-9ktcr\" (UID: \"2043ff0a-8a71-4b71-9750-beb566a12bad\") " pod="e2e-models-e2e-worker-w0-bdcd80/simulator-w0-bdcd80-kserve-5749c847cb-9ktcr" Nov 02 06:29:07.527688 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:29:07.527490 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2043ff0a-8a71-4b71-9750-beb566a12bad-home\") pod \"simulator-w0-bdcd80-kserve-5749c847cb-9ktcr\" (UID: \"2043ff0a-8a71-4b71-9750-beb566a12bad\") " pod="e2e-models-e2e-worker-w0-bdcd80/simulator-w0-bdcd80-kserve-5749c847cb-9ktcr" Nov 02 06:29:07.527688 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:29:07.527539 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2043ff0a-8a71-4b71-9750-beb566a12bad-dshm\") pod \"simulator-w0-bdcd80-kserve-5749c847cb-9ktcr\" (UID: \"2043ff0a-8a71-4b71-9750-beb566a12bad\") " pod="e2e-models-e2e-worker-w0-bdcd80/simulator-w0-bdcd80-kserve-5749c847cb-9ktcr" Nov 02 06:29:07.527688 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:29:07.527562 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2043ff0a-8a71-4b71-9750-beb566a12bad-tmp-dir\") pod \"simulator-w0-bdcd80-kserve-5749c847cb-9ktcr\" (UID: \"2043ff0a-8a71-4b71-9750-beb566a12bad\") " pod="e2e-models-e2e-worker-w0-bdcd80/simulator-w0-bdcd80-kserve-5749c847cb-9ktcr" Nov 02 06:29:07.527688 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:29:07.527647 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2043ff0a-8a71-4b71-9750-beb566a12bad-tls-certs\") pod \"simulator-w0-bdcd80-kserve-5749c847cb-9ktcr\" (UID: \"2043ff0a-8a71-4b71-9750-beb566a12bad\") " pod="e2e-models-e2e-worker-w0-bdcd80/simulator-w0-bdcd80-kserve-5749c847cb-9ktcr" Nov 02 06:29:07.527974 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:29:07.527947 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2043ff0a-8a71-4b71-9750-beb566a12bad-model-cache\") pod \"simulator-w0-bdcd80-kserve-5749c847cb-9ktcr\" (UID: \"2043ff0a-8a71-4b71-9750-beb566a12bad\") " pod="e2e-models-e2e-worker-w0-bdcd80/simulator-w0-bdcd80-kserve-5749c847cb-9ktcr" Nov 02 06:29:07.528234 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:29:07.528209 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2043ff0a-8a71-4b71-9750-beb566a12bad-tmp-dir\") pod \"simulator-w0-bdcd80-kserve-5749c847cb-9ktcr\" (UID: \"2043ff0a-8a71-4b71-9750-beb566a12bad\") " pod="e2e-models-e2e-worker-w0-bdcd80/simulator-w0-bdcd80-kserve-5749c847cb-9ktcr" Nov 02 06:29:07.528481 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:29:07.528450 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2043ff0a-8a71-4b71-9750-beb566a12bad-home\") pod \"simulator-w0-bdcd80-kserve-5749c847cb-9ktcr\" (UID: \"2043ff0a-8a71-4b71-9750-beb566a12bad\") " pod="e2e-models-e2e-worker-w0-bdcd80/simulator-w0-bdcd80-kserve-5749c847cb-9ktcr" Nov 02 06:29:07.530416 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:29:07.530392 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2043ff0a-8a71-4b71-9750-beb566a12bad-tls-certs\") pod \"simulator-w0-bdcd80-kserve-5749c847cb-9ktcr\" (UID: \"2043ff0a-8a71-4b71-9750-beb566a12bad\") " pod="e2e-models-e2e-worker-w0-bdcd80/simulator-w0-bdcd80-kserve-5749c847cb-9ktcr" Nov 02 06:29:07.530511 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:29:07.530390 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2043ff0a-8a71-4b71-9750-beb566a12bad-dshm\") pod \"simulator-w0-bdcd80-kserve-5749c847cb-9ktcr\" (UID: \"2043ff0a-8a71-4b71-9750-beb566a12bad\") " pod="e2e-models-e2e-worker-w0-bdcd80/simulator-w0-bdcd80-kserve-5749c847cb-9ktcr" Nov 02 06:29:07.582262 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:29:07.582227 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6tc7\" (UniqueName: \"kubernetes.io/projected/2043ff0a-8a71-4b71-9750-beb566a12bad-kube-api-access-n6tc7\") pod \"simulator-w0-bdcd80-kserve-5749c847cb-9ktcr\" (UID: \"2043ff0a-8a71-4b71-9750-beb566a12bad\") " pod="e2e-models-e2e-worker-w0-bdcd80/simulator-w0-bdcd80-kserve-5749c847cb-9ktcr" Nov 02 06:29:07.851072 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:29:07.850991 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-bdcd80/simulator-w0-bdcd80-kserve-5749c847cb-9ktcr" Nov 02 06:29:08.230518 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:29:08.230485 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-bdcd80/simulator-w0-bdcd80-kserve-5749c847cb-9ktcr"] Nov 02 06:29:08.235895 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:29:08.235864 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2043ff0a_8a71_4b71_9750_beb566a12bad.slice/crio-cd78adcea75975059a82e0ce6d43dbc9556dae2594f86432e3fbc7653697c8a8 WatchSource:0}: Error finding container cd78adcea75975059a82e0ce6d43dbc9556dae2594f86432e3fbc7653697c8a8: Status 404 returned error can't find the container with id cd78adcea75975059a82e0ce6d43dbc9556dae2594f86432e3fbc7653697c8a8 Nov 02 06:29:08.441933 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:29:08.441880 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-bdcd80/simulator-w0-bdcd80-kserve-5749c847cb-9ktcr" event={"ID":"2043ff0a-8a71-4b71-9750-beb566a12bad","Type":"ContainerStarted","Data":"4d71cefed7597633799490ffef0d1abf6c665ef7b6629a4e265df95957a34be7"} Nov 02 06:29:08.442130 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:29:08.441941 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-bdcd80/simulator-w0-bdcd80-kserve-5749c847cb-9ktcr" event={"ID":"2043ff0a-8a71-4b71-9750-beb566a12bad","Type":"ContainerStarted","Data":"cd78adcea75975059a82e0ce6d43dbc9556dae2594f86432e3fbc7653697c8a8"} Nov 02 06:29:08.550546 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:29:08.550444 5916 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-bdcd80/simulator-w0-bdcd80-kserve-5749c847cb-9ktcr" podStartSLOduration=1.5504289629999999 podStartE2EDuration="1.550428963s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:29:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:29:08.548990117 +0000 UTC m=+736.820810527" watchObservedRunningTime="2026-11-02 06:29:08.550428963 +0000 UTC m=+736.822249411" Nov 02 06:29:08.851926 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:29:08.851827 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-bdcd80/simulator-w0-bdcd80-kserve-5749c847cb-9ktcr" Nov 02 06:29:17.851318 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:29:17.851276 5916 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-bdcd80/simulator-w0-bdcd80-kserve-5749c847cb-9ktcr" Nov 02 06:29:17.863120 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:29:17.863093 5916 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-bdcd80/simulator-w0-bdcd80-kserve-5749c847cb-9ktcr" Nov 02 06:29:18.514484 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:29:18.514447 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-bdcd80/simulator-w0-bdcd80-kserve-5749c847cb-9ktcr" Nov 02 06:29:19.894382 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:29:19.894350 5916 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p98cq" containerName="extractor" Nov 02 06:29:32.519647 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:29:32.519613 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-6ff85e/simulator-w6-6ff85e-kserve-9b9fd978f-q76nl"] Nov 02 06:29:32.524858 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:29:32.524832 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-6ff85e/simulator-w6-6ff85e-kserve-9b9fd978f-q76nl" Nov 02 06:29:32.543779 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:29:32.543743 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30114285-d730-4abb-a618-6833d5746feb-tls-certs\") pod \"simulator-w6-6ff85e-kserve-9b9fd978f-q76nl\" (UID: \"30114285-d730-4abb-a618-6833d5746feb\") " pod="e2e-models-e2e-worker-w6-6ff85e/simulator-w6-6ff85e-kserve-9b9fd978f-q76nl" Nov 02 06:29:32.543972 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:29:32.543799 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30114285-d730-4abb-a618-6833d5746feb-tmp-dir\") pod \"simulator-w6-6ff85e-kserve-9b9fd978f-q76nl\" (UID: \"30114285-d730-4abb-a618-6833d5746feb\") " pod="e2e-models-e2e-worker-w6-6ff85e/simulator-w6-6ff85e-kserve-9b9fd978f-q76nl" Nov 02 06:29:32.544048 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:29:32.543984 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30114285-d730-4abb-a618-6833d5746feb-model-cache\") pod \"simulator-w6-6ff85e-kserve-9b9fd978f-q76nl\" (UID: \"30114285-d730-4abb-a618-6833d5746feb\") " pod="e2e-models-e2e-worker-w6-6ff85e/simulator-w6-6ff85e-kserve-9b9fd978f-q76nl" Nov 02 06:29:32.544102 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:29:32.544078 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30114285-d730-4abb-a618-6833d5746feb-home\") pod \"simulator-w6-6ff85e-kserve-9b9fd978f-q76nl\" (UID: \"30114285-d730-4abb-a618-6833d5746feb\") " pod="e2e-models-e2e-worker-w6-6ff85e/simulator-w6-6ff85e-kserve-9b9fd978f-q76nl" Nov 02 06:29:32.544160 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:29:32.544141 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30114285-d730-4abb-a618-6833d5746feb-dshm\") pod \"simulator-w6-6ff85e-kserve-9b9fd978f-q76nl\" (UID: \"30114285-d730-4abb-a618-6833d5746feb\") " pod="e2e-models-e2e-worker-w6-6ff85e/simulator-w6-6ff85e-kserve-9b9fd978f-q76nl" Nov 02 06:29:32.544215 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:29:32.544202 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5rtk\" (UniqueName: \"kubernetes.io/projected/30114285-d730-4abb-a618-6833d5746feb-kube-api-access-x5rtk\") pod \"simulator-w6-6ff85e-kserve-9b9fd978f-q76nl\" (UID: \"30114285-d730-4abb-a618-6833d5746feb\") " pod="e2e-models-e2e-worker-w6-6ff85e/simulator-w6-6ff85e-kserve-9b9fd978f-q76nl" Nov 02 06:29:32.576464 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:29:32.576434 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-6ff85e/simulator-w6-6ff85e-kserve-9b9fd978f-q76nl"] Nov 02 06:29:32.619480 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:29:32.619422 5916 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-6ff85e\"/\"kube-root-ca.crt\"" Nov 02 06:29:32.619653 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:29:32.619509 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-6ff85e\"/\"default-dockercfg-8vqbx\"" Nov 02 06:29:32.619653 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:29:32.619556 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-6ff85e\"/\"simulator-w6-6ff85e-kserve-self-signed-certs\"" Nov 02 06:29:32.619759 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:29:32.619667 5916 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-6ff85e\"/\"openshift-service-ca.crt\"" Nov 02 06:29:32.645043 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:29:32.645015 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30114285-d730-4abb-a618-6833d5746feb-home\") pod \"simulator-w6-6ff85e-kserve-9b9fd978f-q76nl\" (UID: \"30114285-d730-4abb-a618-6833d5746feb\") " pod="e2e-models-e2e-worker-w6-6ff85e/simulator-w6-6ff85e-kserve-9b9fd978f-q76nl" Nov 02 06:29:32.645166 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:29:32.645060 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30114285-d730-4abb-a618-6833d5746feb-dshm\") pod \"simulator-w6-6ff85e-kserve-9b9fd978f-q76nl\" (UID: \"30114285-d730-4abb-a618-6833d5746feb\") " pod="e2e-models-e2e-worker-w6-6ff85e/simulator-w6-6ff85e-kserve-9b9fd978f-q76nl" Nov 02 06:29:32.645166 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:29:32.645092 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5rtk\" (UniqueName: \"kubernetes.io/projected/30114285-d730-4abb-a618-6833d5746feb-kube-api-access-x5rtk\") pod \"simulator-w6-6ff85e-kserve-9b9fd978f-q76nl\" (UID: \"30114285-d730-4abb-a618-6833d5746feb\") " pod="e2e-models-e2e-worker-w6-6ff85e/simulator-w6-6ff85e-kserve-9b9fd978f-q76nl" Nov 02 06:29:32.645166 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:29:32.645138 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30114285-d730-4abb-a618-6833d5746feb-tls-certs\") pod \"simulator-w6-6ff85e-kserve-9b9fd978f-q76nl\" (UID: \"30114285-d730-4abb-a618-6833d5746feb\") " pod="e2e-models-e2e-worker-w6-6ff85e/simulator-w6-6ff85e-kserve-9b9fd978f-q76nl" Nov 02 06:29:32.645166 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:29:32.645156 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30114285-d730-4abb-a618-6833d5746feb-tmp-dir\") pod \"simulator-w6-6ff85e-kserve-9b9fd978f-q76nl\" (UID: \"30114285-d730-4abb-a618-6833d5746feb\") " pod="e2e-models-e2e-worker-w6-6ff85e/simulator-w6-6ff85e-kserve-9b9fd978f-q76nl" Nov 02 06:29:32.645376 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:29:32.645172 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30114285-d730-4abb-a618-6833d5746feb-model-cache\") pod \"simulator-w6-6ff85e-kserve-9b9fd978f-q76nl\" (UID: \"30114285-d730-4abb-a618-6833d5746feb\") " pod="e2e-models-e2e-worker-w6-6ff85e/simulator-w6-6ff85e-kserve-9b9fd978f-q76nl" Nov 02 06:29:32.645540 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:29:32.645519 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30114285-d730-4abb-a618-6833d5746feb-home\") pod \"simulator-w6-6ff85e-kserve-9b9fd978f-q76nl\" (UID: \"30114285-d730-4abb-a618-6833d5746feb\") " pod="e2e-models-e2e-worker-w6-6ff85e/simulator-w6-6ff85e-kserve-9b9fd978f-q76nl" Nov 02 06:29:32.645607 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:29:32.645580 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30114285-d730-4abb-a618-6833d5746feb-model-cache\") pod \"simulator-w6-6ff85e-kserve-9b9fd978f-q76nl\" (UID: \"30114285-d730-4abb-a618-6833d5746feb\") " pod="e2e-models-e2e-worker-w6-6ff85e/simulator-w6-6ff85e-kserve-9b9fd978f-q76nl" Nov 02 06:29:32.645656 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:29:32.645610 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30114285-d730-4abb-a618-6833d5746feb-tmp-dir\") pod \"simulator-w6-6ff85e-kserve-9b9fd978f-q76nl\" (UID: \"30114285-d730-4abb-a618-6833d5746feb\") " pod="e2e-models-e2e-worker-w6-6ff85e/simulator-w6-6ff85e-kserve-9b9fd978f-q76nl" Nov 02 06:29:32.647253 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:29:32.647227 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30114285-d730-4abb-a618-6833d5746feb-dshm\") pod \"simulator-w6-6ff85e-kserve-9b9fd978f-q76nl\" (UID: \"30114285-d730-4abb-a618-6833d5746feb\") " pod="e2e-models-e2e-worker-w6-6ff85e/simulator-w6-6ff85e-kserve-9b9fd978f-q76nl" Nov 02 06:29:32.647504 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:29:32.647489 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30114285-d730-4abb-a618-6833d5746feb-tls-certs\") pod \"simulator-w6-6ff85e-kserve-9b9fd978f-q76nl\" (UID: \"30114285-d730-4abb-a618-6833d5746feb\") " pod="e2e-models-e2e-worker-w6-6ff85e/simulator-w6-6ff85e-kserve-9b9fd978f-q76nl" Nov 02 06:29:32.711837 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:29:32.711810 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5rtk\" (UniqueName: \"kubernetes.io/projected/30114285-d730-4abb-a618-6833d5746feb-kube-api-access-x5rtk\") pod \"simulator-w6-6ff85e-kserve-9b9fd978f-q76nl\" (UID: \"30114285-d730-4abb-a618-6833d5746feb\") " pod="e2e-models-e2e-worker-w6-6ff85e/simulator-w6-6ff85e-kserve-9b9fd978f-q76nl" Nov 02 06:29:32.837682 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:29:32.837594 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-6ff85e/simulator-w6-6ff85e-kserve-9b9fd978f-q76nl" Nov 02 06:29:33.217035 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:29:33.216992 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-6ff85e/simulator-w6-6ff85e-kserve-9b9fd978f-q76nl"] Nov 02 06:29:33.222000 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:29:33.221970 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod30114285_d730_4abb_a618_6833d5746feb.slice/crio-9bc4e63c81f118f24274f025423f90bdb072a07484a9ded2613bba329a293d85 WatchSource:0}: Error finding container 9bc4e63c81f118f24274f025423f90bdb072a07484a9ded2613bba329a293d85: Status 404 returned error can't find the container with id 9bc4e63c81f118f24274f025423f90bdb072a07484a9ded2613bba329a293d85 Nov 02 06:29:33.596925 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:29:33.596811 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-6ff85e/simulator-w6-6ff85e-kserve-9b9fd978f-q76nl" event={"ID":"30114285-d730-4abb-a618-6833d5746feb","Type":"ContainerStarted","Data":"fc5bf7005819795b62cfed8b3b6ed7aaafa51099457a92ab763b25e22c585130"} Nov 02 06:29:33.596925 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:29:33.596860 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-6ff85e/simulator-w6-6ff85e-kserve-9b9fd978f-q76nl" event={"ID":"30114285-d730-4abb-a618-6833d5746feb","Type":"ContainerStarted","Data":"9bc4e63c81f118f24274f025423f90bdb072a07484a9ded2613bba329a293d85"} Nov 02 06:29:33.703189 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:29:33.703136 5916 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-6ff85e/simulator-w6-6ff85e-kserve-9b9fd978f-q76nl" podStartSLOduration=1.703120963 podStartE2EDuration="1.703120963s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:29:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:29:33.702569225 +0000 UTC m=+761.974389629" watchObservedRunningTime="2026-11-02 06:29:33.703120963 +0000 UTC m=+761.974941359" Nov 02 06:29:33.841511 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:29:33.841481 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-6ff85e/simulator-w6-6ff85e-kserve-9b9fd978f-q76nl" Nov 02 06:29:42.838270 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:29:42.838233 5916 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-6ff85e/simulator-w6-6ff85e-kserve-9b9fd978f-q76nl" Nov 02 06:29:42.850701 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:29:42.850674 5916 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-6ff85e/simulator-w6-6ff85e-kserve-9b9fd978f-q76nl" Nov 02 06:29:43.686437 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:29:43.686409 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-6ff85e/simulator-w6-6ff85e-kserve-9b9fd978f-q76nl" Nov 02 06:29:49.893450 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:29:49.893409 5916 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p98cq" containerName="extractor" Nov 02 06:29:52.141646 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:29:52.141609 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-cdfce9/premium-w1-cdfce9-kserve-598cb9d57f-mrzrz"] Nov 02 06:29:52.146006 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:29:52.145985 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-cdfce9/premium-w1-cdfce9-kserve-598cb9d57f-mrzrz" Nov 02 06:29:52.199044 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:29:52.199010 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-cdfce9/premium-w1-cdfce9-kserve-598cb9d57f-mrzrz"] Nov 02 06:29:52.241880 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:29:52.241854 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-cdfce9\"/\"premium-w1-cdfce9-kserve-self-signed-certs\"" Nov 02 06:29:52.245448 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:29:52.245423 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9c5c1f7-cdca-46d6-8573-c85af1cd1946-model-cache\") pod \"premium-w1-cdfce9-kserve-598cb9d57f-mrzrz\" (UID: \"f9c5c1f7-cdca-46d6-8573-c85af1cd1946\") " pod="e2e-models-e2e-worker-w1-cdfce9/premium-w1-cdfce9-kserve-598cb9d57f-mrzrz" Nov 02 06:29:52.245549 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:29:52.245469 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmlm2\" (UniqueName: \"kubernetes.io/projected/f9c5c1f7-cdca-46d6-8573-c85af1cd1946-kube-api-access-wmlm2\") pod \"premium-w1-cdfce9-kserve-598cb9d57f-mrzrz\" (UID: \"f9c5c1f7-cdca-46d6-8573-c85af1cd1946\") " pod="e2e-models-e2e-worker-w1-cdfce9/premium-w1-cdfce9-kserve-598cb9d57f-mrzrz" Nov 02 06:29:52.245610 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:29:52.245591 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9c5c1f7-cdca-46d6-8573-c85af1cd1946-tls-certs\") pod \"premium-w1-cdfce9-kserve-598cb9d57f-mrzrz\" (UID: \"f9c5c1f7-cdca-46d6-8573-c85af1cd1946\") " pod="e2e-models-e2e-worker-w1-cdfce9/premium-w1-cdfce9-kserve-598cb9d57f-mrzrz" Nov 02 06:29:52.245716 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:29:52.245701 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9c5c1f7-cdca-46d6-8573-c85af1cd1946-tmp-dir\") pod \"premium-w1-cdfce9-kserve-598cb9d57f-mrzrz\" (UID: \"f9c5c1f7-cdca-46d6-8573-c85af1cd1946\") " pod="e2e-models-e2e-worker-w1-cdfce9/premium-w1-cdfce9-kserve-598cb9d57f-mrzrz" Nov 02 06:29:52.245785 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:29:52.245769 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9c5c1f7-cdca-46d6-8573-c85af1cd1946-dshm\") pod \"premium-w1-cdfce9-kserve-598cb9d57f-mrzrz\" (UID: \"f9c5c1f7-cdca-46d6-8573-c85af1cd1946\") " pod="e2e-models-e2e-worker-w1-cdfce9/premium-w1-cdfce9-kserve-598cb9d57f-mrzrz" Nov 02 06:29:52.245861 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:29:52.245821 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9c5c1f7-cdca-46d6-8573-c85af1cd1946-home\") pod \"premium-w1-cdfce9-kserve-598cb9d57f-mrzrz\" (UID: \"f9c5c1f7-cdca-46d6-8573-c85af1cd1946\") " pod="e2e-models-e2e-worker-w1-cdfce9/premium-w1-cdfce9-kserve-598cb9d57f-mrzrz" Nov 02 06:29:52.346470 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:29:52.346418 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9c5c1f7-cdca-46d6-8573-c85af1cd1946-tmp-dir\") pod \"premium-w1-cdfce9-kserve-598cb9d57f-mrzrz\" (UID: \"f9c5c1f7-cdca-46d6-8573-c85af1cd1946\") " pod="e2e-models-e2e-worker-w1-cdfce9/premium-w1-cdfce9-kserve-598cb9d57f-mrzrz" Nov 02 06:29:52.346665 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:29:52.346481 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9c5c1f7-cdca-46d6-8573-c85af1cd1946-dshm\") pod \"premium-w1-cdfce9-kserve-598cb9d57f-mrzrz\" (UID: \"f9c5c1f7-cdca-46d6-8573-c85af1cd1946\") " pod="e2e-models-e2e-worker-w1-cdfce9/premium-w1-cdfce9-kserve-598cb9d57f-mrzrz" Nov 02 06:29:52.346665 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:29:52.346532 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9c5c1f7-cdca-46d6-8573-c85af1cd1946-home\") pod \"premium-w1-cdfce9-kserve-598cb9d57f-mrzrz\" (UID: \"f9c5c1f7-cdca-46d6-8573-c85af1cd1946\") " pod="e2e-models-e2e-worker-w1-cdfce9/premium-w1-cdfce9-kserve-598cb9d57f-mrzrz" Nov 02 06:29:52.346788 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:29:52.346682 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9c5c1f7-cdca-46d6-8573-c85af1cd1946-model-cache\") pod \"premium-w1-cdfce9-kserve-598cb9d57f-mrzrz\" (UID: \"f9c5c1f7-cdca-46d6-8573-c85af1cd1946\") " pod="e2e-models-e2e-worker-w1-cdfce9/premium-w1-cdfce9-kserve-598cb9d57f-mrzrz" Nov 02 06:29:52.346868 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:29:52.346847 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9c5c1f7-cdca-46d6-8573-c85af1cd1946-tmp-dir\") pod \"premium-w1-cdfce9-kserve-598cb9d57f-mrzrz\" (UID: \"f9c5c1f7-cdca-46d6-8573-c85af1cd1946\") " pod="e2e-models-e2e-worker-w1-cdfce9/premium-w1-cdfce9-kserve-598cb9d57f-mrzrz" Nov 02 06:29:52.346963 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:29:52.346944 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9c5c1f7-cdca-46d6-8573-c85af1cd1946-home\") pod \"premium-w1-cdfce9-kserve-598cb9d57f-mrzrz\" (UID: \"f9c5c1f7-cdca-46d6-8573-c85af1cd1946\") " pod="e2e-models-e2e-worker-w1-cdfce9/premium-w1-cdfce9-kserve-598cb9d57f-mrzrz" Nov 02 06:29:52.347230 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:29:52.347213 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9c5c1f7-cdca-46d6-8573-c85af1cd1946-model-cache\") pod \"premium-w1-cdfce9-kserve-598cb9d57f-mrzrz\" (UID: \"f9c5c1f7-cdca-46d6-8573-c85af1cd1946\") " pod="e2e-models-e2e-worker-w1-cdfce9/premium-w1-cdfce9-kserve-598cb9d57f-mrzrz" Nov 02 06:29:52.347306 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:29:52.347292 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmlm2\" (UniqueName: \"kubernetes.io/projected/f9c5c1f7-cdca-46d6-8573-c85af1cd1946-kube-api-access-wmlm2\") pod \"premium-w1-cdfce9-kserve-598cb9d57f-mrzrz\" (UID: \"f9c5c1f7-cdca-46d6-8573-c85af1cd1946\") " pod="e2e-models-e2e-worker-w1-cdfce9/premium-w1-cdfce9-kserve-598cb9d57f-mrzrz" Nov 02 06:29:52.347353 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:29:52.347336 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9c5c1f7-cdca-46d6-8573-c85af1cd1946-tls-certs\") pod \"premium-w1-cdfce9-kserve-598cb9d57f-mrzrz\" (UID: \"f9c5c1f7-cdca-46d6-8573-c85af1cd1946\") " pod="e2e-models-e2e-worker-w1-cdfce9/premium-w1-cdfce9-kserve-598cb9d57f-mrzrz" Nov 02 06:29:52.349012 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:29:52.348987 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9c5c1f7-cdca-46d6-8573-c85af1cd1946-dshm\") pod \"premium-w1-cdfce9-kserve-598cb9d57f-mrzrz\" (UID: \"f9c5c1f7-cdca-46d6-8573-c85af1cd1946\") " pod="e2e-models-e2e-worker-w1-cdfce9/premium-w1-cdfce9-kserve-598cb9d57f-mrzrz" Nov 02 06:29:52.349586 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:29:52.349569 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9c5c1f7-cdca-46d6-8573-c85af1cd1946-tls-certs\") pod \"premium-w1-cdfce9-kserve-598cb9d57f-mrzrz\" (UID: \"f9c5c1f7-cdca-46d6-8573-c85af1cd1946\") " pod="e2e-models-e2e-worker-w1-cdfce9/premium-w1-cdfce9-kserve-598cb9d57f-mrzrz" Nov 02 06:29:52.401175 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:29:52.401107 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmlm2\" (UniqueName: \"kubernetes.io/projected/f9c5c1f7-cdca-46d6-8573-c85af1cd1946-kube-api-access-wmlm2\") pod \"premium-w1-cdfce9-kserve-598cb9d57f-mrzrz\" (UID: \"f9c5c1f7-cdca-46d6-8573-c85af1cd1946\") " pod="e2e-models-e2e-worker-w1-cdfce9/premium-w1-cdfce9-kserve-598cb9d57f-mrzrz" Nov 02 06:29:52.458131 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:29:52.458094 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-cdfce9/premium-w1-cdfce9-kserve-598cb9d57f-mrzrz" Nov 02 06:29:52.858620 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:29:52.858545 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-cdfce9/premium-w1-cdfce9-kserve-598cb9d57f-mrzrz"] Nov 02 06:29:52.863674 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:29:52.863642 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf9c5c1f7_cdca_46d6_8573_c85af1cd1946.slice/crio-407fada787d0e6ed28a7f0688473ad4b7f49737ff236bbb640eab23936fee5ee WatchSource:0}: Error finding container 407fada787d0e6ed28a7f0688473ad4b7f49737ff236bbb640eab23936fee5ee: Status 404 returned error can't find the container with id 407fada787d0e6ed28a7f0688473ad4b7f49737ff236bbb640eab23936fee5ee Nov 02 06:29:53.740349 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:29:53.740307 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-cdfce9/premium-w1-cdfce9-kserve-598cb9d57f-mrzrz" event={"ID":"f9c5c1f7-cdca-46d6-8573-c85af1cd1946","Type":"ContainerStarted","Data":"5796d3c345467f547f1950bdca52fa5d1c016b58d45968d3677d97eb7ecfc443"} Nov 02 06:29:53.740751 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:29:53.740364 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-cdfce9/premium-w1-cdfce9-kserve-598cb9d57f-mrzrz" event={"ID":"f9c5c1f7-cdca-46d6-8573-c85af1cd1946","Type":"ContainerStarted","Data":"407fada787d0e6ed28a7f0688473ad4b7f49737ff236bbb640eab23936fee5ee"} Nov 02 06:29:53.851247 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:29:53.851198 5916 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-cdfce9/premium-w1-cdfce9-kserve-598cb9d57f-mrzrz" podStartSLOduration=1.851184895 podStartE2EDuration="1.851184895s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:29:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:29:53.849269907 +0000 UTC m=+782.121090310" watchObservedRunningTime="2026-11-02 06:29:53.851184895 +0000 UTC m=+782.123005288" Nov 02 06:29:54.459174 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:29:54.459126 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-cdfce9/premium-w1-cdfce9-kserve-598cb9d57f-mrzrz" Nov 02 06:30:00.161383 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:00.161343 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-680b7b-29893350-5wc2c"] Nov 02 06:30:00.165700 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:00.165662 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-dcf9c3-29893350-88rn7"] Nov 02 06:30:00.165956 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:00.165935 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-680b7b-29893350-5wc2c" Nov 02 06:30:00.170666 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:00.170644 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cdfce9-29893350-v468m"] Nov 02 06:30:00.170781 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:00.170762 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-dcf9c3-29893350-88rn7" Nov 02 06:30:00.174777 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:00.174752 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cdfce9-29893350-v468m" Nov 02 06:30:00.206970 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:00.206944 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-581608-29893350-bvdrk"] Nov 02 06:30:00.210554 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:00.210537 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-581608-29893350-bvdrk" Nov 02 06:30:00.219985 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:00.219962 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-68l4r\"" Nov 02 06:30:00.243648 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:00.243620 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f971998-396b-47d6-a7fb-f5f7df0fc2e7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-680b7b-29893350-5wc2c\" (UID: \"2f971998-396b-47d6-a7fb-f5f7df0fc2e7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-680b7b-29893350-5wc2c" Nov 02 06:30:00.243769 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:00.243678 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55759e02-f1b7-408c-ad1f-c132cd5f31a9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-cdfce9-29893350-v468m\" (UID: \"55759e02-f1b7-408c-ad1f-c132cd5f31a9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cdfce9-29893350-v468m" Nov 02 06:30:00.243769 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:00.243718 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a132bc7c-7b89-4076-aa20-a3337f3b7548-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-dcf9c3-29893350-88rn7\" (UID: \"a132bc7c-7b89-4076-aa20-a3337f3b7548\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-dcf9c3-29893350-88rn7" Nov 02 06:30:00.243883 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:00.243787 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhjxz\" (UniqueName: \"kubernetes.io/projected/2f971998-396b-47d6-a7fb-f5f7df0fc2e7-kube-api-access-lhjxz\") pod \"maas-api-key-cleanup-e2e-worker-w4-680b7b-29893350-5wc2c\" (UID: \"2f971998-396b-47d6-a7fb-f5f7df0fc2e7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-680b7b-29893350-5wc2c" Nov 02 06:30:00.243883 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:00.243817 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slfpt\" (UniqueName: \"kubernetes.io/projected/55759e02-f1b7-408c-ad1f-c132cd5f31a9-kube-api-access-slfpt\") pod \"maas-api-key-cleanup-e2e-worker-w1-cdfce9-29893350-v468m\" (UID: \"55759e02-f1b7-408c-ad1f-c132cd5f31a9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cdfce9-29893350-v468m" Nov 02 06:30:00.244016 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:00.243894 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/935b97d5-26cc-4548-bf0d-bfe7b12fda1f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-581608-29893350-bvdrk\" (UID: \"935b97d5-26cc-4548-bf0d-bfe7b12fda1f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-581608-29893350-bvdrk" Nov 02 06:30:00.244016 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:00.243936 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxx95\" (UniqueName: \"kubernetes.io/projected/935b97d5-26cc-4548-bf0d-bfe7b12fda1f-kube-api-access-zxx95\") pod \"maas-api-key-cleanup-e2e-worker-w2-581608-29893350-bvdrk\" (UID: \"935b97d5-26cc-4548-bf0d-bfe7b12fda1f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-581608-29893350-bvdrk" Nov 02 06:30:00.244016 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:00.243998 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztbz8\" (UniqueName: \"kubernetes.io/projected/a132bc7c-7b89-4076-aa20-a3337f3b7548-kube-api-access-ztbz8\") pod \"maas-api-key-cleanup-e2e-worker-w5-dcf9c3-29893350-88rn7\" (UID: \"a132bc7c-7b89-4076-aa20-a3337f3b7548\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-dcf9c3-29893350-88rn7" Nov 02 06:30:00.260393 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:00.260369 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-680b7b-29893350-5wc2c"] Nov 02 06:30:00.264357 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:00.264334 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cdfce9-29893350-v468m"] Nov 02 06:30:00.268119 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:00.268095 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-dcf9c3-29893350-88rn7"] Nov 02 06:30:00.309729 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:00.309340 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-581608-29893350-bvdrk"] Nov 02 06:30:00.344785 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:00.344750 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhjxz\" (UniqueName: \"kubernetes.io/projected/2f971998-396b-47d6-a7fb-f5f7df0fc2e7-kube-api-access-lhjxz\") pod \"maas-api-key-cleanup-e2e-worker-w4-680b7b-29893350-5wc2c\" (UID: \"2f971998-396b-47d6-a7fb-f5f7df0fc2e7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-680b7b-29893350-5wc2c" Nov 02 06:30:00.344986 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:00.344799 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slfpt\" (UniqueName: \"kubernetes.io/projected/55759e02-f1b7-408c-ad1f-c132cd5f31a9-kube-api-access-slfpt\") pod \"maas-api-key-cleanup-e2e-worker-w1-cdfce9-29893350-v468m\" (UID: \"55759e02-f1b7-408c-ad1f-c132cd5f31a9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cdfce9-29893350-v468m" Nov 02 06:30:00.344986 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:00.344875 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/935b97d5-26cc-4548-bf0d-bfe7b12fda1f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-581608-29893350-bvdrk\" (UID: \"935b97d5-26cc-4548-bf0d-bfe7b12fda1f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-581608-29893350-bvdrk" Nov 02 06:30:00.344986 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:00.344919 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxx95\" (UniqueName: \"kubernetes.io/projected/935b97d5-26cc-4548-bf0d-bfe7b12fda1f-kube-api-access-zxx95\") pod \"maas-api-key-cleanup-e2e-worker-w2-581608-29893350-bvdrk\" (UID: \"935b97d5-26cc-4548-bf0d-bfe7b12fda1f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-581608-29893350-bvdrk" Nov 02 06:30:00.344986 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:00.344983 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztbz8\" (UniqueName: \"kubernetes.io/projected/a132bc7c-7b89-4076-aa20-a3337f3b7548-kube-api-access-ztbz8\") pod \"maas-api-key-cleanup-e2e-worker-w5-dcf9c3-29893350-88rn7\" (UID: \"a132bc7c-7b89-4076-aa20-a3337f3b7548\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-dcf9c3-29893350-88rn7" Nov 02 06:30:00.345192 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:00.345021 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f971998-396b-47d6-a7fb-f5f7df0fc2e7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-680b7b-29893350-5wc2c\" (UID: \"2f971998-396b-47d6-a7fb-f5f7df0fc2e7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-680b7b-29893350-5wc2c" Nov 02 06:30:00.345192 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:00.345068 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55759e02-f1b7-408c-ad1f-c132cd5f31a9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-cdfce9-29893350-v468m\" (UID: \"55759e02-f1b7-408c-ad1f-c132cd5f31a9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cdfce9-29893350-v468m" Nov 02 06:30:00.345192 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:00.345110 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a132bc7c-7b89-4076-aa20-a3337f3b7548-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-dcf9c3-29893350-88rn7\" (UID: \"a132bc7c-7b89-4076-aa20-a3337f3b7548\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-dcf9c3-29893350-88rn7" Nov 02 06:30:00.345799 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:00.345773 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/935b97d5-26cc-4548-bf0d-bfe7b12fda1f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-581608-29893350-bvdrk\" (UID: \"935b97d5-26cc-4548-bf0d-bfe7b12fda1f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-581608-29893350-bvdrk" Nov 02 06:30:00.345891 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:00.345870 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55759e02-f1b7-408c-ad1f-c132cd5f31a9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-cdfce9-29893350-v468m\" (UID: \"55759e02-f1b7-408c-ad1f-c132cd5f31a9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cdfce9-29893350-v468m" Nov 02 06:30:00.345963 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:00.345870 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f971998-396b-47d6-a7fb-f5f7df0fc2e7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-680b7b-29893350-5wc2c\" (UID: \"2f971998-396b-47d6-a7fb-f5f7df0fc2e7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-680b7b-29893350-5wc2c" Nov 02 06:30:00.345963 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:00.345896 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a132bc7c-7b89-4076-aa20-a3337f3b7548-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-dcf9c3-29893350-88rn7\" (UID: \"a132bc7c-7b89-4076-aa20-a3337f3b7548\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-dcf9c3-29893350-88rn7" Nov 02 06:30:00.399474 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:00.399444 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slfpt\" (UniqueName: \"kubernetes.io/projected/55759e02-f1b7-408c-ad1f-c132cd5f31a9-kube-api-access-slfpt\") pod \"maas-api-key-cleanup-e2e-worker-w1-cdfce9-29893350-v468m\" (UID: \"55759e02-f1b7-408c-ad1f-c132cd5f31a9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cdfce9-29893350-v468m" Nov 02 06:30:00.399474 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:00.399463 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxx95\" (UniqueName: \"kubernetes.io/projected/935b97d5-26cc-4548-bf0d-bfe7b12fda1f-kube-api-access-zxx95\") pod \"maas-api-key-cleanup-e2e-worker-w2-581608-29893350-bvdrk\" (UID: \"935b97d5-26cc-4548-bf0d-bfe7b12fda1f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-581608-29893350-bvdrk" Nov 02 06:30:00.400041 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:00.400012 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztbz8\" (UniqueName: \"kubernetes.io/projected/a132bc7c-7b89-4076-aa20-a3337f3b7548-kube-api-access-ztbz8\") pod \"maas-api-key-cleanup-e2e-worker-w5-dcf9c3-29893350-88rn7\" (UID: \"a132bc7c-7b89-4076-aa20-a3337f3b7548\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-dcf9c3-29893350-88rn7" Nov 02 06:30:00.400308 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:00.400253 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhjxz\" (UniqueName: \"kubernetes.io/projected/2f971998-396b-47d6-a7fb-f5f7df0fc2e7-kube-api-access-lhjxz\") pod \"maas-api-key-cleanup-e2e-worker-w4-680b7b-29893350-5wc2c\" (UID: \"2f971998-396b-47d6-a7fb-f5f7df0fc2e7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-680b7b-29893350-5wc2c" Nov 02 06:30:00.481925 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:00.481825 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-680b7b-29893350-5wc2c" Nov 02 06:30:00.489330 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:00.489312 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-dcf9c3-29893350-88rn7" Nov 02 06:30:00.495334 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:00.495308 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cdfce9-29893350-v468m" Nov 02 06:30:00.523516 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:00.523283 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-581608-29893350-bvdrk" Nov 02 06:30:00.866579 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:00.866496 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-680b7b-29893350-5wc2c"] Nov 02 06:30:00.872244 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:30:00.872215 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f971998_396b_47d6_a7fb_f5f7df0fc2e7.slice/crio-0db561ef0078aa81160e6036463761c58505d97ad0a35a5f380fb858f08a29e6 WatchSource:0}: Error finding container 0db561ef0078aa81160e6036463761c58505d97ad0a35a5f380fb858f08a29e6: Status 404 returned error can't find the container with id 0db561ef0078aa81160e6036463761c58505d97ad0a35a5f380fb858f08a29e6 Nov 02 06:30:00.916021 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:00.915983 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-dcf9c3-29893350-88rn7"] Nov 02 06:30:00.918616 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:00.918592 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cdfce9-29893350-v468m"] Nov 02 06:30:00.922137 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:30:00.922110 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda132bc7c_7b89_4076_aa20_a3337f3b7548.slice/crio-60c858368d6272c6b1a6d6a3c1b1a4f51880e93eb206d822b41864ddd71f14c8 WatchSource:0}: Error finding container 60c858368d6272c6b1a6d6a3c1b1a4f51880e93eb206d822b41864ddd71f14c8: Status 404 returned error can't find the container with id 60c858368d6272c6b1a6d6a3c1b1a4f51880e93eb206d822b41864ddd71f14c8 Nov 02 06:30:00.968480 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:00.968436 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-581608-29893350-bvdrk"] Nov 02 06:30:00.974362 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:30:00.974316 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod935b97d5_26cc_4548_bf0d_bfe7b12fda1f.slice/crio-78d532726723c77db9c9577cbb18399ead66b9490f35dab3774096fbbb06d2ff WatchSource:0}: Error finding container 78d532726723c77db9c9577cbb18399ead66b9490f35dab3774096fbbb06d2ff: Status 404 returned error can't find the container with id 78d532726723c77db9c9577cbb18399ead66b9490f35dab3774096fbbb06d2ff Nov 02 06:30:01.793319 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:01.793280 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cdfce9-29893350-v468m" event={"ID":"55759e02-f1b7-408c-ad1f-c132cd5f31a9","Type":"ContainerStarted","Data":"154fab886de4869421043e1a8b9aca6e244db15a91c1032ef265825edc9d62e7"} Nov 02 06:30:01.794625 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:01.794593 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-dcf9c3-29893350-88rn7" event={"ID":"a132bc7c-7b89-4076-aa20-a3337f3b7548","Type":"ContainerStarted","Data":"60c858368d6272c6b1a6d6a3c1b1a4f51880e93eb206d822b41864ddd71f14c8"} Nov 02 06:30:01.795927 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:01.795889 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-680b7b-29893350-5wc2c" event={"ID":"2f971998-396b-47d6-a7fb-f5f7df0fc2e7","Type":"ContainerStarted","Data":"0db561ef0078aa81160e6036463761c58505d97ad0a35a5f380fb858f08a29e6"} Nov 02 06:30:01.797153 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:01.797133 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-581608-29893350-bvdrk" event={"ID":"935b97d5-26cc-4548-bf0d-bfe7b12fda1f","Type":"ContainerStarted","Data":"78d532726723c77db9c9577cbb18399ead66b9490f35dab3774096fbbb06d2ff"} Nov 02 06:30:02.458778 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:02.458721 5916 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-cdfce9/premium-w1-cdfce9-kserve-598cb9d57f-mrzrz" Nov 02 06:30:02.472856 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:02.472828 5916 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-cdfce9/premium-w1-cdfce9-kserve-598cb9d57f-mrzrz" Nov 02 06:30:02.819564 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:02.819474 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-cdfce9/premium-w1-cdfce9-kserve-598cb9d57f-mrzrz" Nov 02 06:30:08.855807 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:08.855780 5916 generic.go:356] "Generic (PLEG): container finished" podID="a132bc7c-7b89-4076-aa20-a3337f3b7548" containerID="64b4b366e30ebed51d8dc7605de7c8f13c88b61f2115acc8bfe3817263dc6355" exitCode=60 Nov 02 06:30:08.856236 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:08.855849 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-dcf9c3-29893350-88rn7" event={"ID":"a132bc7c-7b89-4076-aa20-a3337f3b7548","Type":"ContainerDied","Data":"64b4b366e30ebed51d8dc7605de7c8f13c88b61f2115acc8bfe3817263dc6355"} Nov 02 06:30:08.856236 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:08.856099 5916 scope.go:122] "RemoveContainer" containerID="64b4b366e30ebed51d8dc7605de7c8f13c88b61f2115acc8bfe3817263dc6355" Nov 02 06:30:08.857631 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:08.857602 5916 generic.go:356] "Generic (PLEG): container finished" podID="2f971998-396b-47d6-a7fb-f5f7df0fc2e7" containerID="0f30cbce9d7f58a001a87f634edd037bd0583db84bad35d676e3582e72011704" exitCode=60 Nov 02 06:30:08.857748 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:08.857724 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-680b7b-29893350-5wc2c" event={"ID":"2f971998-396b-47d6-a7fb-f5f7df0fc2e7","Type":"ContainerDied","Data":"0f30cbce9d7f58a001a87f634edd037bd0583db84bad35d676e3582e72011704"} Nov 02 06:30:08.857979 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:08.857894 5916 scope.go:122] "RemoveContainer" containerID="0f30cbce9d7f58a001a87f634edd037bd0583db84bad35d676e3582e72011704" Nov 02 06:30:08.859796 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:08.859764 5916 generic.go:356] "Generic (PLEG): container finished" podID="935b97d5-26cc-4548-bf0d-bfe7b12fda1f" containerID="60df8a80bc819bfaeb894d5041a0193f2748a0d68f6b658286d6c2e64e1993a6" exitCode=60 Nov 02 06:30:08.859930 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:08.859885 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-581608-29893350-bvdrk" event={"ID":"935b97d5-26cc-4548-bf0d-bfe7b12fda1f","Type":"ContainerDied","Data":"60df8a80bc819bfaeb894d5041a0193f2748a0d68f6b658286d6c2e64e1993a6"} Nov 02 06:30:08.860102 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:08.860086 5916 scope.go:122] "RemoveContainer" containerID="60df8a80bc819bfaeb894d5041a0193f2748a0d68f6b658286d6c2e64e1993a6" Nov 02 06:30:08.864418 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:08.864395 5916 generic.go:356] "Generic (PLEG): container finished" podID="55759e02-f1b7-408c-ad1f-c132cd5f31a9" containerID="6e4c22c2242c04d18766df46b244cf9808666c349179105b40ad727b2614c4dd" exitCode=60 Nov 02 06:30:08.864502 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:08.864431 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cdfce9-29893350-v468m" event={"ID":"55759e02-f1b7-408c-ad1f-c132cd5f31a9","Type":"ContainerDied","Data":"6e4c22c2242c04d18766df46b244cf9808666c349179105b40ad727b2614c4dd"} Nov 02 06:30:08.864650 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:08.864636 5916 scope.go:122] "RemoveContainer" containerID="6e4c22c2242c04d18766df46b244cf9808666c349179105b40ad727b2614c4dd" Nov 02 06:30:09.877654 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:09.877621 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-581608-29893350-bvdrk" event={"ID":"935b97d5-26cc-4548-bf0d-bfe7b12fda1f","Type":"ContainerStarted","Data":"6b66a9da15bc6cea634a0841678f6794517191faab165172f28e2d89bd75bcad"} Nov 02 06:30:09.879691 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:09.879630 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cdfce9-29893350-v468m" event={"ID":"55759e02-f1b7-408c-ad1f-c132cd5f31a9","Type":"ContainerStarted","Data":"2ce2d9701a77c2813c0fb232d7ff26a042728f6d7a335b98a4cb639d5aec4eaa"} Nov 02 06:30:09.881449 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:09.881421 5916 generic.go:356] "Generic (PLEG): container finished" podID="a132bc7c-7b89-4076-aa20-a3337f3b7548" containerID="a197ffa8be37d1c00e7add7dc84775ec0d0039cc3999918402165562a84547e0" exitCode=60 Nov 02 06:30:09.881531 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:09.881498 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-dcf9c3-29893350-88rn7" event={"ID":"a132bc7c-7b89-4076-aa20-a3337f3b7548","Type":"ContainerDied","Data":"a197ffa8be37d1c00e7add7dc84775ec0d0039cc3999918402165562a84547e0"} Nov 02 06:30:09.881600 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:09.881561 5916 scope.go:122] "RemoveContainer" containerID="64b4b366e30ebed51d8dc7605de7c8f13c88b61f2115acc8bfe3817263dc6355" Nov 02 06:30:09.881731 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:09.881714 5916 scope.go:122] "RemoveContainer" containerID="a197ffa8be37d1c00e7add7dc84775ec0d0039cc3999918402165562a84547e0" Nov 02 06:30:09.882047 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:30:09.882021 5916 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-dcf9c3-29893350-88rn7_odh-ai-gateway-infra(a132bc7c-7b89-4076-aa20-a3337f3b7548)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-dcf9c3-29893350-88rn7" podUID="a132bc7c-7b89-4076-aa20-a3337f3b7548" Nov 02 06:30:09.883437 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:09.883417 5916 generic.go:356] "Generic (PLEG): container finished" podID="2f971998-396b-47d6-a7fb-f5f7df0fc2e7" containerID="142473cd3d13ced81f0454dbcae654614fb6f06f732f282d3933256fee28dc7e" exitCode=60 Nov 02 06:30:09.883514 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:09.883468 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-680b7b-29893350-5wc2c" event={"ID":"2f971998-396b-47d6-a7fb-f5f7df0fc2e7","Type":"ContainerDied","Data":"142473cd3d13ced81f0454dbcae654614fb6f06f732f282d3933256fee28dc7e"} Nov 02 06:30:09.883724 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:09.883707 5916 scope.go:122] "RemoveContainer" containerID="142473cd3d13ced81f0454dbcae654614fb6f06f732f282d3933256fee28dc7e" Nov 02 06:30:09.884013 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:30:09.883985 5916 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-680b7b-29893350-5wc2c_odh-ai-gateway-infra(2f971998-396b-47d6-a7fb-f5f7df0fc2e7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-680b7b-29893350-5wc2c" podUID="2f971998-396b-47d6-a7fb-f5f7df0fc2e7" Nov 02 06:30:09.984759 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:09.984707 5916 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-581608-29893350-bvdrk" podStartSLOduration=3.330569489 podStartE2EDuration="9.984693038s" totalImagesPullingTime="6.654123549s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:30:09.983323403 +0000 UTC m=+798.255143797" watchObservedRunningTime="2026-11-02 06:30:09.984693038 +0000 UTC m=+798.256513432" Nov 02 06:30:10.000854 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:10.000828 5916 scope.go:122] "RemoveContainer" containerID="0f30cbce9d7f58a001a87f634edd037bd0583db84bad35d676e3582e72011704" Nov 02 06:30:10.299189 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:10.299136 5916 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cdfce9-29893350-v468m" podStartSLOduration=3.59369602 podStartE2EDuration="10.299121461s" totalImagesPullingTime="6.705425441s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:30:10.297785373 +0000 UTC m=+798.569605767" watchObservedRunningTime="2026-11-02 06:30:10.299121461 +0000 UTC m=+798.570941882" Nov 02 06:30:10.893565 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:10.893207 5916 generic.go:356] "Generic (PLEG): container finished" podID="935b97d5-26cc-4548-bf0d-bfe7b12fda1f" containerID="6b66a9da15bc6cea634a0841678f6794517191faab165172f28e2d89bd75bcad" exitCode=60 Nov 02 06:30:10.893565 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:10.893324 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-581608-29893350-bvdrk" event={"ID":"935b97d5-26cc-4548-bf0d-bfe7b12fda1f","Type":"ContainerDied","Data":"6b66a9da15bc6cea634a0841678f6794517191faab165172f28e2d89bd75bcad"} Nov 02 06:30:10.893565 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:10.893391 5916 scope.go:122] "RemoveContainer" containerID="60df8a80bc819bfaeb894d5041a0193f2748a0d68f6b658286d6c2e64e1993a6" Nov 02 06:30:10.894144 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:10.893828 5916 scope.go:122] "RemoveContainer" containerID="6b66a9da15bc6cea634a0841678f6794517191faab165172f28e2d89bd75bcad" Nov 02 06:30:10.894144 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:30:10.894119 5916 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-581608-29893350-bvdrk_odh-ai-gateway-infra(935b97d5-26cc-4548-bf0d-bfe7b12fda1f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-581608-29893350-bvdrk" podUID="935b97d5-26cc-4548-bf0d-bfe7b12fda1f" Nov 02 06:30:10.895746 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:10.895726 5916 generic.go:356] "Generic (PLEG): container finished" podID="55759e02-f1b7-408c-ad1f-c132cd5f31a9" containerID="2ce2d9701a77c2813c0fb232d7ff26a042728f6d7a335b98a4cb639d5aec4eaa" exitCode=60 Nov 02 06:30:10.895869 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:10.895807 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cdfce9-29893350-v468m" event={"ID":"55759e02-f1b7-408c-ad1f-c132cd5f31a9","Type":"ContainerDied","Data":"2ce2d9701a77c2813c0fb232d7ff26a042728f6d7a335b98a4cb639d5aec4eaa"} Nov 02 06:30:10.896012 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:10.895993 5916 scope.go:122] "RemoveContainer" containerID="2ce2d9701a77c2813c0fb232d7ff26a042728f6d7a335b98a4cb639d5aec4eaa" Nov 02 06:30:10.896255 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:30:10.896232 5916 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-cdfce9-29893350-v468m_odh-ai-gateway-infra(55759e02-f1b7-408c-ad1f-c132cd5f31a9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cdfce9-29893350-v468m" podUID="55759e02-f1b7-408c-ad1f-c132cd5f31a9" Nov 02 06:30:10.897625 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:10.897610 5916 scope.go:122] "RemoveContainer" containerID="a197ffa8be37d1c00e7add7dc84775ec0d0039cc3999918402165562a84547e0" Nov 02 06:30:10.897801 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:30:10.897779 5916 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-dcf9c3-29893350-88rn7_odh-ai-gateway-infra(a132bc7c-7b89-4076-aa20-a3337f3b7548)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-dcf9c3-29893350-88rn7" podUID="a132bc7c-7b89-4076-aa20-a3337f3b7548" Nov 02 06:30:10.899224 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:10.899205 5916 scope.go:122] "RemoveContainer" containerID="142473cd3d13ced81f0454dbcae654614fb6f06f732f282d3933256fee28dc7e" Nov 02 06:30:10.899365 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:30:10.899349 5916 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-680b7b-29893350-5wc2c_odh-ai-gateway-infra(2f971998-396b-47d6-a7fb-f5f7df0fc2e7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-680b7b-29893350-5wc2c" podUID="2f971998-396b-47d6-a7fb-f5f7df0fc2e7" Nov 02 06:30:10.907648 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:10.907628 5916 scope.go:122] "RemoveContainer" containerID="6e4c22c2242c04d18766df46b244cf9808666c349179105b40ad727b2614c4dd" Nov 02 06:30:11.906952 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:11.906891 5916 scope.go:122] "RemoveContainer" containerID="6b66a9da15bc6cea634a0841678f6794517191faab165172f28e2d89bd75bcad" Nov 02 06:30:11.907395 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:30:11.907137 5916 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-581608-29893350-bvdrk_odh-ai-gateway-infra(935b97d5-26cc-4548-bf0d-bfe7b12fda1f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-581608-29893350-bvdrk" podUID="935b97d5-26cc-4548-bf0d-bfe7b12fda1f" Nov 02 06:30:11.908433 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:11.908414 5916 scope.go:122] "RemoveContainer" containerID="2ce2d9701a77c2813c0fb232d7ff26a042728f6d7a335b98a4cb639d5aec4eaa" Nov 02 06:30:11.908585 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:30:11.908567 5916 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-cdfce9-29893350-v468m_odh-ai-gateway-infra(55759e02-f1b7-408c-ad1f-c132cd5f31a9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cdfce9-29893350-v468m" podUID="55759e02-f1b7-408c-ad1f-c132cd5f31a9" Nov 02 06:30:19.893847 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:30:19.893812 5916 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p98cq" containerName="extractor" Nov 02 06:30:20.835332 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:20.835298 5916 scope.go:122] "RemoveContainer" containerID="142473cd3d13ced81f0454dbcae654614fb6f06f732f282d3933256fee28dc7e" Nov 02 06:30:21.850638 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:21.850598 5916 scope.go:122] "RemoveContainer" containerID="6b66a9da15bc6cea634a0841678f6794517191faab165172f28e2d89bd75bcad" Nov 02 06:30:21.976682 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:21.976647 5916 generic.go:356] "Generic (PLEG): container finished" podID="2f971998-396b-47d6-a7fb-f5f7df0fc2e7" containerID="da8e750ff9fb7b27ade2b7c99973d8a739e277a2c2628619773484af95d8fdd7" exitCode=60 Nov 02 06:30:21.976828 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:21.976729 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-680b7b-29893350-5wc2c" event={"ID":"2f971998-396b-47d6-a7fb-f5f7df0fc2e7","Type":"ContainerDied","Data":"da8e750ff9fb7b27ade2b7c99973d8a739e277a2c2628619773484af95d8fdd7"} Nov 02 06:30:21.976828 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:21.976808 5916 scope.go:122] "RemoveContainer" containerID="142473cd3d13ced81f0454dbcae654614fb6f06f732f282d3933256fee28dc7e" Nov 02 06:30:21.977258 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:21.977240 5916 scope.go:122] "RemoveContainer" containerID="da8e750ff9fb7b27ade2b7c99973d8a739e277a2c2628619773484af95d8fdd7" Nov 02 06:30:21.977546 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:30:21.977516 5916 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-680b7b-29893350-5wc2c_odh-ai-gateway-infra(2f971998-396b-47d6-a7fb-f5f7df0fc2e7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-680b7b-29893350-5wc2c" podUID="2f971998-396b-47d6-a7fb-f5f7df0fc2e7" Nov 02 06:30:22.835655 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:22.835626 5916 scope.go:122] "RemoveContainer" containerID="2ce2d9701a77c2813c0fb232d7ff26a042728f6d7a335b98a4cb639d5aec4eaa" Nov 02 06:30:22.991518 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:22.991486 5916 generic.go:356] "Generic (PLEG): container finished" podID="935b97d5-26cc-4548-bf0d-bfe7b12fda1f" containerID="b1cb7bf919810d0e6f629a8ddcfcd62c4f9ae415b81886832f8e01072f8e29b9" exitCode=60 Nov 02 06:30:22.992017 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:22.991557 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-581608-29893350-bvdrk" event={"ID":"935b97d5-26cc-4548-bf0d-bfe7b12fda1f","Type":"ContainerDied","Data":"b1cb7bf919810d0e6f629a8ddcfcd62c4f9ae415b81886832f8e01072f8e29b9"} Nov 02 06:30:22.992017 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:22.991617 5916 scope.go:122] "RemoveContainer" containerID="6b66a9da15bc6cea634a0841678f6794517191faab165172f28e2d89bd75bcad" Nov 02 06:30:22.992017 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:22.991978 5916 scope.go:122] "RemoveContainer" containerID="b1cb7bf919810d0e6f629a8ddcfcd62c4f9ae415b81886832f8e01072f8e29b9" Nov 02 06:30:22.992223 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:30:22.992203 5916 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-581608-29893350-bvdrk_odh-ai-gateway-infra(935b97d5-26cc-4548-bf0d-bfe7b12fda1f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-581608-29893350-bvdrk" podUID="935b97d5-26cc-4548-bf0d-bfe7b12fda1f" Nov 02 06:30:23.095461 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:23.095436 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-680b7b-29893350-5wc2c"] Nov 02 06:30:23.246966 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:23.245802 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-680b7b-29893350-5wc2c" Nov 02 06:30:23.397245 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:23.396732 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f971998-396b-47d6-a7fb-f5f7df0fc2e7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f971998-396b-47d6-a7fb-f5f7df0fc2e7-maas-api-service-ca\") pod \"2f971998-396b-47d6-a7fb-f5f7df0fc2e7\" (UID: \"2f971998-396b-47d6-a7fb-f5f7df0fc2e7\") " Nov 02 06:30:23.397245 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:23.396818 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f971998-396b-47d6-a7fb-f5f7df0fc2e7-kube-api-access-lhjxz\" (UniqueName: \"kubernetes.io/projected/2f971998-396b-47d6-a7fb-f5f7df0fc2e7-kube-api-access-lhjxz\") pod \"2f971998-396b-47d6-a7fb-f5f7df0fc2e7\" (UID: \"2f971998-396b-47d6-a7fb-f5f7df0fc2e7\") " Nov 02 06:30:23.397245 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:23.397088 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f971998-396b-47d6-a7fb-f5f7df0fc2e7-maas-api-service-ca" pod "2f971998-396b-47d6-a7fb-f5f7df0fc2e7" (UID: "2f971998-396b-47d6-a7fb-f5f7df0fc2e7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:30:23.397245 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:23.397203 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f971998-396b-47d6-a7fb-f5f7df0fc2e7-maas-api-service-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:30:23.399111 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:23.399090 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f971998-396b-47d6-a7fb-f5f7df0fc2e7-kube-api-access-lhjxz" pod "2f971998-396b-47d6-a7fb-f5f7df0fc2e7" (UID: "2f971998-396b-47d6-a7fb-f5f7df0fc2e7"). InnerVolumeSpecName "kube-api-access-lhjxz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:30:23.498752 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:23.498669 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhjxz\" (UniqueName: \"kubernetes.io/projected/2f971998-396b-47d6-a7fb-f5f7df0fc2e7-kube-api-access-lhjxz\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:30:23.835228 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:23.835148 5916 scope.go:122] "RemoveContainer" containerID="a197ffa8be37d1c00e7add7dc84775ec0d0039cc3999918402165562a84547e0" Nov 02 06:30:24.005131 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:24.005097 5916 generic.go:356] "Generic (PLEG): container finished" podID="55759e02-f1b7-408c-ad1f-c132cd5f31a9" containerID="dda23a2cbd4ac92aa686b3c6973fffa43b83fd1a0e47289b2ed22cc4a9e79a5c" exitCode=60 Nov 02 06:30:24.005586 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:24.005165 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cdfce9-29893350-v468m" event={"ID":"55759e02-f1b7-408c-ad1f-c132cd5f31a9","Type":"ContainerDied","Data":"dda23a2cbd4ac92aa686b3c6973fffa43b83fd1a0e47289b2ed22cc4a9e79a5c"} Nov 02 06:30:24.005586 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:24.005219 5916 scope.go:122] "RemoveContainer" containerID="2ce2d9701a77c2813c0fb232d7ff26a042728f6d7a335b98a4cb639d5aec4eaa" Nov 02 06:30:24.005713 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:24.005599 5916 scope.go:122] "RemoveContainer" containerID="dda23a2cbd4ac92aa686b3c6973fffa43b83fd1a0e47289b2ed22cc4a9e79a5c" Nov 02 06:30:24.005874 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:30:24.005851 5916 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-cdfce9-29893350-v468m_odh-ai-gateway-infra(55759e02-f1b7-408c-ad1f-c132cd5f31a9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cdfce9-29893350-v468m" podUID="55759e02-f1b7-408c-ad1f-c132cd5f31a9" Nov 02 06:30:24.006562 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:24.006545 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-680b7b-29893350-5wc2c" Nov 02 06:30:24.006613 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:24.006566 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-680b7b-29893350-5wc2c" event={"ID":"2f971998-396b-47d6-a7fb-f5f7df0fc2e7","Type":"ContainerDied","Data":"0db561ef0078aa81160e6036463761c58505d97ad0a35a5f380fb858f08a29e6"} Nov 02 06:30:24.018381 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:24.018362 5916 scope.go:122] "RemoveContainer" containerID="da8e750ff9fb7b27ade2b7c99973d8a739e277a2c2628619773484af95d8fdd7" Nov 02 06:30:24.117170 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:24.117090 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-581608-29893350-bvdrk"] Nov 02 06:30:24.221194 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:24.221113 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-680b7b-29893350-5wc2c"] Nov 02 06:30:24.230026 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:24.229089 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-680b7b-29893350-5wc2c"] Nov 02 06:30:24.277632 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:24.277607 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-581608-29893350-bvdrk" Nov 02 06:30:24.409745 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:24.409710 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/935b97d5-26cc-4548-bf0d-bfe7b12fda1f-kube-api-access-zxx95\" (UniqueName: \"kubernetes.io/projected/935b97d5-26cc-4548-bf0d-bfe7b12fda1f-kube-api-access-zxx95\") pod \"935b97d5-26cc-4548-bf0d-bfe7b12fda1f\" (UID: \"935b97d5-26cc-4548-bf0d-bfe7b12fda1f\") " Nov 02 06:30:24.409953 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:24.409870 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/935b97d5-26cc-4548-bf0d-bfe7b12fda1f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/935b97d5-26cc-4548-bf0d-bfe7b12fda1f-maas-api-service-ca\") pod \"935b97d5-26cc-4548-bf0d-bfe7b12fda1f\" (UID: \"935b97d5-26cc-4548-bf0d-bfe7b12fda1f\") " Nov 02 06:30:24.410193 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:24.410172 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/935b97d5-26cc-4548-bf0d-bfe7b12fda1f-maas-api-service-ca" pod "935b97d5-26cc-4548-bf0d-bfe7b12fda1f" (UID: "935b97d5-26cc-4548-bf0d-bfe7b12fda1f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:30:24.411787 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:24.411758 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/935b97d5-26cc-4548-bf0d-bfe7b12fda1f-kube-api-access-zxx95" pod "935b97d5-26cc-4548-bf0d-bfe7b12fda1f" (UID: "935b97d5-26cc-4548-bf0d-bfe7b12fda1f"). InnerVolumeSpecName "kube-api-access-zxx95". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:30:24.510653 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:24.510625 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/935b97d5-26cc-4548-bf0d-bfe7b12fda1f-maas-api-service-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:30:24.510653 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:24.510648 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxx95\" (UniqueName: \"kubernetes.io/projected/935b97d5-26cc-4548-bf0d-bfe7b12fda1f-kube-api-access-zxx95\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:30:25.014824 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:25.014743 5916 generic.go:356] "Generic (PLEG): container finished" podID="a132bc7c-7b89-4076-aa20-a3337f3b7548" containerID="69e1c1d178e9896a82d36f716110314cf7784f48d7819255050449e131422cf4" exitCode=60 Nov 02 06:30:25.014824 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:25.014815 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-dcf9c3-29893350-88rn7" event={"ID":"a132bc7c-7b89-4076-aa20-a3337f3b7548","Type":"ContainerDied","Data":"69e1c1d178e9896a82d36f716110314cf7784f48d7819255050449e131422cf4"} Nov 02 06:30:25.015322 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:25.014869 5916 scope.go:122] "RemoveContainer" containerID="a197ffa8be37d1c00e7add7dc84775ec0d0039cc3999918402165562a84547e0" Nov 02 06:30:25.015322 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:25.015216 5916 scope.go:122] "RemoveContainer" containerID="69e1c1d178e9896a82d36f716110314cf7784f48d7819255050449e131422cf4" Nov 02 06:30:25.015483 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:30:25.015460 5916 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-dcf9c3-29893350-88rn7_odh-ai-gateway-infra(a132bc7c-7b89-4076-aa20-a3337f3b7548)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-dcf9c3-29893350-88rn7" podUID="a132bc7c-7b89-4076-aa20-a3337f3b7548" Nov 02 06:30:25.017876 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:25.017857 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-581608-29893350-bvdrk" Nov 02 06:30:25.017979 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:25.017889 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-581608-29893350-bvdrk" event={"ID":"935b97d5-26cc-4548-bf0d-bfe7b12fda1f","Type":"ContainerDied","Data":"78d532726723c77db9c9577cbb18399ead66b9490f35dab3774096fbbb06d2ff"} Nov 02 06:30:25.027860 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:25.027841 5916 scope.go:122] "RemoveContainer" containerID="b1cb7bf919810d0e6f629a8ddcfcd62c4f9ae415b81886832f8e01072f8e29b9" Nov 02 06:30:25.122414 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:25.122386 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cdfce9-29893350-v468m"] Nov 02 06:30:25.227655 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:25.227627 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-581608-29893350-bvdrk"] Nov 02 06:30:25.233127 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:25.233101 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-581608-29893350-bvdrk"] Nov 02 06:30:25.266121 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:25.266066 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cdfce9-29893350-v468m" Nov 02 06:30:25.419580 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:25.419551 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55759e02-f1b7-408c-ad1f-c132cd5f31a9-kube-api-access-slfpt\" (UniqueName: \"kubernetes.io/projected/55759e02-f1b7-408c-ad1f-c132cd5f31a9-kube-api-access-slfpt\") pod \"55759e02-f1b7-408c-ad1f-c132cd5f31a9\" (UID: \"55759e02-f1b7-408c-ad1f-c132cd5f31a9\") " Nov 02 06:30:25.419727 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:25.419606 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55759e02-f1b7-408c-ad1f-c132cd5f31a9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55759e02-f1b7-408c-ad1f-c132cd5f31a9-maas-api-service-ca\") pod \"55759e02-f1b7-408c-ad1f-c132cd5f31a9\" (UID: \"55759e02-f1b7-408c-ad1f-c132cd5f31a9\") " Nov 02 06:30:25.420016 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:25.419990 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55759e02-f1b7-408c-ad1f-c132cd5f31a9-maas-api-service-ca" pod "55759e02-f1b7-408c-ad1f-c132cd5f31a9" (UID: "55759e02-f1b7-408c-ad1f-c132cd5f31a9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:30:25.421679 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:25.421655 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55759e02-f1b7-408c-ad1f-c132cd5f31a9-kube-api-access-slfpt" pod "55759e02-f1b7-408c-ad1f-c132cd5f31a9" (UID: "55759e02-f1b7-408c-ad1f-c132cd5f31a9"). InnerVolumeSpecName "kube-api-access-slfpt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:30:25.521030 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:25.520963 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slfpt\" (UniqueName: \"kubernetes.io/projected/55759e02-f1b7-408c-ad1f-c132cd5f31a9-kube-api-access-slfpt\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:30:25.521030 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:25.520986 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55759e02-f1b7-408c-ad1f-c132cd5f31a9-maas-api-service-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:30:25.845427 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:25.845338 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f971998-396b-47d6-a7fb-f5f7df0fc2e7" path="/var/lib/kubelet/pods/2f971998-396b-47d6-a7fb-f5f7df0fc2e7/volumes" Nov 02 06:30:25.845803 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:25.845786 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="935b97d5-26cc-4548-bf0d-bfe7b12fda1f" path="/var/lib/kubelet/pods/935b97d5-26cc-4548-bf0d-bfe7b12fda1f/volumes" Nov 02 06:30:26.027977 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:26.027945 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cdfce9-29893350-v468m" Nov 02 06:30:26.027977 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:26.027964 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cdfce9-29893350-v468m" event={"ID":"55759e02-f1b7-408c-ad1f-c132cd5f31a9","Type":"ContainerDied","Data":"154fab886de4869421043e1a8b9aca6e244db15a91c1032ef265825edc9d62e7"} Nov 02 06:30:26.028488 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:26.028026 5916 scope.go:122] "RemoveContainer" containerID="dda23a2cbd4ac92aa686b3c6973fffa43b83fd1a0e47289b2ed22cc4a9e79a5c" Nov 02 06:30:26.135851 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:26.135822 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-dcf9c3-29893350-88rn7"] Nov 02 06:30:26.191559 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:26.191361 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cdfce9-29893350-v468m"] Nov 02 06:30:26.206740 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:26.206709 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cdfce9-29893350-v468m"] Nov 02 06:30:26.289007 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:26.288981 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-dcf9c3-29893350-88rn7" Nov 02 06:30:26.430466 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:26.430376 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a132bc7c-7b89-4076-aa20-a3337f3b7548-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a132bc7c-7b89-4076-aa20-a3337f3b7548-maas-api-service-ca\") pod \"a132bc7c-7b89-4076-aa20-a3337f3b7548\" (UID: \"a132bc7c-7b89-4076-aa20-a3337f3b7548\") " Nov 02 06:30:26.430818 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:26.430736 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a132bc7c-7b89-4076-aa20-a3337f3b7548-kube-api-access-ztbz8\" (UniqueName: \"kubernetes.io/projected/a132bc7c-7b89-4076-aa20-a3337f3b7548-kube-api-access-ztbz8\") pod \"a132bc7c-7b89-4076-aa20-a3337f3b7548\" (UID: \"a132bc7c-7b89-4076-aa20-a3337f3b7548\") " Nov 02 06:30:26.431080 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:26.430767 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a132bc7c-7b89-4076-aa20-a3337f3b7548-maas-api-service-ca" pod "a132bc7c-7b89-4076-aa20-a3337f3b7548" (UID: "a132bc7c-7b89-4076-aa20-a3337f3b7548"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:30:26.431419 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:26.431395 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a132bc7c-7b89-4076-aa20-a3337f3b7548-maas-api-service-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:30:26.433339 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:26.433310 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a132bc7c-7b89-4076-aa20-a3337f3b7548-kube-api-access-ztbz8" pod "a132bc7c-7b89-4076-aa20-a3337f3b7548" (UID: "a132bc7c-7b89-4076-aa20-a3337f3b7548"). InnerVolumeSpecName "kube-api-access-ztbz8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:30:26.532830 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:26.532791 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztbz8\" (UniqueName: \"kubernetes.io/projected/a132bc7c-7b89-4076-aa20-a3337f3b7548-kube-api-access-ztbz8\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:30:27.043534 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:27.043504 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-dcf9c3-29893350-88rn7" Nov 02 06:30:27.043534 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:27.043528 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-dcf9c3-29893350-88rn7" event={"ID":"a132bc7c-7b89-4076-aa20-a3337f3b7548","Type":"ContainerDied","Data":"60c858368d6272c6b1a6d6a3c1b1a4f51880e93eb206d822b41864ddd71f14c8"} Nov 02 06:30:27.044093 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:27.043604 5916 scope.go:122] "RemoveContainer" containerID="69e1c1d178e9896a82d36f716110314cf7784f48d7819255050449e131422cf4" Nov 02 06:30:27.199082 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:27.199049 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-dcf9c3-29893350-88rn7"] Nov 02 06:30:27.201857 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:27.201831 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-dcf9c3-29893350-88rn7"] Nov 02 06:30:27.844604 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:27.844561 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55759e02-f1b7-408c-ad1f-c132cd5f31a9" path="/var/lib/kubelet/pods/55759e02-f1b7-408c-ad1f-c132cd5f31a9/volumes" Nov 02 06:30:27.844997 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:27.844981 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a132bc7c-7b89-4076-aa20-a3337f3b7548" path="/var/lib/kubelet/pods/a132bc7c-7b89-4076-aa20-a3337f3b7548/volumes" Nov 02 06:30:38.816948 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:38.816860 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-640421b6-z5df5"] Nov 02 06:30:38.817413 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:38.817321 5916 cpu_manager.go:414] "containerMap: removing container" podUID="935b97d5-26cc-4548-bf0d-bfe7b12fda1f" containerName="cleanup" Nov 02 06:30:38.817413 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:38.817332 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="935b97d5-26cc-4548-bf0d-bfe7b12fda1f" containerName="cleanup" Nov 02 06:30:38.817413 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:38.817341 5916 cpu_manager.go:414] "containerMap: removing container" podUID="935b97d5-26cc-4548-bf0d-bfe7b12fda1f" containerName="cleanup" Nov 02 06:30:38.817413 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:38.817346 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="935b97d5-26cc-4548-bf0d-bfe7b12fda1f" containerName="cleanup" Nov 02 06:30:38.817413 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:38.817358 5916 cpu_manager.go:414] "containerMap: removing container" podUID="2f971998-396b-47d6-a7fb-f5f7df0fc2e7" containerName="cleanup" Nov 02 06:30:38.817413 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:38.817364 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f971998-396b-47d6-a7fb-f5f7df0fc2e7" containerName="cleanup" Nov 02 06:30:38.817413 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:38.817378 5916 cpu_manager.go:414] "containerMap: removing container" podUID="2f971998-396b-47d6-a7fb-f5f7df0fc2e7" containerName="cleanup" Nov 02 06:30:38.817413 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:38.817383 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f971998-396b-47d6-a7fb-f5f7df0fc2e7" containerName="cleanup" Nov 02 06:30:38.817413 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:38.817389 5916 cpu_manager.go:414] "containerMap: removing container" podUID="55759e02-f1b7-408c-ad1f-c132cd5f31a9" containerName="cleanup" Nov 02 06:30:38.817413 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:38.817394 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55759e02-f1b7-408c-ad1f-c132cd5f31a9" containerName="cleanup" Nov 02 06:30:38.817413 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:38.817401 5916 cpu_manager.go:414] "containerMap: removing container" podUID="a132bc7c-7b89-4076-aa20-a3337f3b7548" containerName="cleanup" Nov 02 06:30:38.817413 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:38.817405 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a132bc7c-7b89-4076-aa20-a3337f3b7548" containerName="cleanup" Nov 02 06:30:38.817413 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:38.817412 5916 cpu_manager.go:414] "containerMap: removing container" podUID="55759e02-f1b7-408c-ad1f-c132cd5f31a9" containerName="cleanup" Nov 02 06:30:38.817413 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:38.817417 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55759e02-f1b7-408c-ad1f-c132cd5f31a9" containerName="cleanup" Nov 02 06:30:38.818045 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:38.817450 5916 cpu_manager.go:414] "containerMap: removing container" podUID="55759e02-f1b7-408c-ad1f-c132cd5f31a9" containerName="cleanup" Nov 02 06:30:38.818045 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:38.817457 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55759e02-f1b7-408c-ad1f-c132cd5f31a9" containerName="cleanup" Nov 02 06:30:38.818045 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:38.817462 5916 cpu_manager.go:414] "containerMap: removing container" podUID="a132bc7c-7b89-4076-aa20-a3337f3b7548" containerName="cleanup" Nov 02 06:30:38.818045 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:38.817466 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a132bc7c-7b89-4076-aa20-a3337f3b7548" containerName="cleanup" Nov 02 06:30:38.818045 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:38.817551 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="55759e02-f1b7-408c-ad1f-c132cd5f31a9" containerName="cleanup" Nov 02 06:30:38.818045 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:38.817559 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="a132bc7c-7b89-4076-aa20-a3337f3b7548" containerName="cleanup" Nov 02 06:30:38.818045 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:38.817564 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="a132bc7c-7b89-4076-aa20-a3337f3b7548" containerName="cleanup" Nov 02 06:30:38.818045 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:38.817569 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="935b97d5-26cc-4548-bf0d-bfe7b12fda1f" containerName="cleanup" Nov 02 06:30:38.818045 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:38.817575 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f971998-396b-47d6-a7fb-f5f7df0fc2e7" containerName="cleanup" Nov 02 06:30:38.818045 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:38.817580 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="935b97d5-26cc-4548-bf0d-bfe7b12fda1f" containerName="cleanup" Nov 02 06:30:38.818045 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:38.817584 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="935b97d5-26cc-4548-bf0d-bfe7b12fda1f" containerName="cleanup" Nov 02 06:30:38.818045 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:38.817590 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f971998-396b-47d6-a7fb-f5f7df0fc2e7" containerName="cleanup" Nov 02 06:30:38.818045 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:38.817596 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f971998-396b-47d6-a7fb-f5f7df0fc2e7" containerName="cleanup" Nov 02 06:30:38.818045 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:38.817605 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="a132bc7c-7b89-4076-aa20-a3337f3b7548" containerName="cleanup" Nov 02 06:30:38.818045 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:38.817622 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="55759e02-f1b7-408c-ad1f-c132cd5f31a9" containerName="cleanup" Nov 02 06:30:38.820894 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:38.820878 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-640421b6-z5df5" Nov 02 06:30:38.858576 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:38.858547 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e6672c5-8cec-458c-916c-74f5e417d387-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-640421b6-z5df5\" (UID: \"9e6672c5-8cec-458c-916c-74f5e417d387\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-640421b6-z5df5" Nov 02 06:30:38.858576 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:38.858576 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e6672c5-8cec-458c-916c-74f5e417d387-maas-api-xks-ca\") pod \"maas-api-revoke-sub-640421b6-z5df5\" (UID: \"9e6672c5-8cec-458c-916c-74f5e417d387\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-640421b6-z5df5" Nov 02 06:30:38.858767 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:38.858743 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7nqm\" (UniqueName: \"kubernetes.io/projected/9e6672c5-8cec-458c-916c-74f5e417d387-kube-api-access-d7nqm\") pod \"maas-api-revoke-sub-640421b6-z5df5\" (UID: \"9e6672c5-8cec-458c-916c-74f5e417d387\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-640421b6-z5df5" Nov 02 06:30:38.872948 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:38.872925 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-640421b6-z5df5"] Nov 02 06:30:38.915693 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:38.915654 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 06:30:38.915863 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:38.915777 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-68l4r\"" Nov 02 06:30:38.959551 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:38.959517 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e6672c5-8cec-458c-916c-74f5e417d387-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-640421b6-z5df5\" (UID: \"9e6672c5-8cec-458c-916c-74f5e417d387\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-640421b6-z5df5" Nov 02 06:30:38.959732 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:38.959555 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e6672c5-8cec-458c-916c-74f5e417d387-maas-api-xks-ca\") pod \"maas-api-revoke-sub-640421b6-z5df5\" (UID: \"9e6672c5-8cec-458c-916c-74f5e417d387\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-640421b6-z5df5" Nov 02 06:30:38.959732 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:38.959625 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7nqm\" (UniqueName: \"kubernetes.io/projected/9e6672c5-8cec-458c-916c-74f5e417d387-kube-api-access-d7nqm\") pod \"maas-api-revoke-sub-640421b6-z5df5\" (UID: \"9e6672c5-8cec-458c-916c-74f5e417d387\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-640421b6-z5df5" Nov 02 06:30:38.960266 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:38.960243 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e6672c5-8cec-458c-916c-74f5e417d387-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-640421b6-z5df5\" (UID: \"9e6672c5-8cec-458c-916c-74f5e417d387\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-640421b6-z5df5" Nov 02 06:30:38.961822 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:38.961799 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e6672c5-8cec-458c-916c-74f5e417d387-maas-api-xks-ca\") pod \"maas-api-revoke-sub-640421b6-z5df5\" (UID: \"9e6672c5-8cec-458c-916c-74f5e417d387\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-640421b6-z5df5" Nov 02 06:30:39.012875 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:39.012849 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7nqm\" (UniqueName: \"kubernetes.io/projected/9e6672c5-8cec-458c-916c-74f5e417d387-kube-api-access-d7nqm\") pod \"maas-api-revoke-sub-640421b6-z5df5\" (UID: \"9e6672c5-8cec-458c-916c-74f5e417d387\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-640421b6-z5df5" Nov 02 06:30:39.134169 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:39.134137 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-640421b6-z5df5" Nov 02 06:30:39.510737 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:39.510655 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-640421b6-z5df5"] Nov 02 06:30:40.163137 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:40.163099 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-640421b6-z5df5" event={"ID":"9e6672c5-8cec-458c-916c-74f5e417d387","Type":"ContainerStarted","Data":"f38cd233fe0b65e996dd98e61656367d601fcf6bb59af4497c1451744f2ba32f"} Nov 02 06:30:40.163546 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:40.163187 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-640421b6-z5df5" event={"ID":"9e6672c5-8cec-458c-916c-74f5e417d387","Type":"ContainerStarted","Data":"b08178f55abb08191819444a9ef07617176f7b71ba63486e258b50c4977b0a6b"} Nov 02 06:30:40.270302 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:40.270250 5916 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-640421b6-z5df5" podStartSLOduration=2.270237109 podStartE2EDuration="2.270237109s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:30:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:30:40.268411817 +0000 UTC m=+828.540232210" watchObservedRunningTime="2026-11-02 06:30:40.270237109 +0000 UTC m=+828.542057502" Nov 02 06:30:41.171777 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:41.171735 5916 generic.go:356] "Generic (PLEG): container finished" podID="9e6672c5-8cec-458c-916c-74f5e417d387" containerID="f38cd233fe0b65e996dd98e61656367d601fcf6bb59af4497c1451744f2ba32f" exitCode=0 Nov 02 06:30:41.172242 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:41.171812 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-640421b6-z5df5" event={"ID":"9e6672c5-8cec-458c-916c-74f5e417d387","Type":"ContainerDied","Data":"f38cd233fe0b65e996dd98e61656367d601fcf6bb59af4497c1451744f2ba32f"} Nov 02 06:30:42.330251 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:42.330223 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-640421b6-z5df5" Nov 02 06:30:42.391612 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:42.391582 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e6672c5-8cec-458c-916c-74f5e417d387-kube-api-access-d7nqm\" (UniqueName: \"kubernetes.io/projected/9e6672c5-8cec-458c-916c-74f5e417d387-kube-api-access-d7nqm\") pod \"9e6672c5-8cec-458c-916c-74f5e417d387\" (UID: \"9e6672c5-8cec-458c-916c-74f5e417d387\") " Nov 02 06:30:42.391802 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:42.391658 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e6672c5-8cec-458c-916c-74f5e417d387-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e6672c5-8cec-458c-916c-74f5e417d387-maas-api-xks-ca\") pod \"9e6672c5-8cec-458c-916c-74f5e417d387\" (UID: \"9e6672c5-8cec-458c-916c-74f5e417d387\") " Nov 02 06:30:42.391802 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:42.391745 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e6672c5-8cec-458c-916c-74f5e417d387-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e6672c5-8cec-458c-916c-74f5e417d387-maas-api-openshift-ca\") pod \"9e6672c5-8cec-458c-916c-74f5e417d387\" (UID: \"9e6672c5-8cec-458c-916c-74f5e417d387\") " Nov 02 06:30:42.392120 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:42.392096 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e6672c5-8cec-458c-916c-74f5e417d387-maas-api-openshift-ca" pod "9e6672c5-8cec-458c-916c-74f5e417d387" (UID: "9e6672c5-8cec-458c-916c-74f5e417d387"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:30:42.393732 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:42.393708 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e6672c5-8cec-458c-916c-74f5e417d387-maas-api-xks-ca" pod "9e6672c5-8cec-458c-916c-74f5e417d387" (UID: "9e6672c5-8cec-458c-916c-74f5e417d387"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:30:42.393815 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:42.393731 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e6672c5-8cec-458c-916c-74f5e417d387-kube-api-access-d7nqm" pod "9e6672c5-8cec-458c-916c-74f5e417d387" (UID: "9e6672c5-8cec-458c-916c-74f5e417d387"). InnerVolumeSpecName "kube-api-access-d7nqm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:30:42.492732 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:42.492652 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e6672c5-8cec-458c-916c-74f5e417d387-maas-api-openshift-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:30:42.492732 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:42.492681 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7nqm\" (UniqueName: \"kubernetes.io/projected/9e6672c5-8cec-458c-916c-74f5e417d387-kube-api-access-d7nqm\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:30:42.492732 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:42.492693 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e6672c5-8cec-458c-916c-74f5e417d387-maas-api-xks-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:30:43.185594 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:43.185568 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-640421b6-z5df5" Nov 02 06:30:43.185776 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:43.185607 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-640421b6-z5df5" event={"ID":"9e6672c5-8cec-458c-916c-74f5e417d387","Type":"ContainerDied","Data":"b08178f55abb08191819444a9ef07617176f7b71ba63486e258b50c4977b0a6b"} Nov 02 06:30:43.185776 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:43.185655 5916 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b08178f55abb08191819444a9ef07617176f7b71ba63486e258b50c4977b0a6b" Nov 02 06:30:49.894257 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:30:49.894213 5916 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p98cq" containerName="extractor" Nov 02 06:30:54.105565 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:54.105532 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-094ee0f4-9g8wx"] Nov 02 06:30:54.106031 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:54.106018 5916 cpu_manager.go:414] "containerMap: removing container" podUID="935b97d5-26cc-4548-bf0d-bfe7b12fda1f" containerName="cleanup" Nov 02 06:30:54.106031 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:54.106032 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="935b97d5-26cc-4548-bf0d-bfe7b12fda1f" containerName="cleanup" Nov 02 06:30:54.106119 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:54.106054 5916 cpu_manager.go:414] "containerMap: removing container" podUID="2f971998-396b-47d6-a7fb-f5f7df0fc2e7" containerName="cleanup" Nov 02 06:30:54.106119 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:54.106060 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f971998-396b-47d6-a7fb-f5f7df0fc2e7" containerName="cleanup" Nov 02 06:30:54.106119 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:54.106074 5916 cpu_manager.go:414] "containerMap: removing container" podUID="9e6672c5-8cec-458c-916c-74f5e417d387" containerName="revoke-keys" Nov 02 06:30:54.106119 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:54.106079 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e6672c5-8cec-458c-916c-74f5e417d387" containerName="revoke-keys" Nov 02 06:30:54.106119 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:54.106096 5916 cpu_manager.go:414] "containerMap: removing container" podUID="a132bc7c-7b89-4076-aa20-a3337f3b7548" containerName="cleanup" Nov 02 06:30:54.106119 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:54.106101 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a132bc7c-7b89-4076-aa20-a3337f3b7548" containerName="cleanup" Nov 02 06:30:54.106284 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:54.106172 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e6672c5-8cec-458c-916c-74f5e417d387" containerName="revoke-keys" Nov 02 06:30:54.106284 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:54.106182 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="55759e02-f1b7-408c-ad1f-c132cd5f31a9" containerName="cleanup" Nov 02 06:30:54.112722 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:54.112689 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-094ee0f4-9g8wx" Nov 02 06:30:54.161600 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:54.161573 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 06:30:54.207316 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:54.207251 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-68l4r\"" Nov 02 06:30:54.210443 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:54.210416 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-094ee0f4-9g8wx"] Nov 02 06:30:54.215726 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:54.215697 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzdrh\" (UniqueName: \"kubernetes.io/projected/98c8af93-2990-4e95-82e1-e8f980cb2b9e-kube-api-access-gzdrh\") pod \"maas-api-revoke-sub-094ee0f4-9g8wx\" (UID: \"98c8af93-2990-4e95-82e1-e8f980cb2b9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-094ee0f4-9g8wx" Nov 02 06:30:54.215856 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:54.215829 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98c8af93-2990-4e95-82e1-e8f980cb2b9e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-094ee0f4-9g8wx\" (UID: \"98c8af93-2990-4e95-82e1-e8f980cb2b9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-094ee0f4-9g8wx" Nov 02 06:30:54.216027 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:54.216007 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98c8af93-2990-4e95-82e1-e8f980cb2b9e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-094ee0f4-9g8wx\" (UID: \"98c8af93-2990-4e95-82e1-e8f980cb2b9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-094ee0f4-9g8wx" Nov 02 06:30:54.317401 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:54.317374 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98c8af93-2990-4e95-82e1-e8f980cb2b9e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-094ee0f4-9g8wx\" (UID: \"98c8af93-2990-4e95-82e1-e8f980cb2b9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-094ee0f4-9g8wx" Nov 02 06:30:54.317545 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:54.317440 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98c8af93-2990-4e95-82e1-e8f980cb2b9e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-094ee0f4-9g8wx\" (UID: \"98c8af93-2990-4e95-82e1-e8f980cb2b9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-094ee0f4-9g8wx" Nov 02 06:30:54.317545 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:54.317499 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzdrh\" (UniqueName: \"kubernetes.io/projected/98c8af93-2990-4e95-82e1-e8f980cb2b9e-kube-api-access-gzdrh\") pod \"maas-api-revoke-sub-094ee0f4-9g8wx\" (UID: \"98c8af93-2990-4e95-82e1-e8f980cb2b9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-094ee0f4-9g8wx" Nov 02 06:30:54.318182 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:54.318161 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98c8af93-2990-4e95-82e1-e8f980cb2b9e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-094ee0f4-9g8wx\" (UID: \"98c8af93-2990-4e95-82e1-e8f980cb2b9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-094ee0f4-9g8wx" Nov 02 06:30:54.319626 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:54.319609 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98c8af93-2990-4e95-82e1-e8f980cb2b9e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-094ee0f4-9g8wx\" (UID: \"98c8af93-2990-4e95-82e1-e8f980cb2b9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-094ee0f4-9g8wx" Nov 02 06:30:54.371468 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:54.371412 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzdrh\" (UniqueName: \"kubernetes.io/projected/98c8af93-2990-4e95-82e1-e8f980cb2b9e-kube-api-access-gzdrh\") pod \"maas-api-revoke-sub-094ee0f4-9g8wx\" (UID: \"98c8af93-2990-4e95-82e1-e8f980cb2b9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-094ee0f4-9g8wx" Nov 02 06:30:54.427575 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:54.427551 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-094ee0f4-9g8wx" Nov 02 06:30:54.796272 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:54.796242 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-094ee0f4-9g8wx"] Nov 02 06:30:54.801582 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:30:54.801548 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98c8af93_2990_4e95_82e1_e8f980cb2b9e.slice/crio-8943d62ab91343b8d71c44a020e109d74fc4a0dcba931979cc0863e458102b91 WatchSource:0}: Error finding container 8943d62ab91343b8d71c44a020e109d74fc4a0dcba931979cc0863e458102b91: Status 404 returned error can't find the container with id 8943d62ab91343b8d71c44a020e109d74fc4a0dcba931979cc0863e458102b91 Nov 02 06:30:55.267696 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:55.267661 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-094ee0f4-9g8wx" event={"ID":"98c8af93-2990-4e95-82e1-e8f980cb2b9e","Type":"ContainerStarted","Data":"ad38a11dd46e5d45d23464943dba024e94ab0eebd95a454871394b55ff7652cb"} Nov 02 06:30:55.268101 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:55.267717 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-094ee0f4-9g8wx" event={"ID":"98c8af93-2990-4e95-82e1-e8f980cb2b9e","Type":"ContainerStarted","Data":"8943d62ab91343b8d71c44a020e109d74fc4a0dcba931979cc0863e458102b91"} Nov 02 06:30:55.376733 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:55.376687 5916 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-094ee0f4-9g8wx" podStartSLOduration=1.376674438 podStartE2EDuration="1.376674438s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:30:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:30:55.375090524 +0000 UTC m=+843.646910917" watchObservedRunningTime="2026-11-02 06:30:55.376674438 +0000 UTC m=+843.648494831" Nov 02 06:30:56.275682 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:56.275648 5916 generic.go:356] "Generic (PLEG): container finished" podID="98c8af93-2990-4e95-82e1-e8f980cb2b9e" containerID="ad38a11dd46e5d45d23464943dba024e94ab0eebd95a454871394b55ff7652cb" exitCode=0 Nov 02 06:30:56.276099 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:56.275704 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-094ee0f4-9g8wx" event={"ID":"98c8af93-2990-4e95-82e1-e8f980cb2b9e","Type":"ContainerDied","Data":"ad38a11dd46e5d45d23464943dba024e94ab0eebd95a454871394b55ff7652cb"} Nov 02 06:30:57.422185 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:57.422160 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-094ee0f4-9g8wx" Nov 02 06:30:57.551147 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:57.551061 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98c8af93-2990-4e95-82e1-e8f980cb2b9e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98c8af93-2990-4e95-82e1-e8f980cb2b9e-maas-api-openshift-ca\") pod \"98c8af93-2990-4e95-82e1-e8f980cb2b9e\" (UID: \"98c8af93-2990-4e95-82e1-e8f980cb2b9e\") " Nov 02 06:30:57.551303 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:57.551204 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/98c8af93-2990-4e95-82e1-e8f980cb2b9e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98c8af93-2990-4e95-82e1-e8f980cb2b9e-maas-api-xks-ca\") pod \"98c8af93-2990-4e95-82e1-e8f980cb2b9e\" (UID: \"98c8af93-2990-4e95-82e1-e8f980cb2b9e\") " Nov 02 06:30:57.551303 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:57.551243 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98c8af93-2990-4e95-82e1-e8f980cb2b9e-kube-api-access-gzdrh\" (UniqueName: \"kubernetes.io/projected/98c8af93-2990-4e95-82e1-e8f980cb2b9e-kube-api-access-gzdrh\") pod \"98c8af93-2990-4e95-82e1-e8f980cb2b9e\" (UID: \"98c8af93-2990-4e95-82e1-e8f980cb2b9e\") " Nov 02 06:30:57.551429 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:57.551402 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98c8af93-2990-4e95-82e1-e8f980cb2b9e-maas-api-openshift-ca" pod "98c8af93-2990-4e95-82e1-e8f980cb2b9e" (UID: "98c8af93-2990-4e95-82e1-e8f980cb2b9e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:30:57.551607 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:57.551589 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98c8af93-2990-4e95-82e1-e8f980cb2b9e-maas-api-openshift-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:30:57.553407 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:57.553378 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98c8af93-2990-4e95-82e1-e8f980cb2b9e-maas-api-xks-ca" pod "98c8af93-2990-4e95-82e1-e8f980cb2b9e" (UID: "98c8af93-2990-4e95-82e1-e8f980cb2b9e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:30:57.553494 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:57.553411 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98c8af93-2990-4e95-82e1-e8f980cb2b9e-kube-api-access-gzdrh" pod "98c8af93-2990-4e95-82e1-e8f980cb2b9e" (UID: "98c8af93-2990-4e95-82e1-e8f980cb2b9e"). InnerVolumeSpecName "kube-api-access-gzdrh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:30:57.653133 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:57.653095 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98c8af93-2990-4e95-82e1-e8f980cb2b9e-maas-api-xks-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:30:57.653133 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:57.653128 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gzdrh\" (UniqueName: \"kubernetes.io/projected/98c8af93-2990-4e95-82e1-e8f980cb2b9e-kube-api-access-gzdrh\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:30:58.290622 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:58.290592 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-094ee0f4-9g8wx" Nov 02 06:30:58.290622 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:58.290604 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-094ee0f4-9g8wx" event={"ID":"98c8af93-2990-4e95-82e1-e8f980cb2b9e","Type":"ContainerDied","Data":"8943d62ab91343b8d71c44a020e109d74fc4a0dcba931979cc0863e458102b91"} Nov 02 06:30:58.290830 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:30:58.290651 5916 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8943d62ab91343b8d71c44a020e109d74fc4a0dcba931979cc0863e458102b91" Nov 02 06:31:01.558485 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:01.558444 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-6ff85e/premium-w6-6ff85e-kserve-59848ff6d4-szbxx"] Nov 02 06:31:01.559145 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:01.559127 5916 cpu_manager.go:414] "containerMap: removing container" podUID="98c8af93-2990-4e95-82e1-e8f980cb2b9e" containerName="revoke-keys" Nov 02 06:31:01.559200 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:01.559147 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98c8af93-2990-4e95-82e1-e8f980cb2b9e" containerName="revoke-keys" Nov 02 06:31:01.559301 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:01.559290 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="98c8af93-2990-4e95-82e1-e8f980cb2b9e" containerName="revoke-keys" Nov 02 06:31:01.563275 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:01.563253 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-6ff85e/premium-w6-6ff85e-kserve-59848ff6d4-szbxx" Nov 02 06:31:01.613947 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:01.613916 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-6ff85e/premium-w6-6ff85e-kserve-59848ff6d4-szbxx"] Nov 02 06:31:01.656900 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:01.656868 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-6ff85e\"/\"premium-w6-6ff85e-kserve-self-signed-certs\"" Nov 02 06:31:01.696198 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:01.696166 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9v5bm\" (UniqueName: \"kubernetes.io/projected/eb386976-f898-4f1d-b27c-77bccc735a3b-kube-api-access-9v5bm\") pod \"premium-w6-6ff85e-kserve-59848ff6d4-szbxx\" (UID: \"eb386976-f898-4f1d-b27c-77bccc735a3b\") " pod="e2e-models-e2e-worker-w6-6ff85e/premium-w6-6ff85e-kserve-59848ff6d4-szbxx" Nov 02 06:31:01.696479 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:01.696454 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb386976-f898-4f1d-b27c-77bccc735a3b-home\") pod \"premium-w6-6ff85e-kserve-59848ff6d4-szbxx\" (UID: \"eb386976-f898-4f1d-b27c-77bccc735a3b\") " pod="e2e-models-e2e-worker-w6-6ff85e/premium-w6-6ff85e-kserve-59848ff6d4-szbxx" Nov 02 06:31:01.696658 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:01.696489 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb386976-f898-4f1d-b27c-77bccc735a3b-model-cache\") pod \"premium-w6-6ff85e-kserve-59848ff6d4-szbxx\" (UID: \"eb386976-f898-4f1d-b27c-77bccc735a3b\") " pod="e2e-models-e2e-worker-w6-6ff85e/premium-w6-6ff85e-kserve-59848ff6d4-szbxx" Nov 02 06:31:01.696658 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:01.696533 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb386976-f898-4f1d-b27c-77bccc735a3b-dshm\") pod \"premium-w6-6ff85e-kserve-59848ff6d4-szbxx\" (UID: \"eb386976-f898-4f1d-b27c-77bccc735a3b\") " pod="e2e-models-e2e-worker-w6-6ff85e/premium-w6-6ff85e-kserve-59848ff6d4-szbxx" Nov 02 06:31:01.696658 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:01.696594 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb386976-f898-4f1d-b27c-77bccc735a3b-tmp-dir\") pod \"premium-w6-6ff85e-kserve-59848ff6d4-szbxx\" (UID: \"eb386976-f898-4f1d-b27c-77bccc735a3b\") " pod="e2e-models-e2e-worker-w6-6ff85e/premium-w6-6ff85e-kserve-59848ff6d4-szbxx" Nov 02 06:31:01.696831 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:01.696664 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb386976-f898-4f1d-b27c-77bccc735a3b-tls-certs\") pod \"premium-w6-6ff85e-kserve-59848ff6d4-szbxx\" (UID: \"eb386976-f898-4f1d-b27c-77bccc735a3b\") " pod="e2e-models-e2e-worker-w6-6ff85e/premium-w6-6ff85e-kserve-59848ff6d4-szbxx" Nov 02 06:31:01.797519 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:01.797489 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9v5bm\" (UniqueName: \"kubernetes.io/projected/eb386976-f898-4f1d-b27c-77bccc735a3b-kube-api-access-9v5bm\") pod \"premium-w6-6ff85e-kserve-59848ff6d4-szbxx\" (UID: \"eb386976-f898-4f1d-b27c-77bccc735a3b\") " pod="e2e-models-e2e-worker-w6-6ff85e/premium-w6-6ff85e-kserve-59848ff6d4-szbxx" Nov 02 06:31:01.797721 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:01.797525 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb386976-f898-4f1d-b27c-77bccc735a3b-home\") pod \"premium-w6-6ff85e-kserve-59848ff6d4-szbxx\" (UID: \"eb386976-f898-4f1d-b27c-77bccc735a3b\") " pod="e2e-models-e2e-worker-w6-6ff85e/premium-w6-6ff85e-kserve-59848ff6d4-szbxx" Nov 02 06:31:01.797721 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:01.797545 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb386976-f898-4f1d-b27c-77bccc735a3b-model-cache\") pod \"premium-w6-6ff85e-kserve-59848ff6d4-szbxx\" (UID: \"eb386976-f898-4f1d-b27c-77bccc735a3b\") " pod="e2e-models-e2e-worker-w6-6ff85e/premium-w6-6ff85e-kserve-59848ff6d4-szbxx" Nov 02 06:31:01.797721 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:01.797570 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb386976-f898-4f1d-b27c-77bccc735a3b-dshm\") pod \"premium-w6-6ff85e-kserve-59848ff6d4-szbxx\" (UID: \"eb386976-f898-4f1d-b27c-77bccc735a3b\") " pod="e2e-models-e2e-worker-w6-6ff85e/premium-w6-6ff85e-kserve-59848ff6d4-szbxx" Nov 02 06:31:01.797721 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:01.797603 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb386976-f898-4f1d-b27c-77bccc735a3b-tmp-dir\") pod \"premium-w6-6ff85e-kserve-59848ff6d4-szbxx\" (UID: \"eb386976-f898-4f1d-b27c-77bccc735a3b\") " pod="e2e-models-e2e-worker-w6-6ff85e/premium-w6-6ff85e-kserve-59848ff6d4-szbxx" Nov 02 06:31:01.797721 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:01.797635 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb386976-f898-4f1d-b27c-77bccc735a3b-tls-certs\") pod \"premium-w6-6ff85e-kserve-59848ff6d4-szbxx\" (UID: \"eb386976-f898-4f1d-b27c-77bccc735a3b\") " pod="e2e-models-e2e-worker-w6-6ff85e/premium-w6-6ff85e-kserve-59848ff6d4-szbxx" Nov 02 06:31:01.798001 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:01.797976 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb386976-f898-4f1d-b27c-77bccc735a3b-home\") pod \"premium-w6-6ff85e-kserve-59848ff6d4-szbxx\" (UID: \"eb386976-f898-4f1d-b27c-77bccc735a3b\") " pod="e2e-models-e2e-worker-w6-6ff85e/premium-w6-6ff85e-kserve-59848ff6d4-szbxx" Nov 02 06:31:01.798037 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:01.798015 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb386976-f898-4f1d-b27c-77bccc735a3b-tmp-dir\") pod \"premium-w6-6ff85e-kserve-59848ff6d4-szbxx\" (UID: \"eb386976-f898-4f1d-b27c-77bccc735a3b\") " pod="e2e-models-e2e-worker-w6-6ff85e/premium-w6-6ff85e-kserve-59848ff6d4-szbxx" Nov 02 06:31:01.798080 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:01.798043 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb386976-f898-4f1d-b27c-77bccc735a3b-model-cache\") pod \"premium-w6-6ff85e-kserve-59848ff6d4-szbxx\" (UID: \"eb386976-f898-4f1d-b27c-77bccc735a3b\") " pod="e2e-models-e2e-worker-w6-6ff85e/premium-w6-6ff85e-kserve-59848ff6d4-szbxx" Nov 02 06:31:01.799973 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:01.799944 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb386976-f898-4f1d-b27c-77bccc735a3b-dshm\") pod \"premium-w6-6ff85e-kserve-59848ff6d4-szbxx\" (UID: \"eb386976-f898-4f1d-b27c-77bccc735a3b\") " pod="e2e-models-e2e-worker-w6-6ff85e/premium-w6-6ff85e-kserve-59848ff6d4-szbxx" Nov 02 06:31:01.800295 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:01.800275 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb386976-f898-4f1d-b27c-77bccc735a3b-tls-certs\") pod \"premium-w6-6ff85e-kserve-59848ff6d4-szbxx\" (UID: \"eb386976-f898-4f1d-b27c-77bccc735a3b\") " pod="e2e-models-e2e-worker-w6-6ff85e/premium-w6-6ff85e-kserve-59848ff6d4-szbxx" Nov 02 06:31:01.851324 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:01.851251 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9v5bm\" (UniqueName: \"kubernetes.io/projected/eb386976-f898-4f1d-b27c-77bccc735a3b-kube-api-access-9v5bm\") pod \"premium-w6-6ff85e-kserve-59848ff6d4-szbxx\" (UID: \"eb386976-f898-4f1d-b27c-77bccc735a3b\") " pod="e2e-models-e2e-worker-w6-6ff85e/premium-w6-6ff85e-kserve-59848ff6d4-szbxx" Nov 02 06:31:01.878120 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:01.878086 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-6ff85e/premium-w6-6ff85e-kserve-59848ff6d4-szbxx" Nov 02 06:31:02.274156 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:02.274122 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-6ff85e/premium-w6-6ff85e-kserve-59848ff6d4-szbxx"] Nov 02 06:31:02.279824 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:31:02.279795 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeb386976_f898_4f1d_b27c_77bccc735a3b.slice/crio-ffea19bfea5016d03ff383440b417e03d9599ab839b43625b2c688c451895b70 WatchSource:0}: Error finding container ffea19bfea5016d03ff383440b417e03d9599ab839b43625b2c688c451895b70: Status 404 returned error can't find the container with id ffea19bfea5016d03ff383440b417e03d9599ab839b43625b2c688c451895b70 Nov 02 06:31:02.324673 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:02.324641 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-6ff85e/premium-w6-6ff85e-kserve-59848ff6d4-szbxx" event={"ID":"eb386976-f898-4f1d-b27c-77bccc735a3b","Type":"ContainerStarted","Data":"ffea19bfea5016d03ff383440b417e03d9599ab839b43625b2c688c451895b70"} Nov 02 06:31:03.332363 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:03.332325 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-6ff85e/premium-w6-6ff85e-kserve-59848ff6d4-szbxx" event={"ID":"eb386976-f898-4f1d-b27c-77bccc735a3b","Type":"ContainerStarted","Data":"9d87b95c2d781f8e2b67a8a2cdb01b2e44a1ddd9bd7c0b2b5ee58c70a2973db7"} Nov 02 06:31:03.439687 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:03.439630 5916 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-6ff85e/premium-w6-6ff85e-kserve-59848ff6d4-szbxx" podStartSLOduration=2.439613646 podStartE2EDuration="2.439613646s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:31:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:31:03.438587841 +0000 UTC m=+851.710408235" watchObservedRunningTime="2026-11-02 06:31:03.439613646 +0000 UTC m=+851.711434040" Nov 02 06:31:03.878693 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:03.878657 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-6ff85e/premium-w6-6ff85e-kserve-59848ff6d4-szbxx" Nov 02 06:31:09.581488 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:09.581450 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98d79b6f-hkh6r"] Nov 02 06:31:09.585289 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:09.585269 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98d79b6f-hkh6r" Nov 02 06:31:09.636621 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:09.636587 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 06:31:09.640374 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:09.640345 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98d79b6f-hkh6r"] Nov 02 06:31:09.677772 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:09.677738 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbckf\" (UniqueName: \"kubernetes.io/projected/ab436b1a-64b1-4527-b5c3-b9d9bd0a5883-kube-api-access-fbckf\") pod \"maas-api-revoke-sub-98d79b6f-hkh6r\" (UID: \"ab436b1a-64b1-4527-b5c3-b9d9bd0a5883\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98d79b6f-hkh6r" Nov 02 06:31:09.677946 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:09.677927 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab436b1a-64b1-4527-b5c3-b9d9bd0a5883-maas-api-xks-ca\") pod \"maas-api-revoke-sub-98d79b6f-hkh6r\" (UID: \"ab436b1a-64b1-4527-b5c3-b9d9bd0a5883\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98d79b6f-hkh6r" Nov 02 06:31:09.678012 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:09.677967 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab436b1a-64b1-4527-b5c3-b9d9bd0a5883-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-98d79b6f-hkh6r\" (UID: \"ab436b1a-64b1-4527-b5c3-b9d9bd0a5883\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98d79b6f-hkh6r" Nov 02 06:31:09.682212 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:09.682189 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-68l4r\"" Nov 02 06:31:09.779330 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:09.779301 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbckf\" (UniqueName: \"kubernetes.io/projected/ab436b1a-64b1-4527-b5c3-b9d9bd0a5883-kube-api-access-fbckf\") pod \"maas-api-revoke-sub-98d79b6f-hkh6r\" (UID: \"ab436b1a-64b1-4527-b5c3-b9d9bd0a5883\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98d79b6f-hkh6r" Nov 02 06:31:09.779586 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:09.779385 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab436b1a-64b1-4527-b5c3-b9d9bd0a5883-maas-api-xks-ca\") pod \"maas-api-revoke-sub-98d79b6f-hkh6r\" (UID: \"ab436b1a-64b1-4527-b5c3-b9d9bd0a5883\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98d79b6f-hkh6r" Nov 02 06:31:09.779586 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:09.779406 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab436b1a-64b1-4527-b5c3-b9d9bd0a5883-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-98d79b6f-hkh6r\" (UID: \"ab436b1a-64b1-4527-b5c3-b9d9bd0a5883\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98d79b6f-hkh6r" Nov 02 06:31:09.779984 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:09.779965 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab436b1a-64b1-4527-b5c3-b9d9bd0a5883-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-98d79b6f-hkh6r\" (UID: \"ab436b1a-64b1-4527-b5c3-b9d9bd0a5883\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98d79b6f-hkh6r" Nov 02 06:31:09.781657 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:09.781638 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab436b1a-64b1-4527-b5c3-b9d9bd0a5883-maas-api-xks-ca\") pod \"maas-api-revoke-sub-98d79b6f-hkh6r\" (UID: \"ab436b1a-64b1-4527-b5c3-b9d9bd0a5883\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98d79b6f-hkh6r" Nov 02 06:31:09.834271 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:09.834207 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbckf\" (UniqueName: \"kubernetes.io/projected/ab436b1a-64b1-4527-b5c3-b9d9bd0a5883-kube-api-access-fbckf\") pod \"maas-api-revoke-sub-98d79b6f-hkh6r\" (UID: \"ab436b1a-64b1-4527-b5c3-b9d9bd0a5883\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98d79b6f-hkh6r" Nov 02 06:31:09.899270 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:09.899247 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98d79b6f-hkh6r" Nov 02 06:31:10.274059 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:10.274022 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98d79b6f-hkh6r"] Nov 02 06:31:10.278940 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:31:10.278897 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab436b1a_64b1_4527_b5c3_b9d9bd0a5883.slice/crio-d64506c3a5475dd1a1235657090b0a58573fc2067a1eb49ce915c1ffa8565e49 WatchSource:0}: Error finding container d64506c3a5475dd1a1235657090b0a58573fc2067a1eb49ce915c1ffa8565e49: Status 404 returned error can't find the container with id d64506c3a5475dd1a1235657090b0a58573fc2067a1eb49ce915c1ffa8565e49 Nov 02 06:31:10.390133 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:10.390092 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98d79b6f-hkh6r" event={"ID":"ab436b1a-64b1-4527-b5c3-b9d9bd0a5883","Type":"ContainerStarted","Data":"d64506c3a5475dd1a1235657090b0a58573fc2067a1eb49ce915c1ffa8565e49"} Nov 02 06:31:11.399050 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:11.399019 5916 generic.go:356] "Generic (PLEG): container finished" podID="ab436b1a-64b1-4527-b5c3-b9d9bd0a5883" containerID="120ff91eac2d344fea415a86f04056aa7cc4582284642bf5ffe6cd8c3ffcc407" exitCode=0 Nov 02 06:31:11.399505 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:11.399100 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98d79b6f-hkh6r" event={"ID":"ab436b1a-64b1-4527-b5c3-b9d9bd0a5883","Type":"ContainerDied","Data":"120ff91eac2d344fea415a86f04056aa7cc4582284642bf5ffe6cd8c3ffcc407"} Nov 02 06:31:11.879218 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:11.879118 5916 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-6ff85e/premium-w6-6ff85e-kserve-59848ff6d4-szbxx" Nov 02 06:31:11.892062 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:11.892024 5916 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-6ff85e/premium-w6-6ff85e-kserve-59848ff6d4-szbxx" Nov 02 06:31:12.439237 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:12.439206 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-6ff85e/premium-w6-6ff85e-kserve-59848ff6d4-szbxx" Nov 02 06:31:12.816631 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:12.816604 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98d79b6f-hkh6r" Nov 02 06:31:12.922920 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:12.922862 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab436b1a-64b1-4527-b5c3-b9d9bd0a5883-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab436b1a-64b1-4527-b5c3-b9d9bd0a5883-maas-api-xks-ca\") pod \"ab436b1a-64b1-4527-b5c3-b9d9bd0a5883\" (UID: \"ab436b1a-64b1-4527-b5c3-b9d9bd0a5883\") " Nov 02 06:31:12.923119 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:12.922940 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab436b1a-64b1-4527-b5c3-b9d9bd0a5883-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab436b1a-64b1-4527-b5c3-b9d9bd0a5883-maas-api-openshift-ca\") pod \"ab436b1a-64b1-4527-b5c3-b9d9bd0a5883\" (UID: \"ab436b1a-64b1-4527-b5c3-b9d9bd0a5883\") " Nov 02 06:31:12.923119 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:12.922978 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab436b1a-64b1-4527-b5c3-b9d9bd0a5883-kube-api-access-fbckf\" (UniqueName: \"kubernetes.io/projected/ab436b1a-64b1-4527-b5c3-b9d9bd0a5883-kube-api-access-fbckf\") pod \"ab436b1a-64b1-4527-b5c3-b9d9bd0a5883\" (UID: \"ab436b1a-64b1-4527-b5c3-b9d9bd0a5883\") " Nov 02 06:31:12.923394 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:12.923293 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab436b1a-64b1-4527-b5c3-b9d9bd0a5883-maas-api-openshift-ca" pod "ab436b1a-64b1-4527-b5c3-b9d9bd0a5883" (UID: "ab436b1a-64b1-4527-b5c3-b9d9bd0a5883"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:31:12.923527 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:12.923457 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab436b1a-64b1-4527-b5c3-b9d9bd0a5883-maas-api-openshift-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:31:12.925997 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:12.925408 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab436b1a-64b1-4527-b5c3-b9d9bd0a5883-maas-api-xks-ca" pod "ab436b1a-64b1-4527-b5c3-b9d9bd0a5883" (UID: "ab436b1a-64b1-4527-b5c3-b9d9bd0a5883"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:31:12.925997 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:12.925599 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab436b1a-64b1-4527-b5c3-b9d9bd0a5883-kube-api-access-fbckf" pod "ab436b1a-64b1-4527-b5c3-b9d9bd0a5883" (UID: "ab436b1a-64b1-4527-b5c3-b9d9bd0a5883"). InnerVolumeSpecName "kube-api-access-fbckf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:31:13.024984 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:13.024950 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab436b1a-64b1-4527-b5c3-b9d9bd0a5883-maas-api-xks-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:31:13.024984 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:13.024984 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbckf\" (UniqueName: \"kubernetes.io/projected/ab436b1a-64b1-4527-b5c3-b9d9bd0a5883-kube-api-access-fbckf\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:31:13.417937 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:13.417890 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98d79b6f-hkh6r" Nov 02 06:31:13.418113 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:13.417932 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98d79b6f-hkh6r" event={"ID":"ab436b1a-64b1-4527-b5c3-b9d9bd0a5883","Type":"ContainerDied","Data":"d64506c3a5475dd1a1235657090b0a58573fc2067a1eb49ce915c1ffa8565e49"} Nov 02 06:31:13.418113 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:13.417991 5916 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d64506c3a5475dd1a1235657090b0a58573fc2067a1eb49ce915c1ffa8565e49" Nov 02 06:31:19.892871 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:31:19.892834 5916 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p98cq" containerName="extractor" Nov 02 06:31:29.327669 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:29.327631 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc8a56e2-v5l4q"] Nov 02 06:31:29.328168 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:29.328154 5916 cpu_manager.go:414] "containerMap: removing container" podUID="ab436b1a-64b1-4527-b5c3-b9d9bd0a5883" containerName="revoke-keys" Nov 02 06:31:29.328221 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:29.328169 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab436b1a-64b1-4527-b5c3-b9d9bd0a5883" containerName="revoke-keys" Nov 02 06:31:29.328294 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:29.328284 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab436b1a-64b1-4527-b5c3-b9d9bd0a5883" containerName="revoke-keys" Nov 02 06:31:29.332345 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:29.332327 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc8a56e2-v5l4q" Nov 02 06:31:29.384429 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:29.384392 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc8a56e2-v5l4q"] Nov 02 06:31:29.399918 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:29.399871 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc7f508b-03c7-47a7-9750-a346b915b9f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dc8a56e2-v5l4q\" (UID: \"dc7f508b-03c7-47a7-9750-a346b915b9f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc8a56e2-v5l4q" Nov 02 06:31:29.400102 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:29.399965 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54xb5\" (UniqueName: \"kubernetes.io/projected/dc7f508b-03c7-47a7-9750-a346b915b9f8-kube-api-access-54xb5\") pod \"maas-api-revoke-sub-dc8a56e2-v5l4q\" (UID: \"dc7f508b-03c7-47a7-9750-a346b915b9f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc8a56e2-v5l4q" Nov 02 06:31:29.400102 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:29.400007 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc7f508b-03c7-47a7-9750-a346b915b9f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dc8a56e2-v5l4q\" (UID: \"dc7f508b-03c7-47a7-9750-a346b915b9f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc8a56e2-v5l4q" Nov 02 06:31:29.427307 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:29.427277 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 06:31:29.427491 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:29.427326 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-68l4r\"" Nov 02 06:31:29.501572 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:29.501543 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc7f508b-03c7-47a7-9750-a346b915b9f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dc8a56e2-v5l4q\" (UID: \"dc7f508b-03c7-47a7-9750-a346b915b9f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc8a56e2-v5l4q" Nov 02 06:31:29.501729 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:29.501586 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54xb5\" (UniqueName: \"kubernetes.io/projected/dc7f508b-03c7-47a7-9750-a346b915b9f8-kube-api-access-54xb5\") pod \"maas-api-revoke-sub-dc8a56e2-v5l4q\" (UID: \"dc7f508b-03c7-47a7-9750-a346b915b9f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc8a56e2-v5l4q" Nov 02 06:31:29.501729 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:29.501611 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc7f508b-03c7-47a7-9750-a346b915b9f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dc8a56e2-v5l4q\" (UID: \"dc7f508b-03c7-47a7-9750-a346b915b9f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc8a56e2-v5l4q" Nov 02 06:31:29.502226 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:29.502207 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc7f508b-03c7-47a7-9750-a346b915b9f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dc8a56e2-v5l4q\" (UID: \"dc7f508b-03c7-47a7-9750-a346b915b9f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc8a56e2-v5l4q" Nov 02 06:31:29.504185 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:29.504161 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc7f508b-03c7-47a7-9750-a346b915b9f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dc8a56e2-v5l4q\" (UID: \"dc7f508b-03c7-47a7-9750-a346b915b9f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc8a56e2-v5l4q" Nov 02 06:31:29.554781 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:29.554754 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54xb5\" (UniqueName: \"kubernetes.io/projected/dc7f508b-03c7-47a7-9750-a346b915b9f8-kube-api-access-54xb5\") pod \"maas-api-revoke-sub-dc8a56e2-v5l4q\" (UID: \"dc7f508b-03c7-47a7-9750-a346b915b9f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc8a56e2-v5l4q" Nov 02 06:31:29.646872 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:29.646840 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc8a56e2-v5l4q" Nov 02 06:31:30.028302 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:30.028217 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc8a56e2-v5l4q"] Nov 02 06:31:30.033378 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:31:30.033352 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc7f508b_03c7_47a7_9750_a346b915b9f8.slice/crio-1699f16895aa433d2c5ccd167a76ad608fdbd9183932be3ab03f403d02b6784b WatchSource:0}: Error finding container 1699f16895aa433d2c5ccd167a76ad608fdbd9183932be3ab03f403d02b6784b: Status 404 returned error can't find the container with id 1699f16895aa433d2c5ccd167a76ad608fdbd9183932be3ab03f403d02b6784b Nov 02 06:31:30.531403 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:30.531363 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc8a56e2-v5l4q" event={"ID":"dc7f508b-03c7-47a7-9750-a346b915b9f8","Type":"ContainerStarted","Data":"e07273572efe74b597fc0076dd6f3b83339bf01f3887eba92bd530782e648656"} Nov 02 06:31:30.531859 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:30.531428 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc8a56e2-v5l4q" event={"ID":"dc7f508b-03c7-47a7-9750-a346b915b9f8","Type":"ContainerStarted","Data":"1699f16895aa433d2c5ccd167a76ad608fdbd9183932be3ab03f403d02b6784b"} Nov 02 06:31:30.638450 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:30.638389 5916 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc8a56e2-v5l4q" podStartSLOduration=1.638373494 podStartE2EDuration="1.638373494s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:31:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:31:30.638153884 +0000 UTC m=+878.909974301" watchObservedRunningTime="2026-11-02 06:31:30.638373494 +0000 UTC m=+878.910193889" Nov 02 06:31:31.544163 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:31.544130 5916 generic.go:356] "Generic (PLEG): container finished" podID="dc7f508b-03c7-47a7-9750-a346b915b9f8" containerID="e07273572efe74b597fc0076dd6f3b83339bf01f3887eba92bd530782e648656" exitCode=0 Nov 02 06:31:31.544638 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:31.544193 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc8a56e2-v5l4q" event={"ID":"dc7f508b-03c7-47a7-9750-a346b915b9f8","Type":"ContainerDied","Data":"e07273572efe74b597fc0076dd6f3b83339bf01f3887eba92bd530782e648656"} Nov 02 06:31:32.718827 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:32.718799 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc8a56e2-v5l4q" Nov 02 06:31:32.849330 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:32.849247 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc7f508b-03c7-47a7-9750-a346b915b9f8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc7f508b-03c7-47a7-9750-a346b915b9f8-maas-api-openshift-ca\") pod \"dc7f508b-03c7-47a7-9750-a346b915b9f8\" (UID: \"dc7f508b-03c7-47a7-9750-a346b915b9f8\") " Nov 02 06:31:32.849615 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:32.849590 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc7f508b-03c7-47a7-9750-a346b915b9f8-maas-api-openshift-ca" pod "dc7f508b-03c7-47a7-9750-a346b915b9f8" (UID: "dc7f508b-03c7-47a7-9750-a346b915b9f8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:31:32.849745 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:32.849716 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc7f508b-03c7-47a7-9750-a346b915b9f8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc7f508b-03c7-47a7-9750-a346b915b9f8-maas-api-xks-ca\") pod \"dc7f508b-03c7-47a7-9750-a346b915b9f8\" (UID: \"dc7f508b-03c7-47a7-9750-a346b915b9f8\") " Nov 02 06:31:32.849862 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:32.849764 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc7f508b-03c7-47a7-9750-a346b915b9f8-kube-api-access-54xb5\" (UniqueName: \"kubernetes.io/projected/dc7f508b-03c7-47a7-9750-a346b915b9f8-kube-api-access-54xb5\") pod \"dc7f508b-03c7-47a7-9750-a346b915b9f8\" (UID: \"dc7f508b-03c7-47a7-9750-a346b915b9f8\") " Nov 02 06:31:32.850192 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:32.850172 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc7f508b-03c7-47a7-9750-a346b915b9f8-maas-api-openshift-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:31:32.854053 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:32.854027 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc7f508b-03c7-47a7-9750-a346b915b9f8-kube-api-access-54xb5" pod "dc7f508b-03c7-47a7-9750-a346b915b9f8" (UID: "dc7f508b-03c7-47a7-9750-a346b915b9f8"). InnerVolumeSpecName "kube-api-access-54xb5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:31:32.854157 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:32.854094 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc7f508b-03c7-47a7-9750-a346b915b9f8-maas-api-xks-ca" pod "dc7f508b-03c7-47a7-9750-a346b915b9f8" (UID: "dc7f508b-03c7-47a7-9750-a346b915b9f8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:31:32.951482 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:32.951446 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc7f508b-03c7-47a7-9750-a346b915b9f8-maas-api-xks-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:31:32.951482 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:32.951482 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54xb5\" (UniqueName: \"kubernetes.io/projected/dc7f508b-03c7-47a7-9750-a346b915b9f8-kube-api-access-54xb5\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:31:33.131019 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:33.130936 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0453c67-lct9k"] Nov 02 06:31:33.131637 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:33.131621 5916 cpu_manager.go:414] "containerMap: removing container" podUID="dc7f508b-03c7-47a7-9750-a346b915b9f8" containerName="revoke-keys" Nov 02 06:31:33.131684 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:33.131640 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc7f508b-03c7-47a7-9750-a346b915b9f8" containerName="revoke-keys" Nov 02 06:31:33.131820 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:33.131809 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc7f508b-03c7-47a7-9750-a346b915b9f8" containerName="revoke-keys" Nov 02 06:31:33.135866 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:33.135845 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0453c67-lct9k" Nov 02 06:31:33.187875 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:33.187828 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0453c67-lct9k"] Nov 02 06:31:33.254829 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:33.254792 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q86bn\" (UniqueName: \"kubernetes.io/projected/e9a183ab-1d0c-4b4b-92d5-b051c389aa53-kube-api-access-q86bn\") pod \"maas-api-revoke-sub-f0453c67-lct9k\" (UID: \"e9a183ab-1d0c-4b4b-92d5-b051c389aa53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0453c67-lct9k" Nov 02 06:31:33.255035 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:33.254930 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9a183ab-1d0c-4b4b-92d5-b051c389aa53-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f0453c67-lct9k\" (UID: \"e9a183ab-1d0c-4b4b-92d5-b051c389aa53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0453c67-lct9k" Nov 02 06:31:33.255123 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:33.255055 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9a183ab-1d0c-4b4b-92d5-b051c389aa53-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f0453c67-lct9k\" (UID: \"e9a183ab-1d0c-4b4b-92d5-b051c389aa53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0453c67-lct9k" Nov 02 06:31:33.355801 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:33.355762 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q86bn\" (UniqueName: \"kubernetes.io/projected/e9a183ab-1d0c-4b4b-92d5-b051c389aa53-kube-api-access-q86bn\") pod \"maas-api-revoke-sub-f0453c67-lct9k\" (UID: \"e9a183ab-1d0c-4b4b-92d5-b051c389aa53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0453c67-lct9k" Nov 02 06:31:33.355801 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:33.355810 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9a183ab-1d0c-4b4b-92d5-b051c389aa53-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f0453c67-lct9k\" (UID: \"e9a183ab-1d0c-4b4b-92d5-b051c389aa53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0453c67-lct9k" Nov 02 06:31:33.356066 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:33.355873 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9a183ab-1d0c-4b4b-92d5-b051c389aa53-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f0453c67-lct9k\" (UID: \"e9a183ab-1d0c-4b4b-92d5-b051c389aa53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0453c67-lct9k" Nov 02 06:31:33.356471 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:33.356452 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9a183ab-1d0c-4b4b-92d5-b051c389aa53-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f0453c67-lct9k\" (UID: \"e9a183ab-1d0c-4b4b-92d5-b051c389aa53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0453c67-lct9k" Nov 02 06:31:33.358235 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:33.358213 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9a183ab-1d0c-4b4b-92d5-b051c389aa53-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f0453c67-lct9k\" (UID: \"e9a183ab-1d0c-4b4b-92d5-b051c389aa53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0453c67-lct9k" Nov 02 06:31:33.408994 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:33.408965 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q86bn\" (UniqueName: \"kubernetes.io/projected/e9a183ab-1d0c-4b4b-92d5-b051c389aa53-kube-api-access-q86bn\") pod \"maas-api-revoke-sub-f0453c67-lct9k\" (UID: \"e9a183ab-1d0c-4b4b-92d5-b051c389aa53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0453c67-lct9k" Nov 02 06:31:33.454775 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:33.454734 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0453c67-lct9k" Nov 02 06:31:33.560526 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:33.560489 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc8a56e2-v5l4q" event={"ID":"dc7f508b-03c7-47a7-9750-a346b915b9f8","Type":"ContainerDied","Data":"1699f16895aa433d2c5ccd167a76ad608fdbd9183932be3ab03f403d02b6784b"} Nov 02 06:31:33.560681 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:33.560552 5916 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1699f16895aa433d2c5ccd167a76ad608fdbd9183932be3ab03f403d02b6784b" Nov 02 06:31:33.560681 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:33.560620 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc8a56e2-v5l4q" Nov 02 06:31:33.823267 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:33.823187 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0453c67-lct9k"] Nov 02 06:31:33.829449 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:31:33.829413 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9a183ab_1d0c_4b4b_92d5_b051c389aa53.slice/crio-3760d35916528e16bcad57b776046bc1328c56d25582133e85dec4cc3fd2b040 WatchSource:0}: Error finding container 3760d35916528e16bcad57b776046bc1328c56d25582133e85dec4cc3fd2b040: Status 404 returned error can't find the container with id 3760d35916528e16bcad57b776046bc1328c56d25582133e85dec4cc3fd2b040 Nov 02 06:31:34.569027 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:34.568996 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0453c67-lct9k" event={"ID":"e9a183ab-1d0c-4b4b-92d5-b051c389aa53","Type":"ContainerStarted","Data":"68a7ddae2c1c7b7426e8cdbbf57b8aad43e39dbc5c977f4ff7295f0aeffe4f8d"} Nov 02 06:31:34.569191 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:34.569050 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0453c67-lct9k" event={"ID":"e9a183ab-1d0c-4b4b-92d5-b051c389aa53","Type":"ContainerStarted","Data":"3760d35916528e16bcad57b776046bc1328c56d25582133e85dec4cc3fd2b040"} Nov 02 06:31:34.680847 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:34.680786 5916 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0453c67-lct9k" podStartSLOduration=1.6807679979999999 podStartE2EDuration="1.680767998s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:31:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:31:34.675789343 +0000 UTC m=+882.947609736" watchObservedRunningTime="2026-11-02 06:31:34.680767998 +0000 UTC m=+882.952588395" Nov 02 06:31:35.577510 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:35.577472 5916 generic.go:356] "Generic (PLEG): container finished" podID="e9a183ab-1d0c-4b4b-92d5-b051c389aa53" containerID="68a7ddae2c1c7b7426e8cdbbf57b8aad43e39dbc5c977f4ff7295f0aeffe4f8d" exitCode=0 Nov 02 06:31:35.577952 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:35.577577 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0453c67-lct9k" event={"ID":"e9a183ab-1d0c-4b4b-92d5-b051c389aa53","Type":"ContainerDied","Data":"68a7ddae2c1c7b7426e8cdbbf57b8aad43e39dbc5c977f4ff7295f0aeffe4f8d"} Nov 02 06:31:36.750153 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:36.750120 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0453c67-lct9k" Nov 02 06:31:36.897677 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:36.897647 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9a183ab-1d0c-4b4b-92d5-b051c389aa53-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9a183ab-1d0c-4b4b-92d5-b051c389aa53-maas-api-openshift-ca\") pod \"e9a183ab-1d0c-4b4b-92d5-b051c389aa53\" (UID: \"e9a183ab-1d0c-4b4b-92d5-b051c389aa53\") " Nov 02 06:31:36.897860 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:36.897689 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9a183ab-1d0c-4b4b-92d5-b051c389aa53-kube-api-access-q86bn\" (UniqueName: \"kubernetes.io/projected/e9a183ab-1d0c-4b4b-92d5-b051c389aa53-kube-api-access-q86bn\") pod \"e9a183ab-1d0c-4b4b-92d5-b051c389aa53\" (UID: \"e9a183ab-1d0c-4b4b-92d5-b051c389aa53\") " Nov 02 06:31:36.897860 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:36.897756 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e9a183ab-1d0c-4b4b-92d5-b051c389aa53-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9a183ab-1d0c-4b4b-92d5-b051c389aa53-maas-api-xks-ca\") pod \"e9a183ab-1d0c-4b4b-92d5-b051c389aa53\" (UID: \"e9a183ab-1d0c-4b4b-92d5-b051c389aa53\") " Nov 02 06:31:36.898051 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:36.898029 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9a183ab-1d0c-4b4b-92d5-b051c389aa53-maas-api-openshift-ca" pod "e9a183ab-1d0c-4b4b-92d5-b051c389aa53" (UID: "e9a183ab-1d0c-4b4b-92d5-b051c389aa53"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:31:36.898167 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:36.898145 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9a183ab-1d0c-4b4b-92d5-b051c389aa53-maas-api-openshift-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:31:36.899880 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:36.899847 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9a183ab-1d0c-4b4b-92d5-b051c389aa53-maas-api-xks-ca" pod "e9a183ab-1d0c-4b4b-92d5-b051c389aa53" (UID: "e9a183ab-1d0c-4b4b-92d5-b051c389aa53"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:31:36.899983 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:36.899899 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9a183ab-1d0c-4b4b-92d5-b051c389aa53-kube-api-access-q86bn" pod "e9a183ab-1d0c-4b4b-92d5-b051c389aa53" (UID: "e9a183ab-1d0c-4b4b-92d5-b051c389aa53"). InnerVolumeSpecName "kube-api-access-q86bn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:31:36.998794 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:36.998756 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q86bn\" (UniqueName: \"kubernetes.io/projected/e9a183ab-1d0c-4b4b-92d5-b051c389aa53-kube-api-access-q86bn\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:31:36.998794 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:36.998787 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9a183ab-1d0c-4b4b-92d5-b051c389aa53-maas-api-xks-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:31:37.599928 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:37.596386 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0453c67-lct9k" event={"ID":"e9a183ab-1d0c-4b4b-92d5-b051c389aa53","Type":"ContainerDied","Data":"3760d35916528e16bcad57b776046bc1328c56d25582133e85dec4cc3fd2b040"} Nov 02 06:31:37.599928 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:37.596449 5916 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3760d35916528e16bcad57b776046bc1328c56d25582133e85dec4cc3fd2b040" Nov 02 06:31:37.599928 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:37.596513 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0453c67-lct9k" Nov 02 06:31:40.092856 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:40.092822 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eeebfe4d-2pfpp"] Nov 02 06:31:40.093383 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:40.093368 5916 cpu_manager.go:414] "containerMap: removing container" podUID="e9a183ab-1d0c-4b4b-92d5-b051c389aa53" containerName="revoke-keys" Nov 02 06:31:40.093426 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:40.093384 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9a183ab-1d0c-4b4b-92d5-b051c389aa53" containerName="revoke-keys" Nov 02 06:31:40.093491 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:40.093482 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9a183ab-1d0c-4b4b-92d5-b051c389aa53" containerName="revoke-keys" Nov 02 06:31:40.097146 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:40.097123 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeebfe4d-2pfpp" Nov 02 06:31:40.149758 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:40.149703 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eeebfe4d-2pfpp"] Nov 02 06:31:40.191745 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:40.191706 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 06:31:40.191935 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:40.191892 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-68l4r\"" Nov 02 06:31:40.240305 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:40.240266 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f21f10e-9b76-4a63-9e50-ff819e919b70-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eeebfe4d-2pfpp\" (UID: \"7f21f10e-9b76-4a63-9e50-ff819e919b70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeebfe4d-2pfpp" Nov 02 06:31:40.240449 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:40.240351 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f21f10e-9b76-4a63-9e50-ff819e919b70-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eeebfe4d-2pfpp\" (UID: \"7f21f10e-9b76-4a63-9e50-ff819e919b70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeebfe4d-2pfpp" Nov 02 06:31:40.240449 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:40.240404 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4vw5\" (UniqueName: \"kubernetes.io/projected/7f21f10e-9b76-4a63-9e50-ff819e919b70-kube-api-access-m4vw5\") pod \"maas-api-revoke-sub-eeebfe4d-2pfpp\" (UID: \"7f21f10e-9b76-4a63-9e50-ff819e919b70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeebfe4d-2pfpp" Nov 02 06:31:40.341854 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:40.341817 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f21f10e-9b76-4a63-9e50-ff819e919b70-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eeebfe4d-2pfpp\" (UID: \"7f21f10e-9b76-4a63-9e50-ff819e919b70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeebfe4d-2pfpp" Nov 02 06:31:40.342048 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:40.341958 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f21f10e-9b76-4a63-9e50-ff819e919b70-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eeebfe4d-2pfpp\" (UID: \"7f21f10e-9b76-4a63-9e50-ff819e919b70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeebfe4d-2pfpp" Nov 02 06:31:40.342048 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:40.342024 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4vw5\" (UniqueName: \"kubernetes.io/projected/7f21f10e-9b76-4a63-9e50-ff819e919b70-kube-api-access-m4vw5\") pod \"maas-api-revoke-sub-eeebfe4d-2pfpp\" (UID: \"7f21f10e-9b76-4a63-9e50-ff819e919b70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeebfe4d-2pfpp" Nov 02 06:31:40.342592 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:40.342569 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f21f10e-9b76-4a63-9e50-ff819e919b70-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eeebfe4d-2pfpp\" (UID: \"7f21f10e-9b76-4a63-9e50-ff819e919b70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeebfe4d-2pfpp" Nov 02 06:31:40.344323 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:40.344271 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f21f10e-9b76-4a63-9e50-ff819e919b70-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eeebfe4d-2pfpp\" (UID: \"7f21f10e-9b76-4a63-9e50-ff819e919b70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeebfe4d-2pfpp" Nov 02 06:31:40.395541 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:40.395512 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4vw5\" (UniqueName: \"kubernetes.io/projected/7f21f10e-9b76-4a63-9e50-ff819e919b70-kube-api-access-m4vw5\") pod \"maas-api-revoke-sub-eeebfe4d-2pfpp\" (UID: \"7f21f10e-9b76-4a63-9e50-ff819e919b70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeebfe4d-2pfpp" Nov 02 06:31:40.410351 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:40.410322 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeebfe4d-2pfpp" Nov 02 06:31:40.786782 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:40.786746 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eeebfe4d-2pfpp"] Nov 02 06:31:40.792374 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:31:40.792348 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f21f10e_9b76_4a63_9e50_ff819e919b70.slice/crio-a4aa8cf5c6904072637633c1f0c2fec8b05da4a391411be66e3c5b05d9f85029 WatchSource:0}: Error finding container a4aa8cf5c6904072637633c1f0c2fec8b05da4a391411be66e3c5b05d9f85029: Status 404 returned error can't find the container with id a4aa8cf5c6904072637633c1f0c2fec8b05da4a391411be66e3c5b05d9f85029 Nov 02 06:31:41.623465 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:41.623435 5916 generic.go:356] "Generic (PLEG): container finished" podID="7f21f10e-9b76-4a63-9e50-ff819e919b70" containerID="c04c432224fb63e1ab763aff16faa05df87cc3ad667aa4c5265d0a0fadc1e75c" exitCode=0 Nov 02 06:31:41.623814 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:41.623519 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeebfe4d-2pfpp" event={"ID":"7f21f10e-9b76-4a63-9e50-ff819e919b70","Type":"ContainerDied","Data":"c04c432224fb63e1ab763aff16faa05df87cc3ad667aa4c5265d0a0fadc1e75c"} Nov 02 06:31:41.623814 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:41.623569 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeebfe4d-2pfpp" event={"ID":"7f21f10e-9b76-4a63-9e50-ff819e919b70","Type":"ContainerStarted","Data":"a4aa8cf5c6904072637633c1f0c2fec8b05da4a391411be66e3c5b05d9f85029"} Nov 02 06:31:42.776372 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:42.776349 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeebfe4d-2pfpp" Nov 02 06:31:42.864987 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:42.864950 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f21f10e-9b76-4a63-9e50-ff819e919b70-kube-api-access-m4vw5\" (UniqueName: \"kubernetes.io/projected/7f21f10e-9b76-4a63-9e50-ff819e919b70-kube-api-access-m4vw5\") pod \"7f21f10e-9b76-4a63-9e50-ff819e919b70\" (UID: \"7f21f10e-9b76-4a63-9e50-ff819e919b70\") " Nov 02 06:31:42.865182 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:42.865021 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f21f10e-9b76-4a63-9e50-ff819e919b70-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f21f10e-9b76-4a63-9e50-ff819e919b70-maas-api-xks-ca\") pod \"7f21f10e-9b76-4a63-9e50-ff819e919b70\" (UID: \"7f21f10e-9b76-4a63-9e50-ff819e919b70\") " Nov 02 06:31:42.865182 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:42.865044 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f21f10e-9b76-4a63-9e50-ff819e919b70-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f21f10e-9b76-4a63-9e50-ff819e919b70-maas-api-openshift-ca\") pod \"7f21f10e-9b76-4a63-9e50-ff819e919b70\" (UID: \"7f21f10e-9b76-4a63-9e50-ff819e919b70\") " Nov 02 06:31:42.865483 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:42.865450 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f21f10e-9b76-4a63-9e50-ff819e919b70-maas-api-openshift-ca" pod "7f21f10e-9b76-4a63-9e50-ff819e919b70" (UID: "7f21f10e-9b76-4a63-9e50-ff819e919b70"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:31:42.867160 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:42.867137 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f21f10e-9b76-4a63-9e50-ff819e919b70-maas-api-xks-ca" pod "7f21f10e-9b76-4a63-9e50-ff819e919b70" (UID: "7f21f10e-9b76-4a63-9e50-ff819e919b70"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:31:42.867160 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:42.867149 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f21f10e-9b76-4a63-9e50-ff819e919b70-kube-api-access-m4vw5" pod "7f21f10e-9b76-4a63-9e50-ff819e919b70" (UID: "7f21f10e-9b76-4a63-9e50-ff819e919b70"). InnerVolumeSpecName "kube-api-access-m4vw5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:31:42.966410 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:42.966381 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4vw5\" (UniqueName: \"kubernetes.io/projected/7f21f10e-9b76-4a63-9e50-ff819e919b70-kube-api-access-m4vw5\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:31:42.966410 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:42.966405 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f21f10e-9b76-4a63-9e50-ff819e919b70-maas-api-xks-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:31:42.966410 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:42.966415 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f21f10e-9b76-4a63-9e50-ff819e919b70-maas-api-openshift-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:31:43.641929 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:43.641343 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeebfe4d-2pfpp" Nov 02 06:31:43.641929 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:43.641345 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeebfe4d-2pfpp" event={"ID":"7f21f10e-9b76-4a63-9e50-ff819e919b70","Type":"ContainerDied","Data":"a4aa8cf5c6904072637633c1f0c2fec8b05da4a391411be66e3c5b05d9f85029"} Nov 02 06:31:43.641929 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:43.641535 5916 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a4aa8cf5c6904072637633c1f0c2fec8b05da4a391411be66e3c5b05d9f85029" Nov 02 06:31:49.894143 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:31:49.894100 5916 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p98cq" containerName="extractor" Nov 02 06:31:51.968199 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:51.968167 5916 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-csfhg_17dee72b-2f47-4016-8ffe-3a89934e66ee/console-operator/2.log" Nov 02 06:31:51.984854 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:31:51.984832 5916 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-csfhg_17dee72b-2f47-4016-8ffe-3a89934e66ee/console-operator/2.log" Nov 02 06:32:00.007217 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:00.007177 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20e416e2-6t2tp"] Nov 02 06:32:00.007655 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:00.007641 5916 cpu_manager.go:414] "containerMap: removing container" podUID="7f21f10e-9b76-4a63-9e50-ff819e919b70" containerName="revoke-keys" Nov 02 06:32:00.007655 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:00.007653 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f21f10e-9b76-4a63-9e50-ff819e919b70" containerName="revoke-keys" Nov 02 06:32:00.007778 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:00.007756 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f21f10e-9b76-4a63-9e50-ff819e919b70" containerName="revoke-keys" Nov 02 06:32:00.013553 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:00.013531 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20e416e2-6t2tp" Nov 02 06:32:00.048055 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:00.048026 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e02b453-ca29-488c-a56c-3281e1d39f2f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-20e416e2-6t2tp\" (UID: \"0e02b453-ca29-488c-a56c-3281e1d39f2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20e416e2-6t2tp" Nov 02 06:32:00.048212 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:00.048101 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e02b453-ca29-488c-a56c-3281e1d39f2f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-20e416e2-6t2tp\" (UID: \"0e02b453-ca29-488c-a56c-3281e1d39f2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20e416e2-6t2tp" Nov 02 06:32:00.048212 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:00.048157 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ngxv\" (UniqueName: \"kubernetes.io/projected/0e02b453-ca29-488c-a56c-3281e1d39f2f-kube-api-access-4ngxv\") pod \"maas-api-revoke-sub-20e416e2-6t2tp\" (UID: \"0e02b453-ca29-488c-a56c-3281e1d39f2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20e416e2-6t2tp" Nov 02 06:32:00.063976 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:00.063948 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20e416e2-6t2tp"] Nov 02 06:32:00.065896 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:00.065873 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 06:32:00.066078 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:00.066026 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-68l4r\"" Nov 02 06:32:00.150927 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:00.150473 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e02b453-ca29-488c-a56c-3281e1d39f2f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-20e416e2-6t2tp\" (UID: \"0e02b453-ca29-488c-a56c-3281e1d39f2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20e416e2-6t2tp" Nov 02 06:32:00.150927 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:00.150533 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e02b453-ca29-488c-a56c-3281e1d39f2f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-20e416e2-6t2tp\" (UID: \"0e02b453-ca29-488c-a56c-3281e1d39f2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20e416e2-6t2tp" Nov 02 06:32:00.150927 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:00.150587 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ngxv\" (UniqueName: \"kubernetes.io/projected/0e02b453-ca29-488c-a56c-3281e1d39f2f-kube-api-access-4ngxv\") pod \"maas-api-revoke-sub-20e416e2-6t2tp\" (UID: \"0e02b453-ca29-488c-a56c-3281e1d39f2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20e416e2-6t2tp" Nov 02 06:32:00.153964 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:00.151730 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e02b453-ca29-488c-a56c-3281e1d39f2f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-20e416e2-6t2tp\" (UID: \"0e02b453-ca29-488c-a56c-3281e1d39f2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20e416e2-6t2tp" Nov 02 06:32:00.157925 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:00.154661 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e02b453-ca29-488c-a56c-3281e1d39f2f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-20e416e2-6t2tp\" (UID: \"0e02b453-ca29-488c-a56c-3281e1d39f2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20e416e2-6t2tp" Nov 02 06:32:00.203919 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:00.203879 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ngxv\" (UniqueName: \"kubernetes.io/projected/0e02b453-ca29-488c-a56c-3281e1d39f2f-kube-api-access-4ngxv\") pod \"maas-api-revoke-sub-20e416e2-6t2tp\" (UID: \"0e02b453-ca29-488c-a56c-3281e1d39f2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20e416e2-6t2tp" Nov 02 06:32:00.327733 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:00.327645 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20e416e2-6t2tp" Nov 02 06:32:00.699307 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:00.699273 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20e416e2-6t2tp"] Nov 02 06:32:00.704870 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:32:00.704842 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e02b453_ca29_488c_a56c_3281e1d39f2f.slice/crio-512c12bd445561319222d751c6b160b252ad851a8e4bcad1d7dcc459223d6094 WatchSource:0}: Error finding container 512c12bd445561319222d751c6b160b252ad851a8e4bcad1d7dcc459223d6094: Status 404 returned error can't find the container with id 512c12bd445561319222d751c6b160b252ad851a8e4bcad1d7dcc459223d6094 Nov 02 06:32:00.779966 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:00.779931 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20e416e2-6t2tp" event={"ID":"0e02b453-ca29-488c-a56c-3281e1d39f2f","Type":"ContainerStarted","Data":"512c12bd445561319222d751c6b160b252ad851a8e4bcad1d7dcc459223d6094"} Nov 02 06:32:01.796660 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:01.796622 5916 generic.go:356] "Generic (PLEG): container finished" podID="0e02b453-ca29-488c-a56c-3281e1d39f2f" containerID="63fa2ffb3d304e2025ededdbd05f59c7b5f10a819e93379a48e1076b52df072e" exitCode=0 Nov 02 06:32:01.797148 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:01.796663 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20e416e2-6t2tp" event={"ID":"0e02b453-ca29-488c-a56c-3281e1d39f2f","Type":"ContainerDied","Data":"63fa2ffb3d304e2025ededdbd05f59c7b5f10a819e93379a48e1076b52df072e"} Nov 02 06:32:02.453508 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:02.453476 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38e2d5b3-9qdrh"] Nov 02 06:32:02.457556 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:02.457534 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38e2d5b3-9qdrh" Nov 02 06:32:02.532597 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:02.532563 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38e2d5b3-9qdrh"] Nov 02 06:32:02.576494 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:02.576464 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e308286b-3f6e-4950-9605-daa0d1d68b5d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-38e2d5b3-9qdrh\" (UID: \"e308286b-3f6e-4950-9605-daa0d1d68b5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38e2d5b3-9qdrh" Nov 02 06:32:02.576666 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:02.576640 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kf58g\" (UniqueName: \"kubernetes.io/projected/e308286b-3f6e-4950-9605-daa0d1d68b5d-kube-api-access-kf58g\") pod \"maas-api-revoke-sub-38e2d5b3-9qdrh\" (UID: \"e308286b-3f6e-4950-9605-daa0d1d68b5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38e2d5b3-9qdrh" Nov 02 06:32:02.576721 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:02.576675 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e308286b-3f6e-4950-9605-daa0d1d68b5d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-38e2d5b3-9qdrh\" (UID: \"e308286b-3f6e-4950-9605-daa0d1d68b5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38e2d5b3-9qdrh" Nov 02 06:32:02.677323 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:02.677281 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kf58g\" (UniqueName: \"kubernetes.io/projected/e308286b-3f6e-4950-9605-daa0d1d68b5d-kube-api-access-kf58g\") pod \"maas-api-revoke-sub-38e2d5b3-9qdrh\" (UID: \"e308286b-3f6e-4950-9605-daa0d1d68b5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38e2d5b3-9qdrh" Nov 02 06:32:02.677532 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:02.677344 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e308286b-3f6e-4950-9605-daa0d1d68b5d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-38e2d5b3-9qdrh\" (UID: \"e308286b-3f6e-4950-9605-daa0d1d68b5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38e2d5b3-9qdrh" Nov 02 06:32:02.677532 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:02.677413 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e308286b-3f6e-4950-9605-daa0d1d68b5d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-38e2d5b3-9qdrh\" (UID: \"e308286b-3f6e-4950-9605-daa0d1d68b5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38e2d5b3-9qdrh" Nov 02 06:32:02.678108 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:02.678084 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e308286b-3f6e-4950-9605-daa0d1d68b5d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-38e2d5b3-9qdrh\" (UID: \"e308286b-3f6e-4950-9605-daa0d1d68b5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38e2d5b3-9qdrh" Nov 02 06:32:02.679990 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:02.679961 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e308286b-3f6e-4950-9605-daa0d1d68b5d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-38e2d5b3-9qdrh\" (UID: \"e308286b-3f6e-4950-9605-daa0d1d68b5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38e2d5b3-9qdrh" Nov 02 06:32:02.731293 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:02.731217 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kf58g\" (UniqueName: \"kubernetes.io/projected/e308286b-3f6e-4950-9605-daa0d1d68b5d-kube-api-access-kf58g\") pod \"maas-api-revoke-sub-38e2d5b3-9qdrh\" (UID: \"e308286b-3f6e-4950-9605-daa0d1d68b5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38e2d5b3-9qdrh" Nov 02 06:32:02.770564 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:02.770535 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38e2d5b3-9qdrh" Nov 02 06:32:03.003740 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:03.003713 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20e416e2-6t2tp" Nov 02 06:32:03.083049 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:03.083013 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e02b453-ca29-488c-a56c-3281e1d39f2f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e02b453-ca29-488c-a56c-3281e1d39f2f-maas-api-openshift-ca\") pod \"0e02b453-ca29-488c-a56c-3281e1d39f2f\" (UID: \"0e02b453-ca29-488c-a56c-3281e1d39f2f\") " Nov 02 06:32:03.083049 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:03.083053 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e02b453-ca29-488c-a56c-3281e1d39f2f-kube-api-access-4ngxv\" (UniqueName: \"kubernetes.io/projected/0e02b453-ca29-488c-a56c-3281e1d39f2f-kube-api-access-4ngxv\") pod \"0e02b453-ca29-488c-a56c-3281e1d39f2f\" (UID: \"0e02b453-ca29-488c-a56c-3281e1d39f2f\") " Nov 02 06:32:03.083302 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:03.083083 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e02b453-ca29-488c-a56c-3281e1d39f2f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e02b453-ca29-488c-a56c-3281e1d39f2f-maas-api-xks-ca\") pod \"0e02b453-ca29-488c-a56c-3281e1d39f2f\" (UID: \"0e02b453-ca29-488c-a56c-3281e1d39f2f\") " Nov 02 06:32:03.083444 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:03.083420 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e02b453-ca29-488c-a56c-3281e1d39f2f-maas-api-openshift-ca" pod "0e02b453-ca29-488c-a56c-3281e1d39f2f" (UID: "0e02b453-ca29-488c-a56c-3281e1d39f2f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:32:03.085133 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:03.085111 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e02b453-ca29-488c-a56c-3281e1d39f2f-maas-api-xks-ca" pod "0e02b453-ca29-488c-a56c-3281e1d39f2f" (UID: "0e02b453-ca29-488c-a56c-3281e1d39f2f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:32:03.085209 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:03.085179 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e02b453-ca29-488c-a56c-3281e1d39f2f-kube-api-access-4ngxv" pod "0e02b453-ca29-488c-a56c-3281e1d39f2f" (UID: "0e02b453-ca29-488c-a56c-3281e1d39f2f"). InnerVolumeSpecName "kube-api-access-4ngxv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:32:03.157210 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:32:03.157182 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode308286b_3f6e_4950_9605_daa0d1d68b5d.slice/crio-e69710c09fe2534f43876029494270ab983ee0f1d5b646c1476b9902940396fd WatchSource:0}: Error finding container e69710c09fe2534f43876029494270ab983ee0f1d5b646c1476b9902940396fd: Status 404 returned error can't find the container with id e69710c09fe2534f43876029494270ab983ee0f1d5b646c1476b9902940396fd Nov 02 06:32:03.159834 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:03.159809 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38e2d5b3-9qdrh"] Nov 02 06:32:03.184643 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:03.184610 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e02b453-ca29-488c-a56c-3281e1d39f2f-maas-api-xks-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:32:03.184643 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:03.184637 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e02b453-ca29-488c-a56c-3281e1d39f2f-maas-api-openshift-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:32:03.184802 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:03.184653 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ngxv\" (UniqueName: \"kubernetes.io/projected/0e02b453-ca29-488c-a56c-3281e1d39f2f-kube-api-access-4ngxv\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:32:03.811775 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:03.811732 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38e2d5b3-9qdrh" event={"ID":"e308286b-3f6e-4950-9605-daa0d1d68b5d","Type":"ContainerStarted","Data":"ec69be554402404194189fa794deae4d5f190aa498b36f57b88ae236feabfdc9"} Nov 02 06:32:03.811985 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:03.811832 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38e2d5b3-9qdrh" event={"ID":"e308286b-3f6e-4950-9605-daa0d1d68b5d","Type":"ContainerStarted","Data":"e69710c09fe2534f43876029494270ab983ee0f1d5b646c1476b9902940396fd"} Nov 02 06:32:03.813245 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:03.813220 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20e416e2-6t2tp" event={"ID":"0e02b453-ca29-488c-a56c-3281e1d39f2f","Type":"ContainerDied","Data":"512c12bd445561319222d751c6b160b252ad851a8e4bcad1d7dcc459223d6094"} Nov 02 06:32:03.813366 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:03.813252 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20e416e2-6t2tp" Nov 02 06:32:03.813366 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:03.813259 5916 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="512c12bd445561319222d751c6b160b252ad851a8e4bcad1d7dcc459223d6094" Nov 02 06:32:03.918087 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:03.918037 5916 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38e2d5b3-9qdrh" podStartSLOduration=1.918023072 podStartE2EDuration="1.918023072s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:32:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:32:03.916560031 +0000 UTC m=+912.188380437" watchObservedRunningTime="2026-11-02 06:32:03.918023072 +0000 UTC m=+912.189843465" Nov 02 06:32:03.965471 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:03.965393 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8475f4f7-p54s9"] Nov 02 06:32:03.965872 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:03.965859 5916 cpu_manager.go:414] "containerMap: removing container" podUID="0e02b453-ca29-488c-a56c-3281e1d39f2f" containerName="revoke-keys" Nov 02 06:32:03.965948 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:03.965873 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e02b453-ca29-488c-a56c-3281e1d39f2f" containerName="revoke-keys" Nov 02 06:32:03.966036 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:03.966024 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e02b453-ca29-488c-a56c-3281e1d39f2f" containerName="revoke-keys" Nov 02 06:32:03.969570 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:03.969554 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8475f4f7-p54s9" Nov 02 06:32:03.981370 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:03.981338 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8475f4f7-p54s9"] Nov 02 06:32:04.094533 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:04.094491 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc833f51-60f9-4e3c-8c15-1d0db873eea5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8475f4f7-p54s9\" (UID: \"bc833f51-60f9-4e3c-8c15-1d0db873eea5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8475f4f7-p54s9" Nov 02 06:32:04.094930 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:04.094658 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t75p6\" (UniqueName: \"kubernetes.io/projected/bc833f51-60f9-4e3c-8c15-1d0db873eea5-kube-api-access-t75p6\") pod \"maas-api-revoke-sub-8475f4f7-p54s9\" (UID: \"bc833f51-60f9-4e3c-8c15-1d0db873eea5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8475f4f7-p54s9" Nov 02 06:32:04.094930 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:04.094694 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc833f51-60f9-4e3c-8c15-1d0db873eea5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8475f4f7-p54s9\" (UID: \"bc833f51-60f9-4e3c-8c15-1d0db873eea5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8475f4f7-p54s9" Nov 02 06:32:04.196271 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:04.196234 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc833f51-60f9-4e3c-8c15-1d0db873eea5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8475f4f7-p54s9\" (UID: \"bc833f51-60f9-4e3c-8c15-1d0db873eea5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8475f4f7-p54s9" Nov 02 06:32:04.196445 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:04.196356 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t75p6\" (UniqueName: \"kubernetes.io/projected/bc833f51-60f9-4e3c-8c15-1d0db873eea5-kube-api-access-t75p6\") pod \"maas-api-revoke-sub-8475f4f7-p54s9\" (UID: \"bc833f51-60f9-4e3c-8c15-1d0db873eea5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8475f4f7-p54s9" Nov 02 06:32:04.196445 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:04.196401 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc833f51-60f9-4e3c-8c15-1d0db873eea5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8475f4f7-p54s9\" (UID: \"bc833f51-60f9-4e3c-8c15-1d0db873eea5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8475f4f7-p54s9" Nov 02 06:32:04.197081 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:04.197060 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc833f51-60f9-4e3c-8c15-1d0db873eea5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8475f4f7-p54s9\" (UID: \"bc833f51-60f9-4e3c-8c15-1d0db873eea5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8475f4f7-p54s9" Nov 02 06:32:04.198632 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:04.198614 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc833f51-60f9-4e3c-8c15-1d0db873eea5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8475f4f7-p54s9\" (UID: \"bc833f51-60f9-4e3c-8c15-1d0db873eea5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8475f4f7-p54s9" Nov 02 06:32:04.249649 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:04.249578 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t75p6\" (UniqueName: \"kubernetes.io/projected/bc833f51-60f9-4e3c-8c15-1d0db873eea5-kube-api-access-t75p6\") pod \"maas-api-revoke-sub-8475f4f7-p54s9\" (UID: \"bc833f51-60f9-4e3c-8c15-1d0db873eea5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8475f4f7-p54s9" Nov 02 06:32:04.283734 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:04.283702 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8475f4f7-p54s9" Nov 02 06:32:04.654191 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:04.654155 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8475f4f7-p54s9"] Nov 02 06:32:04.659538 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:32:04.659508 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc833f51_60f9_4e3c_8c15_1d0db873eea5.slice/crio-e7b6ea5166dc03c875b43defe80910c58f1d91745eb623c1b47f3af38e1901ee WatchSource:0}: Error finding container e7b6ea5166dc03c875b43defe80910c58f1d91745eb623c1b47f3af38e1901ee: Status 404 returned error can't find the container with id e7b6ea5166dc03c875b43defe80910c58f1d91745eb623c1b47f3af38e1901ee Nov 02 06:32:04.821300 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:04.821266 5916 generic.go:356] "Generic (PLEG): container finished" podID="e308286b-3f6e-4950-9605-daa0d1d68b5d" containerID="ec69be554402404194189fa794deae4d5f190aa498b36f57b88ae236feabfdc9" exitCode=0 Nov 02 06:32:04.821480 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:04.821345 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38e2d5b3-9qdrh" event={"ID":"e308286b-3f6e-4950-9605-daa0d1d68b5d","Type":"ContainerDied","Data":"ec69be554402404194189fa794deae4d5f190aa498b36f57b88ae236feabfdc9"} Nov 02 06:32:04.822594 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:04.822572 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8475f4f7-p54s9" event={"ID":"bc833f51-60f9-4e3c-8c15-1d0db873eea5","Type":"ContainerStarted","Data":"e7b6ea5166dc03c875b43defe80910c58f1d91745eb623c1b47f3af38e1901ee"} Nov 02 06:32:05.831978 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:05.831877 5916 generic.go:356] "Generic (PLEG): container finished" podID="bc833f51-60f9-4e3c-8c15-1d0db873eea5" containerID="fadb7833ba0092324d5ea7df340ac1f4a989404522061fab3c1dccf120a7977b" exitCode=0 Nov 02 06:32:05.831978 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:05.831941 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8475f4f7-p54s9" event={"ID":"bc833f51-60f9-4e3c-8c15-1d0db873eea5","Type":"ContainerDied","Data":"fadb7833ba0092324d5ea7df340ac1f4a989404522061fab3c1dccf120a7977b"} Nov 02 06:32:05.982696 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:05.982671 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38e2d5b3-9qdrh" Nov 02 06:32:06.117820 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:06.117739 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e308286b-3f6e-4950-9605-daa0d1d68b5d-kube-api-access-kf58g\" (UniqueName: \"kubernetes.io/projected/e308286b-3f6e-4950-9605-daa0d1d68b5d-kube-api-access-kf58g\") pod \"e308286b-3f6e-4950-9605-daa0d1d68b5d\" (UID: \"e308286b-3f6e-4950-9605-daa0d1d68b5d\") " Nov 02 06:32:06.117820 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:06.117791 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e308286b-3f6e-4950-9605-daa0d1d68b5d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e308286b-3f6e-4950-9605-daa0d1d68b5d-maas-api-xks-ca\") pod \"e308286b-3f6e-4950-9605-daa0d1d68b5d\" (UID: \"e308286b-3f6e-4950-9605-daa0d1d68b5d\") " Nov 02 06:32:06.118056 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:06.117960 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e308286b-3f6e-4950-9605-daa0d1d68b5d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e308286b-3f6e-4950-9605-daa0d1d68b5d-maas-api-openshift-ca\") pod \"e308286b-3f6e-4950-9605-daa0d1d68b5d\" (UID: \"e308286b-3f6e-4950-9605-daa0d1d68b5d\") " Nov 02 06:32:06.118313 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:06.118289 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e308286b-3f6e-4950-9605-daa0d1d68b5d-maas-api-openshift-ca" pod "e308286b-3f6e-4950-9605-daa0d1d68b5d" (UID: "e308286b-3f6e-4950-9605-daa0d1d68b5d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:32:06.118425 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:06.118410 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e308286b-3f6e-4950-9605-daa0d1d68b5d-maas-api-openshift-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:32:06.119943 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:06.119922 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e308286b-3f6e-4950-9605-daa0d1d68b5d-maas-api-xks-ca" pod "e308286b-3f6e-4950-9605-daa0d1d68b5d" (UID: "e308286b-3f6e-4950-9605-daa0d1d68b5d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:32:06.120398 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:06.120372 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e308286b-3f6e-4950-9605-daa0d1d68b5d-kube-api-access-kf58g" pod "e308286b-3f6e-4950-9605-daa0d1d68b5d" (UID: "e308286b-3f6e-4950-9605-daa0d1d68b5d"). InnerVolumeSpecName "kube-api-access-kf58g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:32:06.219359 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:06.219329 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kf58g\" (UniqueName: \"kubernetes.io/projected/e308286b-3f6e-4950-9605-daa0d1d68b5d-kube-api-access-kf58g\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:32:06.219359 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:06.219356 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e308286b-3f6e-4950-9605-daa0d1d68b5d-maas-api-xks-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:32:06.841230 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:06.841201 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38e2d5b3-9qdrh" Nov 02 06:32:06.841230 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:06.841197 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38e2d5b3-9qdrh" event={"ID":"e308286b-3f6e-4950-9605-daa0d1d68b5d","Type":"ContainerDied","Data":"e69710c09fe2534f43876029494270ab983ee0f1d5b646c1476b9902940396fd"} Nov 02 06:32:06.841677 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:06.841247 5916 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e69710c09fe2534f43876029494270ab983ee0f1d5b646c1476b9902940396fd" Nov 02 06:32:06.995562 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:06.995518 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8475f4f7-p54s9" Nov 02 06:32:07.128156 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:07.128075 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc833f51-60f9-4e3c-8c15-1d0db873eea5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc833f51-60f9-4e3c-8c15-1d0db873eea5-maas-api-xks-ca\") pod \"bc833f51-60f9-4e3c-8c15-1d0db873eea5\" (UID: \"bc833f51-60f9-4e3c-8c15-1d0db873eea5\") " Nov 02 06:32:07.128156 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:07.128129 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc833f51-60f9-4e3c-8c15-1d0db873eea5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc833f51-60f9-4e3c-8c15-1d0db873eea5-maas-api-openshift-ca\") pod \"bc833f51-60f9-4e3c-8c15-1d0db873eea5\" (UID: \"bc833f51-60f9-4e3c-8c15-1d0db873eea5\") " Nov 02 06:32:07.128352 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:07.128177 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc833f51-60f9-4e3c-8c15-1d0db873eea5-kube-api-access-t75p6\" (UniqueName: \"kubernetes.io/projected/bc833f51-60f9-4e3c-8c15-1d0db873eea5-kube-api-access-t75p6\") pod \"bc833f51-60f9-4e3c-8c15-1d0db873eea5\" (UID: \"bc833f51-60f9-4e3c-8c15-1d0db873eea5\") " Nov 02 06:32:07.128606 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:07.128576 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc833f51-60f9-4e3c-8c15-1d0db873eea5-maas-api-openshift-ca" pod "bc833f51-60f9-4e3c-8c15-1d0db873eea5" (UID: "bc833f51-60f9-4e3c-8c15-1d0db873eea5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:32:07.130306 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:07.130281 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc833f51-60f9-4e3c-8c15-1d0db873eea5-maas-api-xks-ca" pod "bc833f51-60f9-4e3c-8c15-1d0db873eea5" (UID: "bc833f51-60f9-4e3c-8c15-1d0db873eea5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:32:07.130414 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:07.130344 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc833f51-60f9-4e3c-8c15-1d0db873eea5-kube-api-access-t75p6" pod "bc833f51-60f9-4e3c-8c15-1d0db873eea5" (UID: "bc833f51-60f9-4e3c-8c15-1d0db873eea5"). InnerVolumeSpecName "kube-api-access-t75p6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:32:07.229392 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:07.229352 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t75p6\" (UniqueName: \"kubernetes.io/projected/bc833f51-60f9-4e3c-8c15-1d0db873eea5-kube-api-access-t75p6\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:32:07.229392 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:07.229384 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc833f51-60f9-4e3c-8c15-1d0db873eea5-maas-api-xks-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:32:07.229392 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:07.229394 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc833f51-60f9-4e3c-8c15-1d0db873eea5-maas-api-openshift-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:32:07.851440 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:07.851405 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8475f4f7-p54s9" event={"ID":"bc833f51-60f9-4e3c-8c15-1d0db873eea5","Type":"ContainerDied","Data":"e7b6ea5166dc03c875b43defe80910c58f1d91745eb623c1b47f3af38e1901ee"} Nov 02 06:32:07.851829 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:07.851462 5916 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e7b6ea5166dc03c875b43defe80910c58f1d91745eb623c1b47f3af38e1901ee" Nov 02 06:32:07.851829 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:07.851529 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8475f4f7-p54s9" Nov 02 06:32:17.710545 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:17.710511 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8684e79-gjw67"] Nov 02 06:32:17.711780 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:17.711758 5916 cpu_manager.go:414] "containerMap: removing container" podUID="bc833f51-60f9-4e3c-8c15-1d0db873eea5" containerName="revoke-keys" Nov 02 06:32:17.711918 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:17.711885 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc833f51-60f9-4e3c-8c15-1d0db873eea5" containerName="revoke-keys" Nov 02 06:32:17.712031 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:17.711928 5916 cpu_manager.go:414] "containerMap: removing container" podUID="e308286b-3f6e-4950-9605-daa0d1d68b5d" containerName="revoke-keys" Nov 02 06:32:17.712031 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:17.711937 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e308286b-3f6e-4950-9605-daa0d1d68b5d" containerName="revoke-keys" Nov 02 06:32:17.712127 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:17.712116 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc833f51-60f9-4e3c-8c15-1d0db873eea5" containerName="revoke-keys" Nov 02 06:32:17.712168 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:17.712131 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="e308286b-3f6e-4950-9605-daa0d1d68b5d" containerName="revoke-keys" Nov 02 06:32:17.716177 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:17.716156 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8684e79-gjw67" Nov 02 06:32:17.768431 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:17.768397 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 06:32:17.814516 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:17.814474 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-68l4r\"" Nov 02 06:32:17.817397 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:17.817370 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8684e79-gjw67"] Nov 02 06:32:17.835779 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:17.835750 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88aedb5c-54d5-4c77-a228-2064e91e9228-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a8684e79-gjw67\" (UID: \"88aedb5c-54d5-4c77-a228-2064e91e9228\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8684e79-gjw67" Nov 02 06:32:17.835945 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:17.835805 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtr79\" (UniqueName: \"kubernetes.io/projected/88aedb5c-54d5-4c77-a228-2064e91e9228-kube-api-access-vtr79\") pod \"maas-api-revoke-sub-a8684e79-gjw67\" (UID: \"88aedb5c-54d5-4c77-a228-2064e91e9228\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8684e79-gjw67" Nov 02 06:32:17.835945 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:17.835833 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88aedb5c-54d5-4c77-a228-2064e91e9228-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a8684e79-gjw67\" (UID: \"88aedb5c-54d5-4c77-a228-2064e91e9228\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8684e79-gjw67" Nov 02 06:32:17.936957 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:17.936920 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88aedb5c-54d5-4c77-a228-2064e91e9228-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a8684e79-gjw67\" (UID: \"88aedb5c-54d5-4c77-a228-2064e91e9228\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8684e79-gjw67" Nov 02 06:32:17.937159 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:17.936995 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtr79\" (UniqueName: \"kubernetes.io/projected/88aedb5c-54d5-4c77-a228-2064e91e9228-kube-api-access-vtr79\") pod \"maas-api-revoke-sub-a8684e79-gjw67\" (UID: \"88aedb5c-54d5-4c77-a228-2064e91e9228\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8684e79-gjw67" Nov 02 06:32:17.937159 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:17.937015 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88aedb5c-54d5-4c77-a228-2064e91e9228-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a8684e79-gjw67\" (UID: \"88aedb5c-54d5-4c77-a228-2064e91e9228\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8684e79-gjw67" Nov 02 06:32:17.937831 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:17.937809 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88aedb5c-54d5-4c77-a228-2064e91e9228-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a8684e79-gjw67\" (UID: \"88aedb5c-54d5-4c77-a228-2064e91e9228\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8684e79-gjw67" Nov 02 06:32:17.939420 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:17.939394 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88aedb5c-54d5-4c77-a228-2064e91e9228-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a8684e79-gjw67\" (UID: \"88aedb5c-54d5-4c77-a228-2064e91e9228\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8684e79-gjw67" Nov 02 06:32:17.993974 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:17.993529 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtr79\" (UniqueName: \"kubernetes.io/projected/88aedb5c-54d5-4c77-a228-2064e91e9228-kube-api-access-vtr79\") pod \"maas-api-revoke-sub-a8684e79-gjw67\" (UID: \"88aedb5c-54d5-4c77-a228-2064e91e9228\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8684e79-gjw67" Nov 02 06:32:18.031429 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:18.031394 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8684e79-gjw67" Nov 02 06:32:18.317621 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:18.317538 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b66e4e9-hpz5x"] Nov 02 06:32:18.338761 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:18.327980 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b66e4e9-hpz5x" Nov 02 06:32:18.374169 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:18.374136 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b66e4e9-hpz5x"] Nov 02 06:32:18.426624 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:18.426588 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8684e79-gjw67"] Nov 02 06:32:18.432263 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:32:18.432232 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88aedb5c_54d5_4c77_a228_2064e91e9228.slice/crio-61f514adb00e69f0070a298027299baa22c6351775fb17d9e1e57a318d3be836 WatchSource:0}: Error finding container 61f514adb00e69f0070a298027299baa22c6351775fb17d9e1e57a318d3be836: Status 404 returned error can't find the container with id 61f514adb00e69f0070a298027299baa22c6351775fb17d9e1e57a318d3be836 Nov 02 06:32:18.445381 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:18.445354 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f950069-ac4b-4c86-ab48-eb0496c385f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2b66e4e9-hpz5x\" (UID: \"2f950069-ac4b-4c86-ab48-eb0496c385f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b66e4e9-hpz5x" Nov 02 06:32:18.445578 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:18.445554 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f950069-ac4b-4c86-ab48-eb0496c385f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2b66e4e9-hpz5x\" (UID: \"2f950069-ac4b-4c86-ab48-eb0496c385f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b66e4e9-hpz5x" Nov 02 06:32:18.445643 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:18.445630 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z99q7\" (UniqueName: \"kubernetes.io/projected/2f950069-ac4b-4c86-ab48-eb0496c385f5-kube-api-access-z99q7\") pod \"maas-api-revoke-sub-2b66e4e9-hpz5x\" (UID: \"2f950069-ac4b-4c86-ab48-eb0496c385f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b66e4e9-hpz5x" Nov 02 06:32:18.547224 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:18.547190 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z99q7\" (UniqueName: \"kubernetes.io/projected/2f950069-ac4b-4c86-ab48-eb0496c385f5-kube-api-access-z99q7\") pod \"maas-api-revoke-sub-2b66e4e9-hpz5x\" (UID: \"2f950069-ac4b-4c86-ab48-eb0496c385f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b66e4e9-hpz5x" Nov 02 06:32:18.547392 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:18.547249 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f950069-ac4b-4c86-ab48-eb0496c385f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2b66e4e9-hpz5x\" (UID: \"2f950069-ac4b-4c86-ab48-eb0496c385f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b66e4e9-hpz5x" Nov 02 06:32:18.547392 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:18.547356 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f950069-ac4b-4c86-ab48-eb0496c385f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2b66e4e9-hpz5x\" (UID: \"2f950069-ac4b-4c86-ab48-eb0496c385f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b66e4e9-hpz5x" Nov 02 06:32:18.547993 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:18.547969 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f950069-ac4b-4c86-ab48-eb0496c385f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2b66e4e9-hpz5x\" (UID: \"2f950069-ac4b-4c86-ab48-eb0496c385f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b66e4e9-hpz5x" Nov 02 06:32:18.549660 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:18.549641 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f950069-ac4b-4c86-ab48-eb0496c385f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2b66e4e9-hpz5x\" (UID: \"2f950069-ac4b-4c86-ab48-eb0496c385f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b66e4e9-hpz5x" Nov 02 06:32:18.600880 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:18.600801 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z99q7\" (UniqueName: \"kubernetes.io/projected/2f950069-ac4b-4c86-ab48-eb0496c385f5-kube-api-access-z99q7\") pod \"maas-api-revoke-sub-2b66e4e9-hpz5x\" (UID: \"2f950069-ac4b-4c86-ab48-eb0496c385f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b66e4e9-hpz5x" Nov 02 06:32:18.647785 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:18.647752 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b66e4e9-hpz5x" Nov 02 06:32:18.924463 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:18.924426 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8684e79-gjw67" event={"ID":"88aedb5c-54d5-4c77-a228-2064e91e9228","Type":"ContainerStarted","Data":"6642695f0202413e2ddb181b9e21d3ecd366fc0d5b4023eb6af7564f6a89c7fc"} Nov 02 06:32:18.924878 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:18.924490 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8684e79-gjw67" event={"ID":"88aedb5c-54d5-4c77-a228-2064e91e9228","Type":"ContainerStarted","Data":"61f514adb00e69f0070a298027299baa22c6351775fb17d9e1e57a318d3be836"} Nov 02 06:32:19.032688 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:19.032633 5916 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8684e79-gjw67" podStartSLOduration=2.032618765 podStartE2EDuration="2.032618765s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:32:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:32:19.030473516 +0000 UTC m=+927.302293910" watchObservedRunningTime="2026-11-02 06:32:19.032618765 +0000 UTC m=+927.304439190" Nov 02 06:32:19.051333 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:19.051301 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b66e4e9-hpz5x"] Nov 02 06:32:19.056736 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:32:19.056702 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f950069_ac4b_4c86_ab48_eb0496c385f5.slice/crio-8998778c0b959c6e7dab32fa302cd0fa54f39ae68268200d23cba15a0de663a9 WatchSource:0}: Error finding container 8998778c0b959c6e7dab32fa302cd0fa54f39ae68268200d23cba15a0de663a9: Status 404 returned error can't find the container with id 8998778c0b959c6e7dab32fa302cd0fa54f39ae68268200d23cba15a0de663a9 Nov 02 06:32:19.898773 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:32:19.898740 5916 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p98cq" containerName="extractor" Nov 02 06:32:19.932315 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:19.932276 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b66e4e9-hpz5x" event={"ID":"2f950069-ac4b-4c86-ab48-eb0496c385f5","Type":"ContainerStarted","Data":"e852ba9a877b46c96e7e512bec2fee1de3f6e2c1a1504cfcaeab5d7ccf65e1ef"} Nov 02 06:32:19.932794 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:19.932343 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b66e4e9-hpz5x" event={"ID":"2f950069-ac4b-4c86-ab48-eb0496c385f5","Type":"ContainerStarted","Data":"8998778c0b959c6e7dab32fa302cd0fa54f39ae68268200d23cba15a0de663a9"} Nov 02 06:32:19.934050 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:19.934020 5916 generic.go:356] "Generic (PLEG): container finished" podID="88aedb5c-54d5-4c77-a228-2064e91e9228" containerID="6642695f0202413e2ddb181b9e21d3ecd366fc0d5b4023eb6af7564f6a89c7fc" exitCode=0 Nov 02 06:32:19.934174 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:19.934080 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8684e79-gjw67" event={"ID":"88aedb5c-54d5-4c77-a228-2064e91e9228","Type":"ContainerDied","Data":"6642695f0202413e2ddb181b9e21d3ecd366fc0d5b4023eb6af7564f6a89c7fc"} Nov 02 06:32:20.039629 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:20.039581 5916 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b66e4e9-hpz5x" podStartSLOduration=2.039567601 podStartE2EDuration="2.039567601s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:32:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:32:20.037851525 +0000 UTC m=+928.309671918" watchObservedRunningTime="2026-11-02 06:32:20.039567601 +0000 UTC m=+928.311387995" Nov 02 06:32:20.943337 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:20.943300 5916 generic.go:356] "Generic (PLEG): container finished" podID="2f950069-ac4b-4c86-ab48-eb0496c385f5" containerID="e852ba9a877b46c96e7e512bec2fee1de3f6e2c1a1504cfcaeab5d7ccf65e1ef" exitCode=0 Nov 02 06:32:20.943817 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:20.943395 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b66e4e9-hpz5x" event={"ID":"2f950069-ac4b-4c86-ab48-eb0496c385f5","Type":"ContainerDied","Data":"e852ba9a877b46c96e7e512bec2fee1de3f6e2c1a1504cfcaeab5d7ccf65e1ef"} Nov 02 06:32:21.113184 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:21.113157 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8684e79-gjw67" Nov 02 06:32:21.280180 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:21.280095 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88aedb5c-54d5-4c77-a228-2064e91e9228-kube-api-access-vtr79\" (UniqueName: \"kubernetes.io/projected/88aedb5c-54d5-4c77-a228-2064e91e9228-kube-api-access-vtr79\") pod \"88aedb5c-54d5-4c77-a228-2064e91e9228\" (UID: \"88aedb5c-54d5-4c77-a228-2064e91e9228\") " Nov 02 06:32:21.280180 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:21.280177 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88aedb5c-54d5-4c77-a228-2064e91e9228-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88aedb5c-54d5-4c77-a228-2064e91e9228-maas-api-openshift-ca\") pod \"88aedb5c-54d5-4c77-a228-2064e91e9228\" (UID: \"88aedb5c-54d5-4c77-a228-2064e91e9228\") " Nov 02 06:32:21.280405 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:21.280286 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/88aedb5c-54d5-4c77-a228-2064e91e9228-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88aedb5c-54d5-4c77-a228-2064e91e9228-maas-api-xks-ca\") pod \"88aedb5c-54d5-4c77-a228-2064e91e9228\" (UID: \"88aedb5c-54d5-4c77-a228-2064e91e9228\") " Nov 02 06:32:21.280562 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:21.280525 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88aedb5c-54d5-4c77-a228-2064e91e9228-maas-api-openshift-ca" pod "88aedb5c-54d5-4c77-a228-2064e91e9228" (UID: "88aedb5c-54d5-4c77-a228-2064e91e9228"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:32:21.282335 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:21.282311 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88aedb5c-54d5-4c77-a228-2064e91e9228-maas-api-xks-ca" pod "88aedb5c-54d5-4c77-a228-2064e91e9228" (UID: "88aedb5c-54d5-4c77-a228-2064e91e9228"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:32:21.282335 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:21.282308 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88aedb5c-54d5-4c77-a228-2064e91e9228-kube-api-access-vtr79" pod "88aedb5c-54d5-4c77-a228-2064e91e9228" (UID: "88aedb5c-54d5-4c77-a228-2064e91e9228"). InnerVolumeSpecName "kube-api-access-vtr79". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:32:21.381813 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:21.381779 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88aedb5c-54d5-4c77-a228-2064e91e9228-maas-api-xks-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:32:21.381813 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:21.381807 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vtr79\" (UniqueName: \"kubernetes.io/projected/88aedb5c-54d5-4c77-a228-2064e91e9228-kube-api-access-vtr79\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:32:21.381813 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:21.381821 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88aedb5c-54d5-4c77-a228-2064e91e9228-maas-api-openshift-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:32:21.953305 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:21.953270 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8684e79-gjw67" Nov 02 06:32:21.953305 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:21.953298 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8684e79-gjw67" event={"ID":"88aedb5c-54d5-4c77-a228-2064e91e9228","Type":"ContainerDied","Data":"61f514adb00e69f0070a298027299baa22c6351775fb17d9e1e57a318d3be836"} Nov 02 06:32:21.953858 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:21.953353 5916 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="61f514adb00e69f0070a298027299baa22c6351775fb17d9e1e57a318d3be836" Nov 02 06:32:22.082658 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:22.082623 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-184131a0-djz9z"] Nov 02 06:32:22.083355 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:22.083335 5916 cpu_manager.go:414] "containerMap: removing container" podUID="88aedb5c-54d5-4c77-a228-2064e91e9228" containerName="revoke-keys" Nov 02 06:32:22.083355 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:22.083353 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88aedb5c-54d5-4c77-a228-2064e91e9228" containerName="revoke-keys" Nov 02 06:32:22.083508 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:22.083486 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="88aedb5c-54d5-4c77-a228-2064e91e9228" containerName="revoke-keys" Nov 02 06:32:22.088698 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:22.087493 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-184131a0-djz9z" Nov 02 06:32:22.147158 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:22.141377 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-184131a0-djz9z"] Nov 02 06:32:22.147323 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:22.147303 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b66e4e9-hpz5x" Nov 02 06:32:22.189157 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:22.189126 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kfw4\" (UniqueName: \"kubernetes.io/projected/1ae95479-786b-4854-987c-d5cf22401220-kube-api-access-2kfw4\") pod \"maas-api-revoke-sub-184131a0-djz9z\" (UID: \"1ae95479-786b-4854-987c-d5cf22401220\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-184131a0-djz9z" Nov 02 06:32:22.189334 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:22.189254 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ae95479-786b-4854-987c-d5cf22401220-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-184131a0-djz9z\" (UID: \"1ae95479-786b-4854-987c-d5cf22401220\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-184131a0-djz9z" Nov 02 06:32:22.189334 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:22.189303 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ae95479-786b-4854-987c-d5cf22401220-maas-api-xks-ca\") pod \"maas-api-revoke-sub-184131a0-djz9z\" (UID: \"1ae95479-786b-4854-987c-d5cf22401220\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-184131a0-djz9z" Nov 02 06:32:22.290223 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:22.290187 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f950069-ac4b-4c86-ab48-eb0496c385f5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f950069-ac4b-4c86-ab48-eb0496c385f5-maas-api-xks-ca\") pod \"2f950069-ac4b-4c86-ab48-eb0496c385f5\" (UID: \"2f950069-ac4b-4c86-ab48-eb0496c385f5\") " Nov 02 06:32:22.290408 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:22.290305 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f950069-ac4b-4c86-ab48-eb0496c385f5-kube-api-access-z99q7\" (UniqueName: \"kubernetes.io/projected/2f950069-ac4b-4c86-ab48-eb0496c385f5-kube-api-access-z99q7\") pod \"2f950069-ac4b-4c86-ab48-eb0496c385f5\" (UID: \"2f950069-ac4b-4c86-ab48-eb0496c385f5\") " Nov 02 06:32:22.290408 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:22.290379 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f950069-ac4b-4c86-ab48-eb0496c385f5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f950069-ac4b-4c86-ab48-eb0496c385f5-maas-api-openshift-ca\") pod \"2f950069-ac4b-4c86-ab48-eb0496c385f5\" (UID: \"2f950069-ac4b-4c86-ab48-eb0496c385f5\") " Nov 02 06:32:22.290566 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:22.290545 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ae95479-786b-4854-987c-d5cf22401220-maas-api-xks-ca\") pod \"maas-api-revoke-sub-184131a0-djz9z\" (UID: \"1ae95479-786b-4854-987c-d5cf22401220\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-184131a0-djz9z" Nov 02 06:32:22.290708 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:22.290623 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kfw4\" (UniqueName: \"kubernetes.io/projected/1ae95479-786b-4854-987c-d5cf22401220-kube-api-access-2kfw4\") pod \"maas-api-revoke-sub-184131a0-djz9z\" (UID: \"1ae95479-786b-4854-987c-d5cf22401220\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-184131a0-djz9z" Nov 02 06:32:22.290768 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:22.290725 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f950069-ac4b-4c86-ab48-eb0496c385f5-maas-api-openshift-ca" pod "2f950069-ac4b-4c86-ab48-eb0496c385f5" (UID: "2f950069-ac4b-4c86-ab48-eb0496c385f5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:32:22.290768 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:22.290735 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ae95479-786b-4854-987c-d5cf22401220-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-184131a0-djz9z\" (UID: \"1ae95479-786b-4854-987c-d5cf22401220\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-184131a0-djz9z" Nov 02 06:32:22.290882 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:22.290822 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f950069-ac4b-4c86-ab48-eb0496c385f5-maas-api-openshift-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:32:22.291470 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:22.291447 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ae95479-786b-4854-987c-d5cf22401220-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-184131a0-djz9z\" (UID: \"1ae95479-786b-4854-987c-d5cf22401220\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-184131a0-djz9z" Nov 02 06:32:22.292487 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:22.292464 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f950069-ac4b-4c86-ab48-eb0496c385f5-maas-api-xks-ca" pod "2f950069-ac4b-4c86-ab48-eb0496c385f5" (UID: "2f950069-ac4b-4c86-ab48-eb0496c385f5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:32:22.292594 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:22.292517 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f950069-ac4b-4c86-ab48-eb0496c385f5-kube-api-access-z99q7" pod "2f950069-ac4b-4c86-ab48-eb0496c385f5" (UID: "2f950069-ac4b-4c86-ab48-eb0496c385f5"). InnerVolumeSpecName "kube-api-access-z99q7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:32:22.293168 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:22.293149 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ae95479-786b-4854-987c-d5cf22401220-maas-api-xks-ca\") pod \"maas-api-revoke-sub-184131a0-djz9z\" (UID: \"1ae95479-786b-4854-987c-d5cf22401220\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-184131a0-djz9z" Nov 02 06:32:22.346957 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:22.346171 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kfw4\" (UniqueName: \"kubernetes.io/projected/1ae95479-786b-4854-987c-d5cf22401220-kube-api-access-2kfw4\") pod \"maas-api-revoke-sub-184131a0-djz9z\" (UID: \"1ae95479-786b-4854-987c-d5cf22401220\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-184131a0-djz9z" Nov 02 06:32:22.396389 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:22.396351 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z99q7\" (UniqueName: \"kubernetes.io/projected/2f950069-ac4b-4c86-ab48-eb0496c385f5-kube-api-access-z99q7\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:32:22.396389 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:22.396391 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f950069-ac4b-4c86-ab48-eb0496c385f5-maas-api-xks-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:32:22.449544 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:22.449516 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-184131a0-djz9z" Nov 02 06:32:22.826355 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:22.826269 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-184131a0-djz9z"] Nov 02 06:32:22.831953 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:32:22.831922 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ae95479_786b_4854_987c_d5cf22401220.slice/crio-9f09fe8cc43e0fdba1057534a05a9dfbed69dad8c3e73a75f3496b80b828261e WatchSource:0}: Error finding container 9f09fe8cc43e0fdba1057534a05a9dfbed69dad8c3e73a75f3496b80b828261e: Status 404 returned error can't find the container with id 9f09fe8cc43e0fdba1057534a05a9dfbed69dad8c3e73a75f3496b80b828261e Nov 02 06:32:22.964221 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:22.964179 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-184131a0-djz9z" event={"ID":"1ae95479-786b-4854-987c-d5cf22401220","Type":"ContainerStarted","Data":"9f09fe8cc43e0fdba1057534a05a9dfbed69dad8c3e73a75f3496b80b828261e"} Nov 02 06:32:22.966727 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:22.966698 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b66e4e9-hpz5x" event={"ID":"2f950069-ac4b-4c86-ab48-eb0496c385f5","Type":"ContainerDied","Data":"8998778c0b959c6e7dab32fa302cd0fa54f39ae68268200d23cba15a0de663a9"} Nov 02 06:32:22.966894 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:22.966747 5916 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8998778c0b959c6e7dab32fa302cd0fa54f39ae68268200d23cba15a0de663a9" Nov 02 06:32:22.966894 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:22.966752 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b66e4e9-hpz5x" Nov 02 06:32:23.976625 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:23.976586 5916 generic.go:356] "Generic (PLEG): container finished" podID="1ae95479-786b-4854-987c-d5cf22401220" containerID="4e77001acde5fbe21c1888db1488f3ce8c25bcc4839153ba714547d7bc1020f7" exitCode=0 Nov 02 06:32:23.977145 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:23.976660 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-184131a0-djz9z" event={"ID":"1ae95479-786b-4854-987c-d5cf22401220","Type":"ContainerDied","Data":"4e77001acde5fbe21c1888db1488f3ce8c25bcc4839153ba714547d7bc1020f7"} Nov 02 06:32:25.126604 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:25.126581 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-184131a0-djz9z" Nov 02 06:32:25.227674 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:25.226986 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ae95479-786b-4854-987c-d5cf22401220-kube-api-access-2kfw4\" (UniqueName: \"kubernetes.io/projected/1ae95479-786b-4854-987c-d5cf22401220-kube-api-access-2kfw4\") pod \"1ae95479-786b-4854-987c-d5cf22401220\" (UID: \"1ae95479-786b-4854-987c-d5cf22401220\") " Nov 02 06:32:25.227674 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:25.227121 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ae95479-786b-4854-987c-d5cf22401220-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ae95479-786b-4854-987c-d5cf22401220-maas-api-xks-ca\") pod \"1ae95479-786b-4854-987c-d5cf22401220\" (UID: \"1ae95479-786b-4854-987c-d5cf22401220\") " Nov 02 06:32:25.227674 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:25.227176 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ae95479-786b-4854-987c-d5cf22401220-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ae95479-786b-4854-987c-d5cf22401220-maas-api-openshift-ca\") pod \"1ae95479-786b-4854-987c-d5cf22401220\" (UID: \"1ae95479-786b-4854-987c-d5cf22401220\") " Nov 02 06:32:25.234663 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:25.233928 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ae95479-786b-4854-987c-d5cf22401220-maas-api-openshift-ca" pod "1ae95479-786b-4854-987c-d5cf22401220" (UID: "1ae95479-786b-4854-987c-d5cf22401220"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:32:25.236322 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:25.236218 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ae95479-786b-4854-987c-d5cf22401220-maas-api-openshift-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:32:25.239018 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:25.238983 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ae95479-786b-4854-987c-d5cf22401220-maas-api-xks-ca" pod "1ae95479-786b-4854-987c-d5cf22401220" (UID: "1ae95479-786b-4854-987c-d5cf22401220"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:32:25.246720 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:25.243178 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ae95479-786b-4854-987c-d5cf22401220-kube-api-access-2kfw4" pod "1ae95479-786b-4854-987c-d5cf22401220" (UID: "1ae95479-786b-4854-987c-d5cf22401220"). InnerVolumeSpecName "kube-api-access-2kfw4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:32:25.337366 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:25.337329 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2kfw4\" (UniqueName: \"kubernetes.io/projected/1ae95479-786b-4854-987c-d5cf22401220-kube-api-access-2kfw4\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:32:25.337366 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:25.337364 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ae95479-786b-4854-987c-d5cf22401220-maas-api-xks-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:32:25.995550 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:25.995516 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-184131a0-djz9z" Nov 02 06:32:25.995756 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:25.995543 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-184131a0-djz9z" event={"ID":"1ae95479-786b-4854-987c-d5cf22401220","Type":"ContainerDied","Data":"9f09fe8cc43e0fdba1057534a05a9dfbed69dad8c3e73a75f3496b80b828261e"} Nov 02 06:32:25.995756 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:25.995602 5916 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9f09fe8cc43e0fdba1057534a05a9dfbed69dad8c3e73a75f3496b80b828261e" Nov 02 06:32:29.036376 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:29.036336 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e2c401b-drbp6"] Nov 02 06:32:29.037114 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:29.037094 5916 cpu_manager.go:414] "containerMap: removing container" podUID="2f950069-ac4b-4c86-ab48-eb0496c385f5" containerName="revoke-keys" Nov 02 06:32:29.037202 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:29.037116 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f950069-ac4b-4c86-ab48-eb0496c385f5" containerName="revoke-keys" Nov 02 06:32:29.037202 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:29.037127 5916 cpu_manager.go:414] "containerMap: removing container" podUID="1ae95479-786b-4854-987c-d5cf22401220" containerName="revoke-keys" Nov 02 06:32:29.037202 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:29.037135 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ae95479-786b-4854-987c-d5cf22401220" containerName="revoke-keys" Nov 02 06:32:29.037355 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:29.037276 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f950069-ac4b-4c86-ab48-eb0496c385f5" containerName="revoke-keys" Nov 02 06:32:29.037355 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:29.037288 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ae95479-786b-4854-987c-d5cf22401220" containerName="revoke-keys" Nov 02 06:32:29.041885 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:29.041863 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e2c401b-drbp6" Nov 02 06:32:29.076463 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:29.076434 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12e9b5d0-00a7-4b96-9a9a-d403c5bae27c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e2c401b-drbp6\" (UID: \"12e9b5d0-00a7-4b96-9a9a-d403c5bae27c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e2c401b-drbp6" Nov 02 06:32:29.076599 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:29.076479 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72mgs\" (UniqueName: \"kubernetes.io/projected/12e9b5d0-00a7-4b96-9a9a-d403c5bae27c-kube-api-access-72mgs\") pod \"maas-api-revoke-sub-6e2c401b-drbp6\" (UID: \"12e9b5d0-00a7-4b96-9a9a-d403c5bae27c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e2c401b-drbp6" Nov 02 06:32:29.076641 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:29.076607 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12e9b5d0-00a7-4b96-9a9a-d403c5bae27c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e2c401b-drbp6\" (UID: \"12e9b5d0-00a7-4b96-9a9a-d403c5bae27c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e2c401b-drbp6" Nov 02 06:32:29.091671 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:29.091643 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e2c401b-drbp6"] Nov 02 06:32:29.134029 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:29.134002 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 06:32:29.134175 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:29.134070 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-68l4r\"" Nov 02 06:32:29.178020 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:29.177987 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12e9b5d0-00a7-4b96-9a9a-d403c5bae27c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e2c401b-drbp6\" (UID: \"12e9b5d0-00a7-4b96-9a9a-d403c5bae27c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e2c401b-drbp6" Nov 02 06:32:29.178196 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:29.178046 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72mgs\" (UniqueName: \"kubernetes.io/projected/12e9b5d0-00a7-4b96-9a9a-d403c5bae27c-kube-api-access-72mgs\") pod \"maas-api-revoke-sub-6e2c401b-drbp6\" (UID: \"12e9b5d0-00a7-4b96-9a9a-d403c5bae27c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e2c401b-drbp6" Nov 02 06:32:29.178196 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:29.178129 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12e9b5d0-00a7-4b96-9a9a-d403c5bae27c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e2c401b-drbp6\" (UID: \"12e9b5d0-00a7-4b96-9a9a-d403c5bae27c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e2c401b-drbp6" Nov 02 06:32:29.178785 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:29.178762 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12e9b5d0-00a7-4b96-9a9a-d403c5bae27c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e2c401b-drbp6\" (UID: \"12e9b5d0-00a7-4b96-9a9a-d403c5bae27c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e2c401b-drbp6" Nov 02 06:32:29.180355 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:29.180336 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12e9b5d0-00a7-4b96-9a9a-d403c5bae27c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e2c401b-drbp6\" (UID: \"12e9b5d0-00a7-4b96-9a9a-d403c5bae27c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e2c401b-drbp6" Nov 02 06:32:29.231508 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:29.231478 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72mgs\" (UniqueName: \"kubernetes.io/projected/12e9b5d0-00a7-4b96-9a9a-d403c5bae27c-kube-api-access-72mgs\") pod \"maas-api-revoke-sub-6e2c401b-drbp6\" (UID: \"12e9b5d0-00a7-4b96-9a9a-d403c5bae27c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e2c401b-drbp6" Nov 02 06:32:29.358947 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:29.358846 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e2c401b-drbp6" Nov 02 06:32:29.739781 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:29.739748 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e2c401b-drbp6"] Nov 02 06:32:29.744853 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:32:29.744823 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12e9b5d0_00a7_4b96_9a9a_d403c5bae27c.slice/crio-604a57cc180e81d1009e9dc9c084a65054cdbe4f6e7d447f36e31af92f816daf WatchSource:0}: Error finding container 604a57cc180e81d1009e9dc9c084a65054cdbe4f6e7d447f36e31af92f816daf: Status 404 returned error can't find the container with id 604a57cc180e81d1009e9dc9c084a65054cdbe4f6e7d447f36e31af92f816daf Nov 02 06:32:30.031192 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:30.031160 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e2c401b-drbp6" event={"ID":"12e9b5d0-00a7-4b96-9a9a-d403c5bae27c","Type":"ContainerStarted","Data":"604a57cc180e81d1009e9dc9c084a65054cdbe4f6e7d447f36e31af92f816daf"} Nov 02 06:32:31.040802 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:31.040768 5916 generic.go:356] "Generic (PLEG): container finished" podID="12e9b5d0-00a7-4b96-9a9a-d403c5bae27c" containerID="41c963fefe4b4c30e0ca41187723250553da6f5a56dbdc215f0541443e815ef2" exitCode=0 Nov 02 06:32:31.041223 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:31.040807 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e2c401b-drbp6" event={"ID":"12e9b5d0-00a7-4b96-9a9a-d403c5bae27c","Type":"ContainerDied","Data":"41c963fefe4b4c30e0ca41187723250553da6f5a56dbdc215f0541443e815ef2"} Nov 02 06:32:32.196581 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:32.196557 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e2c401b-drbp6" Nov 02 06:32:32.311100 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:32.311024 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/12e9b5d0-00a7-4b96-9a9a-d403c5bae27c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12e9b5d0-00a7-4b96-9a9a-d403c5bae27c-maas-api-xks-ca\") pod \"12e9b5d0-00a7-4b96-9a9a-d403c5bae27c\" (UID: \"12e9b5d0-00a7-4b96-9a9a-d403c5bae27c\") " Nov 02 06:32:32.311258 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:32.311145 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12e9b5d0-00a7-4b96-9a9a-d403c5bae27c-kube-api-access-72mgs\" (UniqueName: \"kubernetes.io/projected/12e9b5d0-00a7-4b96-9a9a-d403c5bae27c-kube-api-access-72mgs\") pod \"12e9b5d0-00a7-4b96-9a9a-d403c5bae27c\" (UID: \"12e9b5d0-00a7-4b96-9a9a-d403c5bae27c\") " Nov 02 06:32:32.311258 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:32.311231 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12e9b5d0-00a7-4b96-9a9a-d403c5bae27c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12e9b5d0-00a7-4b96-9a9a-d403c5bae27c-maas-api-openshift-ca\") pod \"12e9b5d0-00a7-4b96-9a9a-d403c5bae27c\" (UID: \"12e9b5d0-00a7-4b96-9a9a-d403c5bae27c\") " Nov 02 06:32:32.311697 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:32.311661 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12e9b5d0-00a7-4b96-9a9a-d403c5bae27c-maas-api-openshift-ca" pod "12e9b5d0-00a7-4b96-9a9a-d403c5bae27c" (UID: "12e9b5d0-00a7-4b96-9a9a-d403c5bae27c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:32:32.313536 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:32.313509 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12e9b5d0-00a7-4b96-9a9a-d403c5bae27c-maas-api-xks-ca" pod "12e9b5d0-00a7-4b96-9a9a-d403c5bae27c" (UID: "12e9b5d0-00a7-4b96-9a9a-d403c5bae27c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:32:32.313758 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:32.313734 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12e9b5d0-00a7-4b96-9a9a-d403c5bae27c-kube-api-access-72mgs" pod "12e9b5d0-00a7-4b96-9a9a-d403c5bae27c" (UID: "12e9b5d0-00a7-4b96-9a9a-d403c5bae27c"). InnerVolumeSpecName "kube-api-access-72mgs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:32:32.412428 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:32.412399 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12e9b5d0-00a7-4b96-9a9a-d403c5bae27c-maas-api-openshift-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:32:32.412428 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:32.412429 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12e9b5d0-00a7-4b96-9a9a-d403c5bae27c-maas-api-xks-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:32:32.412623 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:32.412442 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72mgs\" (UniqueName: \"kubernetes.io/projected/12e9b5d0-00a7-4b96-9a9a-d403c5bae27c-kube-api-access-72mgs\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:32:33.055827 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:33.055798 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e2c401b-drbp6" Nov 02 06:32:33.056013 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:33.055840 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e2c401b-drbp6" event={"ID":"12e9b5d0-00a7-4b96-9a9a-d403c5bae27c","Type":"ContainerDied","Data":"604a57cc180e81d1009e9dc9c084a65054cdbe4f6e7d447f36e31af92f816daf"} Nov 02 06:32:33.056013 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:33.055900 5916 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="604a57cc180e81d1009e9dc9c084a65054cdbe4f6e7d447f36e31af92f816daf" Nov 02 06:32:33.169751 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:33.169721 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-lvn8x-gtslh"] Nov 02 06:32:33.170241 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:33.170220 5916 cpu_manager.go:414] "containerMap: removing container" podUID="12e9b5d0-00a7-4b96-9a9a-d403c5bae27c" containerName="revoke-keys" Nov 02 06:32:33.170241 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:33.170238 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12e9b5d0-00a7-4b96-9a9a-d403c5bae27c" containerName="revoke-keys" Nov 02 06:32:33.170441 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:33.170372 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="12e9b5d0-00a7-4b96-9a9a-d403c5bae27c" containerName="revoke-keys" Nov 02 06:32:33.174297 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:33.174275 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-lvn8x-gtslh" Nov 02 06:32:33.219198 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:33.219174 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4fb4f2e-7ef9-4223-973f-494bbcb98a82-service-ca-bundle\") pod \"periodic-gathering-lvn8x-gtslh\" (UID: \"e4fb4f2e-7ef9-4223-973f-494bbcb98a82\") " pod="openshift-insights/periodic-gathering-lvn8x-gtslh" Nov 02 06:32:33.219513 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:33.219236 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4fb4f2e-7ef9-4223-973f-494bbcb98a82-trusted-ca-bundle\") pod \"periodic-gathering-lvn8x-gtslh\" (UID: \"e4fb4f2e-7ef9-4223-973f-494bbcb98a82\") " pod="openshift-insights/periodic-gathering-lvn8x-gtslh" Nov 02 06:32:33.219513 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:33.219319 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/e4fb4f2e-7ef9-4223-973f-494bbcb98a82-archives-path\") pod \"periodic-gathering-lvn8x-gtslh\" (UID: \"e4fb4f2e-7ef9-4223-973f-494bbcb98a82\") " pod="openshift-insights/periodic-gathering-lvn8x-gtslh" Nov 02 06:32:33.219513 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:33.219369 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lszfj\" (UniqueName: \"kubernetes.io/projected/e4fb4f2e-7ef9-4223-973f-494bbcb98a82-kube-api-access-lszfj\") pod \"periodic-gathering-lvn8x-gtslh\" (UID: \"e4fb4f2e-7ef9-4223-973f-494bbcb98a82\") " pod="openshift-insights/periodic-gathering-lvn8x-gtslh" Nov 02 06:32:33.223691 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:33.223661 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-mf7sr\"" Nov 02 06:32:33.226123 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:33.226100 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-lvn8x-gtslh"] Nov 02 06:32:33.320512 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:33.320423 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4fb4f2e-7ef9-4223-973f-494bbcb98a82-service-ca-bundle\") pod \"periodic-gathering-lvn8x-gtslh\" (UID: \"e4fb4f2e-7ef9-4223-973f-494bbcb98a82\") " pod="openshift-insights/periodic-gathering-lvn8x-gtslh" Nov 02 06:32:33.320512 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:33.320505 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4fb4f2e-7ef9-4223-973f-494bbcb98a82-trusted-ca-bundle\") pod \"periodic-gathering-lvn8x-gtslh\" (UID: \"e4fb4f2e-7ef9-4223-973f-494bbcb98a82\") " pod="openshift-insights/periodic-gathering-lvn8x-gtslh" Nov 02 06:32:33.320747 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:33.320660 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/e4fb4f2e-7ef9-4223-973f-494bbcb98a82-archives-path\") pod \"periodic-gathering-lvn8x-gtslh\" (UID: \"e4fb4f2e-7ef9-4223-973f-494bbcb98a82\") " pod="openshift-insights/periodic-gathering-lvn8x-gtslh" Nov 02 06:32:33.320809 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:33.320752 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lszfj\" (UniqueName: \"kubernetes.io/projected/e4fb4f2e-7ef9-4223-973f-494bbcb98a82-kube-api-access-lszfj\") pod \"periodic-gathering-lvn8x-gtslh\" (UID: \"e4fb4f2e-7ef9-4223-973f-494bbcb98a82\") " pod="openshift-insights/periodic-gathering-lvn8x-gtslh" Nov 02 06:32:33.321020 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:33.321002 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/e4fb4f2e-7ef9-4223-973f-494bbcb98a82-archives-path\") pod \"periodic-gathering-lvn8x-gtslh\" (UID: \"e4fb4f2e-7ef9-4223-973f-494bbcb98a82\") " pod="openshift-insights/periodic-gathering-lvn8x-gtslh" Nov 02 06:32:33.321255 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:33.321237 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4fb4f2e-7ef9-4223-973f-494bbcb98a82-service-ca-bundle\") pod \"periodic-gathering-lvn8x-gtslh\" (UID: \"e4fb4f2e-7ef9-4223-973f-494bbcb98a82\") " pod="openshift-insights/periodic-gathering-lvn8x-gtslh" Nov 02 06:32:33.321351 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:33.321334 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4fb4f2e-7ef9-4223-973f-494bbcb98a82-trusted-ca-bundle\") pod \"periodic-gathering-lvn8x-gtslh\" (UID: \"e4fb4f2e-7ef9-4223-973f-494bbcb98a82\") " pod="openshift-insights/periodic-gathering-lvn8x-gtslh" Nov 02 06:32:33.376153 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:33.376118 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lszfj\" (UniqueName: \"kubernetes.io/projected/e4fb4f2e-7ef9-4223-973f-494bbcb98a82-kube-api-access-lszfj\") pod \"periodic-gathering-lvn8x-gtslh\" (UID: \"e4fb4f2e-7ef9-4223-973f-494bbcb98a82\") " pod="openshift-insights/periodic-gathering-lvn8x-gtslh" Nov 02 06:32:33.497181 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:33.497149 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-lvn8x-gtslh" Nov 02 06:32:33.868222 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:33.868188 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-lvn8x-gtslh"] Nov 02 06:32:33.872866 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:32:33.872840 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4fb4f2e_7ef9_4223_973f_494bbcb98a82.slice/crio-40b31d3013011aab2807d354beb5340ceb06f052eb8e223f4faa894679b07721 WatchSource:0}: Error finding container 40b31d3013011aab2807d354beb5340ceb06f052eb8e223f4faa894679b07721: Status 404 returned error can't find the container with id 40b31d3013011aab2807d354beb5340ceb06f052eb8e223f4faa894679b07721 Nov 02 06:32:34.011250 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:34.011208 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2e0b70a-b2m7s"] Nov 02 06:32:34.020072 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:34.020045 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2e0b70a-b2m7s" Nov 02 06:32:34.063360 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:34.063330 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2e0b70a-b2m7s"] Nov 02 06:32:34.071977 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:34.066460 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-lvn8x-gtslh" event={"ID":"e4fb4f2e-7ef9-4223-973f-494bbcb98a82","Type":"ContainerStarted","Data":"7fe13de0cea50638b78dd0af0926f417675b17e76904acc1af1a4679c9624d2f"} Nov 02 06:32:34.071977 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:34.066520 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-lvn8x-gtslh" event={"ID":"e4fb4f2e-7ef9-4223-973f-494bbcb98a82","Type":"ContainerStarted","Data":"40b31d3013011aab2807d354beb5340ceb06f052eb8e223f4faa894679b07721"} Nov 02 06:32:34.071977 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:34.068520 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 06:32:34.071977 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:34.068573 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-68l4r\"" Nov 02 06:32:34.142142 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:34.142114 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92434cc8-1efe-4e63-8a29-b9093d774e51-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a2e0b70a-b2m7s\" (UID: \"92434cc8-1efe-4e63-8a29-b9093d774e51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2e0b70a-b2m7s" Nov 02 06:32:34.142330 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:34.142255 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92434cc8-1efe-4e63-8a29-b9093d774e51-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a2e0b70a-b2m7s\" (UID: \"92434cc8-1efe-4e63-8a29-b9093d774e51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2e0b70a-b2m7s" Nov 02 06:32:34.142429 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:34.142407 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzkdd\" (UniqueName: \"kubernetes.io/projected/92434cc8-1efe-4e63-8a29-b9093d774e51-kube-api-access-qzkdd\") pod \"maas-api-revoke-sub-a2e0b70a-b2m7s\" (UID: \"92434cc8-1efe-4e63-8a29-b9093d774e51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2e0b70a-b2m7s" Nov 02 06:32:34.230363 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:34.230316 5916 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-lvn8x-gtslh" podStartSLOduration=1.230303397 podStartE2EDuration="1.230303397s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:32:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:32:34.229521311 +0000 UTC m=+942.501341730" watchObservedRunningTime="2026-11-02 06:32:34.230303397 +0000 UTC m=+942.502123817" Nov 02 06:32:34.243349 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:34.243319 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92434cc8-1efe-4e63-8a29-b9093d774e51-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a2e0b70a-b2m7s\" (UID: \"92434cc8-1efe-4e63-8a29-b9093d774e51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2e0b70a-b2m7s" Nov 02 06:32:34.243481 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:34.243381 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92434cc8-1efe-4e63-8a29-b9093d774e51-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a2e0b70a-b2m7s\" (UID: \"92434cc8-1efe-4e63-8a29-b9093d774e51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2e0b70a-b2m7s" Nov 02 06:32:34.243481 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:34.243463 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzkdd\" (UniqueName: \"kubernetes.io/projected/92434cc8-1efe-4e63-8a29-b9093d774e51-kube-api-access-qzkdd\") pod \"maas-api-revoke-sub-a2e0b70a-b2m7s\" (UID: \"92434cc8-1efe-4e63-8a29-b9093d774e51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2e0b70a-b2m7s" Nov 02 06:32:34.244098 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:34.244076 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92434cc8-1efe-4e63-8a29-b9093d774e51-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a2e0b70a-b2m7s\" (UID: \"92434cc8-1efe-4e63-8a29-b9093d774e51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2e0b70a-b2m7s" Nov 02 06:32:34.245608 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:34.245590 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92434cc8-1efe-4e63-8a29-b9093d774e51-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a2e0b70a-b2m7s\" (UID: \"92434cc8-1efe-4e63-8a29-b9093d774e51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2e0b70a-b2m7s" Nov 02 06:32:34.297674 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:34.297640 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzkdd\" (UniqueName: \"kubernetes.io/projected/92434cc8-1efe-4e63-8a29-b9093d774e51-kube-api-access-qzkdd\") pod \"maas-api-revoke-sub-a2e0b70a-b2m7s\" (UID: \"92434cc8-1efe-4e63-8a29-b9093d774e51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2e0b70a-b2m7s" Nov 02 06:32:34.334439 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:34.334411 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2e0b70a-b2m7s" Nov 02 06:32:34.712340 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:34.712261 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2e0b70a-b2m7s"] Nov 02 06:32:35.074838 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:35.074802 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2e0b70a-b2m7s" event={"ID":"92434cc8-1efe-4e63-8a29-b9093d774e51","Type":"ContainerStarted","Data":"e27ed737da6f8f9053ffbb759fc738062685d3238d9ad9f745f727635bc9b18a"} Nov 02 06:32:35.075041 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:35.074866 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2e0b70a-b2m7s" event={"ID":"92434cc8-1efe-4e63-8a29-b9093d774e51","Type":"ContainerStarted","Data":"8f062222c9b6f0bc384c1aed519c5f6931572a0bae4bb13db368e80c72851171"} Nov 02 06:32:35.181789 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:35.181738 5916 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2e0b70a-b2m7s" podStartSLOduration=2.181724845 podStartE2EDuration="2.181724845s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:32:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:32:35.179166113 +0000 UTC m=+943.450986505" watchObservedRunningTime="2026-11-02 06:32:35.181724845 +0000 UTC m=+943.453545238" Nov 02 06:32:36.083248 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:36.083155 5916 generic.go:356] "Generic (PLEG): container finished" podID="92434cc8-1efe-4e63-8a29-b9093d774e51" containerID="e27ed737da6f8f9053ffbb759fc738062685d3238d9ad9f745f727635bc9b18a" exitCode=0 Nov 02 06:32:36.083662 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:36.083240 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2e0b70a-b2m7s" event={"ID":"92434cc8-1efe-4e63-8a29-b9093d774e51","Type":"ContainerDied","Data":"e27ed737da6f8f9053ffbb759fc738062685d3238d9ad9f745f727635bc9b18a"} Nov 02 06:32:37.262948 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:37.262876 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2e0b70a-b2m7s" Nov 02 06:32:37.379059 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:37.378962 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/92434cc8-1efe-4e63-8a29-b9093d774e51-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92434cc8-1efe-4e63-8a29-b9093d774e51-maas-api-xks-ca\") pod \"92434cc8-1efe-4e63-8a29-b9093d774e51\" (UID: \"92434cc8-1efe-4e63-8a29-b9093d774e51\") " Nov 02 06:32:37.379218 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:37.379081 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92434cc8-1efe-4e63-8a29-b9093d774e51-kube-api-access-qzkdd\" (UniqueName: \"kubernetes.io/projected/92434cc8-1efe-4e63-8a29-b9093d774e51-kube-api-access-qzkdd\") pod \"92434cc8-1efe-4e63-8a29-b9093d774e51\" (UID: \"92434cc8-1efe-4e63-8a29-b9093d774e51\") " Nov 02 06:32:37.379218 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:37.379132 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92434cc8-1efe-4e63-8a29-b9093d774e51-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92434cc8-1efe-4e63-8a29-b9093d774e51-maas-api-openshift-ca\") pod \"92434cc8-1efe-4e63-8a29-b9093d774e51\" (UID: \"92434cc8-1efe-4e63-8a29-b9093d774e51\") " Nov 02 06:32:37.379830 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:37.379783 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92434cc8-1efe-4e63-8a29-b9093d774e51-maas-api-openshift-ca" pod "92434cc8-1efe-4e63-8a29-b9093d774e51" (UID: "92434cc8-1efe-4e63-8a29-b9093d774e51"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:32:37.381654 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:37.381620 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92434cc8-1efe-4e63-8a29-b9093d774e51-maas-api-xks-ca" pod "92434cc8-1efe-4e63-8a29-b9093d774e51" (UID: "92434cc8-1efe-4e63-8a29-b9093d774e51"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:32:37.384069 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:37.384037 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92434cc8-1efe-4e63-8a29-b9093d774e51-kube-api-access-qzkdd" pod "92434cc8-1efe-4e63-8a29-b9093d774e51" (UID: "92434cc8-1efe-4e63-8a29-b9093d774e51"). InnerVolumeSpecName "kube-api-access-qzkdd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:32:37.480601 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:37.480568 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qzkdd\" (UniqueName: \"kubernetes.io/projected/92434cc8-1efe-4e63-8a29-b9093d774e51-kube-api-access-qzkdd\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:32:37.480601 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:37.480594 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92434cc8-1efe-4e63-8a29-b9093d774e51-maas-api-openshift-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:32:37.480601 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:37.480604 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92434cc8-1efe-4e63-8a29-b9093d774e51-maas-api-xks-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:32:38.098418 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:38.098391 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2e0b70a-b2m7s" Nov 02 06:32:38.098594 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:38.098387 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2e0b70a-b2m7s" event={"ID":"92434cc8-1efe-4e63-8a29-b9093d774e51","Type":"ContainerDied","Data":"8f062222c9b6f0bc384c1aed519c5f6931572a0bae4bb13db368e80c72851171"} Nov 02 06:32:38.098594 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:38.098518 5916 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8f062222c9b6f0bc384c1aed519c5f6931572a0bae4bb13db368e80c72851171" Nov 02 06:32:44.978707 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:44.978675 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b4c2887-fq7c4"] Nov 02 06:32:44.979293 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:44.979276 5916 cpu_manager.go:414] "containerMap: removing container" podUID="92434cc8-1efe-4e63-8a29-b9093d774e51" containerName="revoke-keys" Nov 02 06:32:44.979349 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:44.979296 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92434cc8-1efe-4e63-8a29-b9093d774e51" containerName="revoke-keys" Nov 02 06:32:44.979396 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:44.979386 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="92434cc8-1efe-4e63-8a29-b9093d774e51" containerName="revoke-keys" Nov 02 06:32:44.982885 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:44.982867 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b4c2887-fq7c4" Nov 02 06:32:45.035993 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:45.035965 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b4c2887-fq7c4"] Nov 02 06:32:45.077641 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:45.077615 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 06:32:45.077795 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:45.077662 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-68l4r\"" Nov 02 06:32:45.162956 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:45.162899 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bb59ace-463d-45b3-8201-7b13efd8b097-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3b4c2887-fq7c4\" (UID: \"2bb59ace-463d-45b3-8201-7b13efd8b097\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b4c2887-fq7c4" Nov 02 06:32:45.163113 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:45.163027 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bb59ace-463d-45b3-8201-7b13efd8b097-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3b4c2887-fq7c4\" (UID: \"2bb59ace-463d-45b3-8201-7b13efd8b097\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b4c2887-fq7c4" Nov 02 06:32:45.163113 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:45.163081 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpmt2\" (UniqueName: \"kubernetes.io/projected/2bb59ace-463d-45b3-8201-7b13efd8b097-kube-api-access-hpmt2\") pod \"maas-api-revoke-sub-3b4c2887-fq7c4\" (UID: \"2bb59ace-463d-45b3-8201-7b13efd8b097\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b4c2887-fq7c4" Nov 02 06:32:45.264355 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:45.264271 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bb59ace-463d-45b3-8201-7b13efd8b097-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3b4c2887-fq7c4\" (UID: \"2bb59ace-463d-45b3-8201-7b13efd8b097\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b4c2887-fq7c4" Nov 02 06:32:45.264510 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:45.264350 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpmt2\" (UniqueName: \"kubernetes.io/projected/2bb59ace-463d-45b3-8201-7b13efd8b097-kube-api-access-hpmt2\") pod \"maas-api-revoke-sub-3b4c2887-fq7c4\" (UID: \"2bb59ace-463d-45b3-8201-7b13efd8b097\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b4c2887-fq7c4" Nov 02 06:32:45.264510 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:45.264488 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bb59ace-463d-45b3-8201-7b13efd8b097-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3b4c2887-fq7c4\" (UID: \"2bb59ace-463d-45b3-8201-7b13efd8b097\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b4c2887-fq7c4" Nov 02 06:32:45.265274 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:45.265244 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bb59ace-463d-45b3-8201-7b13efd8b097-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3b4c2887-fq7c4\" (UID: \"2bb59ace-463d-45b3-8201-7b13efd8b097\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b4c2887-fq7c4" Nov 02 06:32:45.266727 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:45.266703 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bb59ace-463d-45b3-8201-7b13efd8b097-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3b4c2887-fq7c4\" (UID: \"2bb59ace-463d-45b3-8201-7b13efd8b097\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b4c2887-fq7c4" Nov 02 06:32:45.317040 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:45.317015 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpmt2\" (UniqueName: \"kubernetes.io/projected/2bb59ace-463d-45b3-8201-7b13efd8b097-kube-api-access-hpmt2\") pod \"maas-api-revoke-sub-3b4c2887-fq7c4\" (UID: \"2bb59ace-463d-45b3-8201-7b13efd8b097\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b4c2887-fq7c4" Nov 02 06:32:45.596030 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:45.595940 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b4c2887-fq7c4" Nov 02 06:32:45.975499 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:45.975454 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b4c2887-fq7c4"] Nov 02 06:32:45.981514 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:32:45.981484 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bb59ace_463d_45b3_8201_7b13efd8b097.slice/crio-a9da6a7b683ad22228a5e2834236e489cae2cadd3516864cbf72373890863437 WatchSource:0}: Error finding container a9da6a7b683ad22228a5e2834236e489cae2cadd3516864cbf72373890863437: Status 404 returned error can't find the container with id a9da6a7b683ad22228a5e2834236e489cae2cadd3516864cbf72373890863437 Nov 02 06:32:46.158713 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:46.158672 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b4c2887-fq7c4" event={"ID":"2bb59ace-463d-45b3-8201-7b13efd8b097","Type":"ContainerStarted","Data":"a9da6a7b683ad22228a5e2834236e489cae2cadd3516864cbf72373890863437"} Nov 02 06:32:47.168156 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:47.168071 5916 generic.go:356] "Generic (PLEG): container finished" podID="2bb59ace-463d-45b3-8201-7b13efd8b097" containerID="0d968e7ca96e5944c2f67ec7d9904b8b71c8a2337b26a236b19c978e50577e73" exitCode=0 Nov 02 06:32:47.168614 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:47.168197 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b4c2887-fq7c4" event={"ID":"2bb59ace-463d-45b3-8201-7b13efd8b097","Type":"ContainerDied","Data":"0d968e7ca96e5944c2f67ec7d9904b8b71c8a2337b26a236b19c978e50577e73"} Nov 02 06:32:48.315106 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:48.315081 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b4c2887-fq7c4" Nov 02 06:32:48.401555 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:48.401525 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bb59ace-463d-45b3-8201-7b13efd8b097-kube-api-access-hpmt2\" (UniqueName: \"kubernetes.io/projected/2bb59ace-463d-45b3-8201-7b13efd8b097-kube-api-access-hpmt2\") pod \"2bb59ace-463d-45b3-8201-7b13efd8b097\" (UID: \"2bb59ace-463d-45b3-8201-7b13efd8b097\") " Nov 02 06:32:48.401696 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:48.401655 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2bb59ace-463d-45b3-8201-7b13efd8b097-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bb59ace-463d-45b3-8201-7b13efd8b097-maas-api-openshift-ca\") pod \"2bb59ace-463d-45b3-8201-7b13efd8b097\" (UID: \"2bb59ace-463d-45b3-8201-7b13efd8b097\") " Nov 02 06:32:48.401745 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:48.401691 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2bb59ace-463d-45b3-8201-7b13efd8b097-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bb59ace-463d-45b3-8201-7b13efd8b097-maas-api-xks-ca\") pod \"2bb59ace-463d-45b3-8201-7b13efd8b097\" (UID: \"2bb59ace-463d-45b3-8201-7b13efd8b097\") " Nov 02 06:32:48.402007 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:48.401983 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bb59ace-463d-45b3-8201-7b13efd8b097-maas-api-openshift-ca" pod "2bb59ace-463d-45b3-8201-7b13efd8b097" (UID: "2bb59ace-463d-45b3-8201-7b13efd8b097"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:32:48.402102 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:48.402062 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bb59ace-463d-45b3-8201-7b13efd8b097-maas-api-openshift-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:32:48.403608 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:48.403586 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2bb59ace-463d-45b3-8201-7b13efd8b097-maas-api-xks-ca" pod "2bb59ace-463d-45b3-8201-7b13efd8b097" (UID: "2bb59ace-463d-45b3-8201-7b13efd8b097"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:32:48.403671 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:48.403629 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bb59ace-463d-45b3-8201-7b13efd8b097-kube-api-access-hpmt2" pod "2bb59ace-463d-45b3-8201-7b13efd8b097" (UID: "2bb59ace-463d-45b3-8201-7b13efd8b097"). InnerVolumeSpecName "kube-api-access-hpmt2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:32:48.502798 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:48.502725 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bb59ace-463d-45b3-8201-7b13efd8b097-maas-api-xks-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:32:48.502798 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:48.502769 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hpmt2\" (UniqueName: \"kubernetes.io/projected/2bb59ace-463d-45b3-8201-7b13efd8b097-kube-api-access-hpmt2\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:32:49.182942 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:49.182914 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b4c2887-fq7c4" Nov 02 06:32:49.183091 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:49.182935 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b4c2887-fq7c4" event={"ID":"2bb59ace-463d-45b3-8201-7b13efd8b097","Type":"ContainerDied","Data":"a9da6a7b683ad22228a5e2834236e489cae2cadd3516864cbf72373890863437"} Nov 02 06:32:49.183091 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:49.182996 5916 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a9da6a7b683ad22228a5e2834236e489cae2cadd3516864cbf72373890863437" Nov 02 06:32:49.472020 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:49.471942 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a55c94e5-tw9c2"] Nov 02 06:32:49.472436 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:49.472424 5916 cpu_manager.go:414] "containerMap: removing container" podUID="2bb59ace-463d-45b3-8201-7b13efd8b097" containerName="revoke-keys" Nov 02 06:32:49.472524 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:49.472438 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bb59ace-463d-45b3-8201-7b13efd8b097" containerName="revoke-keys" Nov 02 06:32:49.472570 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:49.472546 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bb59ace-463d-45b3-8201-7b13efd8b097" containerName="revoke-keys" Nov 02 06:32:49.476457 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:49.476433 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a55c94e5-tw9c2" Nov 02 06:32:49.513865 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:49.513836 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgdrs\" (UniqueName: \"kubernetes.io/projected/bcf9063a-1c58-445e-91bf-cccca7ee34b2-kube-api-access-rgdrs\") pod \"maas-api-revoke-sub-a55c94e5-tw9c2\" (UID: \"bcf9063a-1c58-445e-91bf-cccca7ee34b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a55c94e5-tw9c2" Nov 02 06:32:49.514033 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:49.513923 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcf9063a-1c58-445e-91bf-cccca7ee34b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a55c94e5-tw9c2\" (UID: \"bcf9063a-1c58-445e-91bf-cccca7ee34b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a55c94e5-tw9c2" Nov 02 06:32:49.514079 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:49.514051 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcf9063a-1c58-445e-91bf-cccca7ee34b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a55c94e5-tw9c2\" (UID: \"bcf9063a-1c58-445e-91bf-cccca7ee34b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a55c94e5-tw9c2" Nov 02 06:32:49.524954 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:49.524901 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 06:32:49.525154 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:49.525139 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-68l4r\"" Nov 02 06:32:49.537851 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:49.537825 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a55c94e5-tw9c2"] Nov 02 06:32:49.615519 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:49.615484 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcf9063a-1c58-445e-91bf-cccca7ee34b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a55c94e5-tw9c2\" (UID: \"bcf9063a-1c58-445e-91bf-cccca7ee34b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a55c94e5-tw9c2" Nov 02 06:32:49.615697 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:49.615541 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgdrs\" (UniqueName: \"kubernetes.io/projected/bcf9063a-1c58-445e-91bf-cccca7ee34b2-kube-api-access-rgdrs\") pod \"maas-api-revoke-sub-a55c94e5-tw9c2\" (UID: \"bcf9063a-1c58-445e-91bf-cccca7ee34b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a55c94e5-tw9c2" Nov 02 06:32:49.615697 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:49.615589 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcf9063a-1c58-445e-91bf-cccca7ee34b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a55c94e5-tw9c2\" (UID: \"bcf9063a-1c58-445e-91bf-cccca7ee34b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a55c94e5-tw9c2" Nov 02 06:32:49.616213 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:49.616194 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcf9063a-1c58-445e-91bf-cccca7ee34b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a55c94e5-tw9c2\" (UID: \"bcf9063a-1c58-445e-91bf-cccca7ee34b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a55c94e5-tw9c2" Nov 02 06:32:49.617778 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:49.617754 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcf9063a-1c58-445e-91bf-cccca7ee34b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a55c94e5-tw9c2\" (UID: \"bcf9063a-1c58-445e-91bf-cccca7ee34b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a55c94e5-tw9c2" Nov 02 06:32:49.669089 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:49.669051 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgdrs\" (UniqueName: \"kubernetes.io/projected/bcf9063a-1c58-445e-91bf-cccca7ee34b2-kube-api-access-rgdrs\") pod \"maas-api-revoke-sub-a55c94e5-tw9c2\" (UID: \"bcf9063a-1c58-445e-91bf-cccca7ee34b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a55c94e5-tw9c2" Nov 02 06:32:49.790843 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:49.790757 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a55c94e5-tw9c2" Nov 02 06:32:49.900168 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:32:49.899223 5916 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p98cq" containerName="extractor" Nov 02 06:32:50.172315 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:50.172280 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a55c94e5-tw9c2"] Nov 02 06:32:50.178002 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:32:50.177971 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcf9063a_1c58_445e_91bf_cccca7ee34b2.slice/crio-1887f9b0ae68dafc0799ba30384c24bad60e32d603bb513ed4081abc250dbb66 WatchSource:0}: Error finding container 1887f9b0ae68dafc0799ba30384c24bad60e32d603bb513ed4081abc250dbb66: Status 404 returned error can't find the container with id 1887f9b0ae68dafc0799ba30384c24bad60e32d603bb513ed4081abc250dbb66 Nov 02 06:32:50.199076 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:50.199044 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a55c94e5-tw9c2" event={"ID":"bcf9063a-1c58-445e-91bf-cccca7ee34b2","Type":"ContainerStarted","Data":"1887f9b0ae68dafc0799ba30384c24bad60e32d603bb513ed4081abc250dbb66"} Nov 02 06:32:51.206809 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:51.206771 5916 generic.go:356] "Generic (PLEG): container finished" podID="bcf9063a-1c58-445e-91bf-cccca7ee34b2" containerID="44fc1b54d1a6c81c5029b5f28100a193e9bf2088944f9961337ed9e3124ed787" exitCode=0 Nov 02 06:32:51.207249 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:51.206855 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a55c94e5-tw9c2" event={"ID":"bcf9063a-1c58-445e-91bf-cccca7ee34b2","Type":"ContainerDied","Data":"44fc1b54d1a6c81c5029b5f28100a193e9bf2088944f9961337ed9e3124ed787"} Nov 02 06:32:52.359288 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:52.359264 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a55c94e5-tw9c2" Nov 02 06:32:52.447662 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:52.447628 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bcf9063a-1c58-445e-91bf-cccca7ee34b2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcf9063a-1c58-445e-91bf-cccca7ee34b2-maas-api-xks-ca\") pod \"bcf9063a-1c58-445e-91bf-cccca7ee34b2\" (UID: \"bcf9063a-1c58-445e-91bf-cccca7ee34b2\") " Nov 02 06:32:52.447809 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:52.447745 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bcf9063a-1c58-445e-91bf-cccca7ee34b2-kube-api-access-rgdrs\" (UniqueName: \"kubernetes.io/projected/bcf9063a-1c58-445e-91bf-cccca7ee34b2-kube-api-access-rgdrs\") pod \"bcf9063a-1c58-445e-91bf-cccca7ee34b2\" (UID: \"bcf9063a-1c58-445e-91bf-cccca7ee34b2\") " Nov 02 06:32:52.447809 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:52.447790 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bcf9063a-1c58-445e-91bf-cccca7ee34b2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcf9063a-1c58-445e-91bf-cccca7ee34b2-maas-api-openshift-ca\") pod \"bcf9063a-1c58-445e-91bf-cccca7ee34b2\" (UID: \"bcf9063a-1c58-445e-91bf-cccca7ee34b2\") " Nov 02 06:32:52.448196 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:52.448173 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bcf9063a-1c58-445e-91bf-cccca7ee34b2-maas-api-openshift-ca" pod "bcf9063a-1c58-445e-91bf-cccca7ee34b2" (UID: "bcf9063a-1c58-445e-91bf-cccca7ee34b2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:32:52.449646 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:52.449622 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bcf9063a-1c58-445e-91bf-cccca7ee34b2-maas-api-xks-ca" pod "bcf9063a-1c58-445e-91bf-cccca7ee34b2" (UID: "bcf9063a-1c58-445e-91bf-cccca7ee34b2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:32:52.449832 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:52.449815 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcf9063a-1c58-445e-91bf-cccca7ee34b2-kube-api-access-rgdrs" pod "bcf9063a-1c58-445e-91bf-cccca7ee34b2" (UID: "bcf9063a-1c58-445e-91bf-cccca7ee34b2"). InnerVolumeSpecName "kube-api-access-rgdrs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:32:52.549361 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:52.549287 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rgdrs\" (UniqueName: \"kubernetes.io/projected/bcf9063a-1c58-445e-91bf-cccca7ee34b2-kube-api-access-rgdrs\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:32:52.549361 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:52.549315 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcf9063a-1c58-445e-91bf-cccca7ee34b2-maas-api-openshift-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:32:52.549361 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:52.549325 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcf9063a-1c58-445e-91bf-cccca7ee34b2-maas-api-xks-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:32:53.223792 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:53.223763 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a55c94e5-tw9c2" Nov 02 06:32:53.223792 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:53.223784 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a55c94e5-tw9c2" event={"ID":"bcf9063a-1c58-445e-91bf-cccca7ee34b2","Type":"ContainerDied","Data":"1887f9b0ae68dafc0799ba30384c24bad60e32d603bb513ed4081abc250dbb66"} Nov 02 06:32:53.224050 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:32:53.223833 5916 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1887f9b0ae68dafc0799ba30384c24bad60e32d603bb513ed4081abc250dbb66" Nov 02 06:33:00.556302 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:00.556264 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-wk9b2"] Nov 02 06:33:00.557059 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:00.557039 5916 cpu_manager.go:414] "containerMap: removing container" podUID="bcf9063a-1c58-445e-91bf-cccca7ee34b2" containerName="revoke-keys" Nov 02 06:33:00.557152 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:00.557061 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bcf9063a-1c58-445e-91bf-cccca7ee34b2" containerName="revoke-keys" Nov 02 06:33:00.557230 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:00.557219 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="bcf9063a-1c58-445e-91bf-cccca7ee34b2" containerName="revoke-keys" Nov 02 06:33:00.562396 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:00.562376 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-wk9b2" Nov 02 06:33:00.600394 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:00.600362 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-baa3f47a-mhvfc"] Nov 02 06:33:00.604255 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:00.604239 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-baa3f47a-mhvfc" Nov 02 06:33:00.614112 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:00.614093 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-680b7b-openshift-default-dockercfg-sccx9\"" Nov 02 06:33:00.628831 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:00.628804 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d75560c-3f13-4f05-9e64-b78fdb08e49c-istio-podinfo\") pod \"e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-wk9b2\" (UID: \"6d75560c-3f13-4f05-9e64-b78fdb08e49c\") " pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-wk9b2" Nov 02 06:33:00.628944 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:00.628846 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d75560c-3f13-4f05-9e64-b78fdb08e49c-istio-envoy\") pod \"e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-wk9b2\" (UID: \"6d75560c-3f13-4f05-9e64-b78fdb08e49c\") " pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-wk9b2" Nov 02 06:33:00.629006 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:00.628943 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d75560c-3f13-4f05-9e64-b78fdb08e49c-istio-data\") pod \"e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-wk9b2\" (UID: \"6d75560c-3f13-4f05-9e64-b78fdb08e49c\") " pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-wk9b2" Nov 02 06:33:00.629006 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:00.628969 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d75560c-3f13-4f05-9e64-b78fdb08e49c-istiod-ca-cert\") pod \"e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-wk9b2\" (UID: \"6d75560c-3f13-4f05-9e64-b78fdb08e49c\") " pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-wk9b2" Nov 02 06:33:00.629006 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:00.628993 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26kdr\" (UniqueName: \"kubernetes.io/projected/6d75560c-3f13-4f05-9e64-b78fdb08e49c-kube-api-access-26kdr\") pod \"e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-wk9b2\" (UID: \"6d75560c-3f13-4f05-9e64-b78fdb08e49c\") " pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-wk9b2" Nov 02 06:33:00.629143 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:00.629121 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d75560c-3f13-4f05-9e64-b78fdb08e49c-credential-socket\") pod \"e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-wk9b2\" (UID: \"6d75560c-3f13-4f05-9e64-b78fdb08e49c\") " pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-wk9b2" Nov 02 06:33:00.629226 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:00.629214 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d75560c-3f13-4f05-9e64-b78fdb08e49c-workload-socket\") pod \"e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-wk9b2\" (UID: \"6d75560c-3f13-4f05-9e64-b78fdb08e49c\") " pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-wk9b2" Nov 02 06:33:00.629262 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:00.629242 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d75560c-3f13-4f05-9e64-b78fdb08e49c-istio-token\") pod \"e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-wk9b2\" (UID: \"6d75560c-3f13-4f05-9e64-b78fdb08e49c\") " pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-wk9b2" Nov 02 06:33:00.629367 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:00.629353 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d75560c-3f13-4f05-9e64-b78fdb08e49c-workload-certs\") pod \"e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-wk9b2\" (UID: \"6d75560c-3f13-4f05-9e64-b78fdb08e49c\") " pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-wk9b2" Nov 02 06:33:00.650887 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:00.650862 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-baa3f47a-mhvfc"] Nov 02 06:33:00.662133 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:00.662105 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 06:33:00.662265 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:00.662202 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-wk9b2"] Nov 02 06:33:00.662522 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:00.662494 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-68l4r\"" Nov 02 06:33:00.730919 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:00.730873 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d75560c-3f13-4f05-9e64-b78fdb08e49c-istio-data\") pod \"e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-wk9b2\" (UID: \"6d75560c-3f13-4f05-9e64-b78fdb08e49c\") " pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-wk9b2" Nov 02 06:33:00.731092 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:00.730940 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d75560c-3f13-4f05-9e64-b78fdb08e49c-istiod-ca-cert\") pod \"e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-wk9b2\" (UID: \"6d75560c-3f13-4f05-9e64-b78fdb08e49c\") " pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-wk9b2" Nov 02 06:33:00.731092 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:00.730975 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26kdr\" (UniqueName: \"kubernetes.io/projected/6d75560c-3f13-4f05-9e64-b78fdb08e49c-kube-api-access-26kdr\") pod \"e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-wk9b2\" (UID: \"6d75560c-3f13-4f05-9e64-b78fdb08e49c\") " pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-wk9b2" Nov 02 06:33:00.731092 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:00.731008 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff33efe1-03ac-4157-acda-a302dd262e10-maas-api-xks-ca\") pod \"maas-api-revoke-sub-baa3f47a-mhvfc\" (UID: \"ff33efe1-03ac-4157-acda-a302dd262e10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-baa3f47a-mhvfc" Nov 02 06:33:00.731092 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:00.731030 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d75560c-3f13-4f05-9e64-b78fdb08e49c-credential-socket\") pod \"e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-wk9b2\" (UID: \"6d75560c-3f13-4f05-9e64-b78fdb08e49c\") " pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-wk9b2" Nov 02 06:33:00.731092 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:00.731062 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff33efe1-03ac-4157-acda-a302dd262e10-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-baa3f47a-mhvfc\" (UID: \"ff33efe1-03ac-4157-acda-a302dd262e10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-baa3f47a-mhvfc" Nov 02 06:33:00.731092 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:00.731090 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d75560c-3f13-4f05-9e64-b78fdb08e49c-workload-socket\") pod \"e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-wk9b2\" (UID: \"6d75560c-3f13-4f05-9e64-b78fdb08e49c\") " pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-wk9b2" Nov 02 06:33:00.731395 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:00.731105 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d75560c-3f13-4f05-9e64-b78fdb08e49c-istio-token\") pod \"e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-wk9b2\" (UID: \"6d75560c-3f13-4f05-9e64-b78fdb08e49c\") " pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-wk9b2" Nov 02 06:33:00.731395 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:00.731130 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdxtl\" (UniqueName: \"kubernetes.io/projected/ff33efe1-03ac-4157-acda-a302dd262e10-kube-api-access-xdxtl\") pod \"maas-api-revoke-sub-baa3f47a-mhvfc\" (UID: \"ff33efe1-03ac-4157-acda-a302dd262e10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-baa3f47a-mhvfc" Nov 02 06:33:00.731395 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:00.731179 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d75560c-3f13-4f05-9e64-b78fdb08e49c-workload-certs\") pod \"e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-wk9b2\" (UID: \"6d75560c-3f13-4f05-9e64-b78fdb08e49c\") " pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-wk9b2" Nov 02 06:33:00.731395 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:00.731230 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d75560c-3f13-4f05-9e64-b78fdb08e49c-istio-podinfo\") pod \"e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-wk9b2\" (UID: \"6d75560c-3f13-4f05-9e64-b78fdb08e49c\") " pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-wk9b2" Nov 02 06:33:00.731395 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:00.731255 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d75560c-3f13-4f05-9e64-b78fdb08e49c-istio-envoy\") pod \"e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-wk9b2\" (UID: \"6d75560c-3f13-4f05-9e64-b78fdb08e49c\") " pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-wk9b2" Nov 02 06:33:00.731664 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:00.731476 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d75560c-3f13-4f05-9e64-b78fdb08e49c-istio-data\") pod \"e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-wk9b2\" (UID: \"6d75560c-3f13-4f05-9e64-b78fdb08e49c\") " pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-wk9b2" Nov 02 06:33:00.731750 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:00.731729 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d75560c-3f13-4f05-9e64-b78fdb08e49c-workload-socket\") pod \"e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-wk9b2\" (UID: \"6d75560c-3f13-4f05-9e64-b78fdb08e49c\") " pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-wk9b2" Nov 02 06:33:00.731810 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:00.731783 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d75560c-3f13-4f05-9e64-b78fdb08e49c-istiod-ca-cert\") pod \"e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-wk9b2\" (UID: \"6d75560c-3f13-4f05-9e64-b78fdb08e49c\") " pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-wk9b2" Nov 02 06:33:00.732052 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:00.732032 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d75560c-3f13-4f05-9e64-b78fdb08e49c-credential-socket\") pod \"e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-wk9b2\" (UID: \"6d75560c-3f13-4f05-9e64-b78fdb08e49c\") " pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-wk9b2" Nov 02 06:33:00.732205 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:00.732183 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d75560c-3f13-4f05-9e64-b78fdb08e49c-workload-certs\") pod \"e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-wk9b2\" (UID: \"6d75560c-3f13-4f05-9e64-b78fdb08e49c\") " pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-wk9b2" Nov 02 06:33:00.733695 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:00.733671 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d75560c-3f13-4f05-9e64-b78fdb08e49c-istio-envoy\") pod \"e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-wk9b2\" (UID: \"6d75560c-3f13-4f05-9e64-b78fdb08e49c\") " pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-wk9b2" Nov 02 06:33:00.734252 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:00.734235 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d75560c-3f13-4f05-9e64-b78fdb08e49c-istio-podinfo\") pod \"e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-wk9b2\" (UID: \"6d75560c-3f13-4f05-9e64-b78fdb08e49c\") " pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-wk9b2" Nov 02 06:33:00.785191 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:00.785158 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d75560c-3f13-4f05-9e64-b78fdb08e49c-istio-token\") pod \"e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-wk9b2\" (UID: \"6d75560c-3f13-4f05-9e64-b78fdb08e49c\") " pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-wk9b2" Nov 02 06:33:00.785359 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:00.785270 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26kdr\" (UniqueName: \"kubernetes.io/projected/6d75560c-3f13-4f05-9e64-b78fdb08e49c-kube-api-access-26kdr\") pod \"e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-wk9b2\" (UID: \"6d75560c-3f13-4f05-9e64-b78fdb08e49c\") " pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-wk9b2" Nov 02 06:33:00.832515 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:00.832420 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff33efe1-03ac-4157-acda-a302dd262e10-maas-api-xks-ca\") pod \"maas-api-revoke-sub-baa3f47a-mhvfc\" (UID: \"ff33efe1-03ac-4157-acda-a302dd262e10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-baa3f47a-mhvfc" Nov 02 06:33:00.832515 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:00.832507 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff33efe1-03ac-4157-acda-a302dd262e10-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-baa3f47a-mhvfc\" (UID: \"ff33efe1-03ac-4157-acda-a302dd262e10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-baa3f47a-mhvfc" Nov 02 06:33:00.832761 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:00.832592 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdxtl\" (UniqueName: \"kubernetes.io/projected/ff33efe1-03ac-4157-acda-a302dd262e10-kube-api-access-xdxtl\") pod \"maas-api-revoke-sub-baa3f47a-mhvfc\" (UID: \"ff33efe1-03ac-4157-acda-a302dd262e10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-baa3f47a-mhvfc" Nov 02 06:33:00.833296 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:00.833268 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff33efe1-03ac-4157-acda-a302dd262e10-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-baa3f47a-mhvfc\" (UID: \"ff33efe1-03ac-4157-acda-a302dd262e10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-baa3f47a-mhvfc" Nov 02 06:33:00.835179 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:00.835154 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff33efe1-03ac-4157-acda-a302dd262e10-maas-api-xks-ca\") pod \"maas-api-revoke-sub-baa3f47a-mhvfc\" (UID: \"ff33efe1-03ac-4157-acda-a302dd262e10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-baa3f47a-mhvfc" Nov 02 06:33:00.877605 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:00.877571 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-wk9b2" Nov 02 06:33:00.886449 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:00.886393 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdxtl\" (UniqueName: \"kubernetes.io/projected/ff33efe1-03ac-4157-acda-a302dd262e10-kube-api-access-xdxtl\") pod \"maas-api-revoke-sub-baa3f47a-mhvfc\" (UID: \"ff33efe1-03ac-4157-acda-a302dd262e10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-baa3f47a-mhvfc" Nov 02 06:33:00.919613 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:00.918383 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-baa3f47a-mhvfc" Nov 02 06:33:01.322695 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:01.322098 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-baa3f47a-mhvfc"] Nov 02 06:33:01.327467 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:01.326490 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-wk9b2"] Nov 02 06:33:01.329887 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:33:01.329858 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d75560c_3f13_4f05_9e64_b78fdb08e49c.slice/crio-afd48f9f92b96ad905bc784179cbf5cd8d9b65e4d5a15221c2ef81b5e1aa90c2 WatchSource:0}: Error finding container afd48f9f92b96ad905bc784179cbf5cd8d9b65e4d5a15221c2ef81b5e1aa90c2: Status 404 returned error can't find the container with id afd48f9f92b96ad905bc784179cbf5cd8d9b65e4d5a15221c2ef81b5e1aa90c2 Nov 02 06:33:01.332810 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:01.332718 5916 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 06:33:01.332810 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:01.332774 5916 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 06:33:01.332810 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:01.332806 5916 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 06:33:02.289800 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:02.289757 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-baa3f47a-mhvfc" event={"ID":"ff33efe1-03ac-4157-acda-a302dd262e10","Type":"ContainerStarted","Data":"19a1bbf8ac6189361105f3753094e71be952e9093ea39e26b920272cb8f5917b"} Nov 02 06:33:02.290336 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:02.289830 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-baa3f47a-mhvfc" event={"ID":"ff33efe1-03ac-4157-acda-a302dd262e10","Type":"ContainerStarted","Data":"670acb44ad08a5684363b9ebae5ebe4bb9e2fd8f70619f7e3ca8f536d11de2b2"} Nov 02 06:33:02.292153 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:02.292126 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-wk9b2" event={"ID":"6d75560c-3f13-4f05-9e64-b78fdb08e49c","Type":"ContainerStarted","Data":"750beb3a20eea2be0e840f58c20768ab933dc29bf3da021ae61f7b76635e1368"} Nov 02 06:33:02.293044 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:02.292174 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-wk9b2" event={"ID":"6d75560c-3f13-4f05-9e64-b78fdb08e49c","Type":"ContainerStarted","Data":"afd48f9f92b96ad905bc784179cbf5cd8d9b65e4d5a15221c2ef81b5e1aa90c2"} Nov 02 06:33:02.414608 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:02.414543 5916 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-baa3f47a-mhvfc" podStartSLOduration=2.414524551 podStartE2EDuration="2.414524551s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:33:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:33:02.410213004 +0000 UTC m=+970.682033400" watchObservedRunningTime="2026-11-02 06:33:02.414524551 +0000 UTC m=+970.686344943" Nov 02 06:33:02.538930 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:02.530771 5916 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-wk9b2" podStartSLOduration=2.530751935 podStartE2EDuration="2.530751935s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:33:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:33:02.528784508 +0000 UTC m=+970.800604903" watchObservedRunningTime="2026-11-02 06:33:02.530751935 +0000 UTC m=+970.802572330" Nov 02 06:33:02.882928 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:02.879185 5916 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-wk9b2" Nov 02 06:33:02.887935 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:02.887034 5916 patch_prober.go:29] interesting pod/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-wk9b2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 02 06:33:02.887935 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:02.887091 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-wk9b2" podUID="6d75560c-3f13-4f05-9e64-b78fdb08e49c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 02 06:33:03.301963 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:03.301865 5916 generic.go:356] "Generic (PLEG): container finished" podID="ff33efe1-03ac-4157-acda-a302dd262e10" containerID="19a1bbf8ac6189361105f3753094e71be952e9093ea39e26b920272cb8f5917b" exitCode=0 Nov 02 06:33:03.302463 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:03.302026 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-baa3f47a-mhvfc" event={"ID":"ff33efe1-03ac-4157-acda-a302dd262e10","Type":"ContainerDied","Data":"19a1bbf8ac6189361105f3753094e71be952e9093ea39e26b920272cb8f5917b"} Nov 02 06:33:03.881524 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:03.879287 5916 patch_prober.go:29] interesting pod/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-wk9b2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 02 06:33:03.881524 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:03.879348 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-wk9b2" podUID="6d75560c-3f13-4f05-9e64-b78fdb08e49c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 02 06:33:04.578338 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:04.578273 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-baa3f47a-mhvfc" Nov 02 06:33:04.699935 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:04.699862 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff33efe1-03ac-4157-acda-a302dd262e10-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff33efe1-03ac-4157-acda-a302dd262e10-maas-api-xks-ca\") pod \"ff33efe1-03ac-4157-acda-a302dd262e10\" (UID: \"ff33efe1-03ac-4157-acda-a302dd262e10\") " Nov 02 06:33:04.700127 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:04.699957 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff33efe1-03ac-4157-acda-a302dd262e10-kube-api-access-xdxtl\" (UniqueName: \"kubernetes.io/projected/ff33efe1-03ac-4157-acda-a302dd262e10-kube-api-access-xdxtl\") pod \"ff33efe1-03ac-4157-acda-a302dd262e10\" (UID: \"ff33efe1-03ac-4157-acda-a302dd262e10\") " Nov 02 06:33:04.700127 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:04.700035 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff33efe1-03ac-4157-acda-a302dd262e10-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff33efe1-03ac-4157-acda-a302dd262e10-maas-api-openshift-ca\") pod \"ff33efe1-03ac-4157-acda-a302dd262e10\" (UID: \"ff33efe1-03ac-4157-acda-a302dd262e10\") " Nov 02 06:33:04.700936 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:04.700522 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff33efe1-03ac-4157-acda-a302dd262e10-maas-api-openshift-ca" pod "ff33efe1-03ac-4157-acda-a302dd262e10" (UID: "ff33efe1-03ac-4157-acda-a302dd262e10"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:33:04.702566 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:04.702534 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff33efe1-03ac-4157-acda-a302dd262e10-maas-api-xks-ca" pod "ff33efe1-03ac-4157-acda-a302dd262e10" (UID: "ff33efe1-03ac-4157-acda-a302dd262e10"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:33:04.703565 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:04.703545 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff33efe1-03ac-4157-acda-a302dd262e10-kube-api-access-xdxtl" pod "ff33efe1-03ac-4157-acda-a302dd262e10" (UID: "ff33efe1-03ac-4157-acda-a302dd262e10"). InnerVolumeSpecName "kube-api-access-xdxtl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:33:04.801778 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:04.801738 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff33efe1-03ac-4157-acda-a302dd262e10-maas-api-openshift-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:33:04.801778 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:04.801771 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff33efe1-03ac-4157-acda-a302dd262e10-maas-api-xks-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:33:04.801778 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:04.801781 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdxtl\" (UniqueName: \"kubernetes.io/projected/ff33efe1-03ac-4157-acda-a302dd262e10-kube-api-access-xdxtl\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:33:04.880423 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:04.880336 5916 patch_prober.go:29] interesting pod/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-wk9b2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 02 06:33:04.881020 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:04.880628 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-wk9b2" podUID="6d75560c-3f13-4f05-9e64-b78fdb08e49c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 02 06:33:05.320961 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:05.320925 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-baa3f47a-mhvfc" Nov 02 06:33:05.321372 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:05.321227 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-baa3f47a-mhvfc" event={"ID":"ff33efe1-03ac-4157-acda-a302dd262e10","Type":"ContainerDied","Data":"670acb44ad08a5684363b9ebae5ebe4bb9e2fd8f70619f7e3ca8f536d11de2b2"} Nov 02 06:33:05.321372 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:05.321276 5916 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="670acb44ad08a5684363b9ebae5ebe4bb9e2fd8f70619f7e3ca8f536d11de2b2" Nov 02 06:33:05.880387 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:05.880347 5916 patch_prober.go:29] interesting pod/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-wk9b2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 02 06:33:05.881024 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:05.880990 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-wk9b2" podUID="6d75560c-3f13-4f05-9e64-b78fdb08e49c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 02 06:33:06.878820 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:06.878779 5916 patch_prober.go:29] interesting pod/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-wk9b2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 02 06:33:06.879053 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:06.878851 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-wk9b2" podUID="6d75560c-3f13-4f05-9e64-b78fdb08e49c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 02 06:33:07.946731 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:07.946697 5916 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-wk9b2" Nov 02 06:33:07.947334 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:07.947311 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-wk9b2" Nov 02 06:33:07.948287 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:07.948261 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-wk9b2" Nov 02 06:33:11.033161 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:11.033128 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-bdcd80/distinct-w0-bdcd80-kserve-5d9df79d48-bn5d4"] Nov 02 06:33:11.033643 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:11.033629 5916 cpu_manager.go:414] "containerMap: removing container" podUID="ff33efe1-03ac-4157-acda-a302dd262e10" containerName="revoke-keys" Nov 02 06:33:11.033738 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:11.033645 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff33efe1-03ac-4157-acda-a302dd262e10" containerName="revoke-keys" Nov 02 06:33:11.033779 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:11.033744 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff33efe1-03ac-4157-acda-a302dd262e10" containerName="revoke-keys" Nov 02 06:33:11.038688 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:11.038668 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-bdcd80/distinct-w0-bdcd80-kserve-5d9df79d48-bn5d4" Nov 02 06:33:11.089673 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:11.089638 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-bdcd80/distinct-w0-bdcd80-kserve-5d9df79d48-bn5d4"] Nov 02 06:33:11.131724 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:11.131691 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-bdcd80\"/\"distinct-w0-bdcd80-kserve-self-signed-certs\"" Nov 02 06:33:11.165100 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:11.165064 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a7c7a03-e0cf-44ca-8eee-6c3e16112b3a-home\") pod \"distinct-w0-bdcd80-kserve-5d9df79d48-bn5d4\" (UID: \"1a7c7a03-e0cf-44ca-8eee-6c3e16112b3a\") " pod="e2e-models-e2e-worker-w0-bdcd80/distinct-w0-bdcd80-kserve-5d9df79d48-bn5d4" Nov 02 06:33:11.165264 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:11.165173 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a7c7a03-e0cf-44ca-8eee-6c3e16112b3a-dshm\") pod \"distinct-w0-bdcd80-kserve-5d9df79d48-bn5d4\" (UID: \"1a7c7a03-e0cf-44ca-8eee-6c3e16112b3a\") " pod="e2e-models-e2e-worker-w0-bdcd80/distinct-w0-bdcd80-kserve-5d9df79d48-bn5d4" Nov 02 06:33:11.165264 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:11.165236 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rc6r5\" (UniqueName: \"kubernetes.io/projected/1a7c7a03-e0cf-44ca-8eee-6c3e16112b3a-kube-api-access-rc6r5\") pod \"distinct-w0-bdcd80-kserve-5d9df79d48-bn5d4\" (UID: \"1a7c7a03-e0cf-44ca-8eee-6c3e16112b3a\") " pod="e2e-models-e2e-worker-w0-bdcd80/distinct-w0-bdcd80-kserve-5d9df79d48-bn5d4" Nov 02 06:33:11.165342 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:11.165286 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a7c7a03-e0cf-44ca-8eee-6c3e16112b3a-tls-certs\") pod \"distinct-w0-bdcd80-kserve-5d9df79d48-bn5d4\" (UID: \"1a7c7a03-e0cf-44ca-8eee-6c3e16112b3a\") " pod="e2e-models-e2e-worker-w0-bdcd80/distinct-w0-bdcd80-kserve-5d9df79d48-bn5d4" Nov 02 06:33:11.165342 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:11.165331 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a7c7a03-e0cf-44ca-8eee-6c3e16112b3a-model-cache\") pod \"distinct-w0-bdcd80-kserve-5d9df79d48-bn5d4\" (UID: \"1a7c7a03-e0cf-44ca-8eee-6c3e16112b3a\") " pod="e2e-models-e2e-worker-w0-bdcd80/distinct-w0-bdcd80-kserve-5d9df79d48-bn5d4" Nov 02 06:33:11.165452 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:11.165385 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a7c7a03-e0cf-44ca-8eee-6c3e16112b3a-tmp-dir\") pod \"distinct-w0-bdcd80-kserve-5d9df79d48-bn5d4\" (UID: \"1a7c7a03-e0cf-44ca-8eee-6c3e16112b3a\") " pod="e2e-models-e2e-worker-w0-bdcd80/distinct-w0-bdcd80-kserve-5d9df79d48-bn5d4" Nov 02 06:33:11.266434 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:11.266402 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a7c7a03-e0cf-44ca-8eee-6c3e16112b3a-tmp-dir\") pod \"distinct-w0-bdcd80-kserve-5d9df79d48-bn5d4\" (UID: \"1a7c7a03-e0cf-44ca-8eee-6c3e16112b3a\") " pod="e2e-models-e2e-worker-w0-bdcd80/distinct-w0-bdcd80-kserve-5d9df79d48-bn5d4" Nov 02 06:33:11.266623 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:11.266500 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a7c7a03-e0cf-44ca-8eee-6c3e16112b3a-home\") pod \"distinct-w0-bdcd80-kserve-5d9df79d48-bn5d4\" (UID: \"1a7c7a03-e0cf-44ca-8eee-6c3e16112b3a\") " pod="e2e-models-e2e-worker-w0-bdcd80/distinct-w0-bdcd80-kserve-5d9df79d48-bn5d4" Nov 02 06:33:11.266623 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:11.266537 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a7c7a03-e0cf-44ca-8eee-6c3e16112b3a-dshm\") pod \"distinct-w0-bdcd80-kserve-5d9df79d48-bn5d4\" (UID: \"1a7c7a03-e0cf-44ca-8eee-6c3e16112b3a\") " pod="e2e-models-e2e-worker-w0-bdcd80/distinct-w0-bdcd80-kserve-5d9df79d48-bn5d4" Nov 02 06:33:11.266623 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:11.266572 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rc6r5\" (UniqueName: \"kubernetes.io/projected/1a7c7a03-e0cf-44ca-8eee-6c3e16112b3a-kube-api-access-rc6r5\") pod \"distinct-w0-bdcd80-kserve-5d9df79d48-bn5d4\" (UID: \"1a7c7a03-e0cf-44ca-8eee-6c3e16112b3a\") " pod="e2e-models-e2e-worker-w0-bdcd80/distinct-w0-bdcd80-kserve-5d9df79d48-bn5d4" Nov 02 06:33:11.266623 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:11.266603 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a7c7a03-e0cf-44ca-8eee-6c3e16112b3a-tls-certs\") pod \"distinct-w0-bdcd80-kserve-5d9df79d48-bn5d4\" (UID: \"1a7c7a03-e0cf-44ca-8eee-6c3e16112b3a\") " pod="e2e-models-e2e-worker-w0-bdcd80/distinct-w0-bdcd80-kserve-5d9df79d48-bn5d4" Nov 02 06:33:11.266842 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:11.266634 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a7c7a03-e0cf-44ca-8eee-6c3e16112b3a-model-cache\") pod \"distinct-w0-bdcd80-kserve-5d9df79d48-bn5d4\" (UID: \"1a7c7a03-e0cf-44ca-8eee-6c3e16112b3a\") " pod="e2e-models-e2e-worker-w0-bdcd80/distinct-w0-bdcd80-kserve-5d9df79d48-bn5d4" Nov 02 06:33:11.266900 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:11.266838 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a7c7a03-e0cf-44ca-8eee-6c3e16112b3a-tmp-dir\") pod \"distinct-w0-bdcd80-kserve-5d9df79d48-bn5d4\" (UID: \"1a7c7a03-e0cf-44ca-8eee-6c3e16112b3a\") " pod="e2e-models-e2e-worker-w0-bdcd80/distinct-w0-bdcd80-kserve-5d9df79d48-bn5d4" Nov 02 06:33:11.267011 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:11.266988 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a7c7a03-e0cf-44ca-8eee-6c3e16112b3a-home\") pod \"distinct-w0-bdcd80-kserve-5d9df79d48-bn5d4\" (UID: \"1a7c7a03-e0cf-44ca-8eee-6c3e16112b3a\") " pod="e2e-models-e2e-worker-w0-bdcd80/distinct-w0-bdcd80-kserve-5d9df79d48-bn5d4" Nov 02 06:33:11.267101 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:11.267080 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a7c7a03-e0cf-44ca-8eee-6c3e16112b3a-model-cache\") pod \"distinct-w0-bdcd80-kserve-5d9df79d48-bn5d4\" (UID: \"1a7c7a03-e0cf-44ca-8eee-6c3e16112b3a\") " pod="e2e-models-e2e-worker-w0-bdcd80/distinct-w0-bdcd80-kserve-5d9df79d48-bn5d4" Nov 02 06:33:11.269060 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:11.269035 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a7c7a03-e0cf-44ca-8eee-6c3e16112b3a-dshm\") pod \"distinct-w0-bdcd80-kserve-5d9df79d48-bn5d4\" (UID: \"1a7c7a03-e0cf-44ca-8eee-6c3e16112b3a\") " pod="e2e-models-e2e-worker-w0-bdcd80/distinct-w0-bdcd80-kserve-5d9df79d48-bn5d4" Nov 02 06:33:11.270217 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:11.270194 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a7c7a03-e0cf-44ca-8eee-6c3e16112b3a-tls-certs\") pod \"distinct-w0-bdcd80-kserve-5d9df79d48-bn5d4\" (UID: \"1a7c7a03-e0cf-44ca-8eee-6c3e16112b3a\") " pod="e2e-models-e2e-worker-w0-bdcd80/distinct-w0-bdcd80-kserve-5d9df79d48-bn5d4" Nov 02 06:33:11.320159 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:11.320083 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rc6r5\" (UniqueName: \"kubernetes.io/projected/1a7c7a03-e0cf-44ca-8eee-6c3e16112b3a-kube-api-access-rc6r5\") pod \"distinct-w0-bdcd80-kserve-5d9df79d48-bn5d4\" (UID: \"1a7c7a03-e0cf-44ca-8eee-6c3e16112b3a\") " pod="e2e-models-e2e-worker-w0-bdcd80/distinct-w0-bdcd80-kserve-5d9df79d48-bn5d4" Nov 02 06:33:11.351311 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:11.351285 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-bdcd80/distinct-w0-bdcd80-kserve-5d9df79d48-bn5d4" Nov 02 06:33:11.775464 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:11.775427 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-bdcd80/distinct-w0-bdcd80-kserve-5d9df79d48-bn5d4"] Nov 02 06:33:12.425369 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:12.425327 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-bdcd80/distinct-w0-bdcd80-kserve-5d9df79d48-bn5d4" event={"ID":"1a7c7a03-e0cf-44ca-8eee-6c3e16112b3a","Type":"ContainerStarted","Data":"b217ac5694f6fcc460c824d3050dbdc22c73771f5122ce3909539b5d333e98f5"} Nov 02 06:33:12.425842 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:12.425381 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-bdcd80/distinct-w0-bdcd80-kserve-5d9df79d48-bn5d4" event={"ID":"1a7c7a03-e0cf-44ca-8eee-6c3e16112b3a","Type":"ContainerStarted","Data":"6e34c8f347ae1df192fbcaaea6847044f0477d7133ddae164eff387725face5d"} Nov 02 06:33:12.545636 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:12.545582 5916 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-bdcd80/distinct-w0-bdcd80-kserve-5d9df79d48-bn5d4" podStartSLOduration=2.545570314 podStartE2EDuration="2.545570314s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:33:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:33:12.543076216 +0000 UTC m=+980.814896612" watchObservedRunningTime="2026-11-02 06:33:12.545570314 +0000 UTC m=+980.817390706" Nov 02 06:33:13.352182 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:13.352136 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-bdcd80/distinct-w0-bdcd80-kserve-5d9df79d48-bn5d4" Nov 02 06:33:19.893713 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:33:19.893676 5916 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p98cq" containerName="extractor" Nov 02 06:33:21.352110 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:21.352070 5916 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-bdcd80/distinct-w0-bdcd80-kserve-5d9df79d48-bn5d4" Nov 02 06:33:21.365591 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:21.365560 5916 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-bdcd80/distinct-w0-bdcd80-kserve-5d9df79d48-bn5d4" Nov 02 06:33:21.510460 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:21.510427 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-bdcd80/distinct-w0-bdcd80-kserve-5d9df79d48-bn5d4" Nov 02 06:33:45.307304 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:45.307221 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gbxhl"] Nov 02 06:33:45.311198 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:45.311178 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gbxhl" Nov 02 06:33:45.361722 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:45.361689 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-fb1398-openshift-default-dockercfg-rn84l\"" Nov 02 06:33:45.411586 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:45.411550 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gbxhl"] Nov 02 06:33:45.451260 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:45.451229 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/807aa567-b901-4c00-a31d-198216fe94d8-istio-data\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gbxhl\" (UID: \"807aa567-b901-4c00-a31d-198216fe94d8\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gbxhl" Nov 02 06:33:45.451442 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:45.451282 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/807aa567-b901-4c00-a31d-198216fe94d8-istiod-ca-cert\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gbxhl\" (UID: \"807aa567-b901-4c00-a31d-198216fe94d8\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gbxhl" Nov 02 06:33:45.451442 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:45.451302 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s972w\" (UniqueName: \"kubernetes.io/projected/807aa567-b901-4c00-a31d-198216fe94d8-kube-api-access-s972w\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gbxhl\" (UID: \"807aa567-b901-4c00-a31d-198216fe94d8\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gbxhl" Nov 02 06:33:45.451442 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:45.451411 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/807aa567-b901-4c00-a31d-198216fe94d8-istio-podinfo\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gbxhl\" (UID: \"807aa567-b901-4c00-a31d-198216fe94d8\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gbxhl" Nov 02 06:33:45.451610 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:45.451456 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/807aa567-b901-4c00-a31d-198216fe94d8-istio-token\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gbxhl\" (UID: \"807aa567-b901-4c00-a31d-198216fe94d8\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gbxhl" Nov 02 06:33:45.451610 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:45.451518 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/807aa567-b901-4c00-a31d-198216fe94d8-workload-certs\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gbxhl\" (UID: \"807aa567-b901-4c00-a31d-198216fe94d8\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gbxhl" Nov 02 06:33:45.451610 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:45.451586 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/807aa567-b901-4c00-a31d-198216fe94d8-credential-socket\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gbxhl\" (UID: \"807aa567-b901-4c00-a31d-198216fe94d8\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gbxhl" Nov 02 06:33:45.451755 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:45.451711 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/807aa567-b901-4c00-a31d-198216fe94d8-istio-envoy\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gbxhl\" (UID: \"807aa567-b901-4c00-a31d-198216fe94d8\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gbxhl" Nov 02 06:33:45.451807 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:45.451766 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/807aa567-b901-4c00-a31d-198216fe94d8-workload-socket\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gbxhl\" (UID: \"807aa567-b901-4c00-a31d-198216fe94d8\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gbxhl" Nov 02 06:33:45.552599 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:45.552568 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/807aa567-b901-4c00-a31d-198216fe94d8-workload-socket\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gbxhl\" (UID: \"807aa567-b901-4c00-a31d-198216fe94d8\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gbxhl" Nov 02 06:33:45.552799 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:45.552607 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/807aa567-b901-4c00-a31d-198216fe94d8-istio-data\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gbxhl\" (UID: \"807aa567-b901-4c00-a31d-198216fe94d8\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gbxhl" Nov 02 06:33:45.552799 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:45.552654 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/807aa567-b901-4c00-a31d-198216fe94d8-istiod-ca-cert\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gbxhl\" (UID: \"807aa567-b901-4c00-a31d-198216fe94d8\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gbxhl" Nov 02 06:33:45.552799 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:45.552679 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s972w\" (UniqueName: \"kubernetes.io/projected/807aa567-b901-4c00-a31d-198216fe94d8-kube-api-access-s972w\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gbxhl\" (UID: \"807aa567-b901-4c00-a31d-198216fe94d8\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gbxhl" Nov 02 06:33:45.552799 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:45.552710 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/807aa567-b901-4c00-a31d-198216fe94d8-istio-podinfo\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gbxhl\" (UID: \"807aa567-b901-4c00-a31d-198216fe94d8\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gbxhl" Nov 02 06:33:45.552799 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:45.552750 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/807aa567-b901-4c00-a31d-198216fe94d8-istio-token\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gbxhl\" (UID: \"807aa567-b901-4c00-a31d-198216fe94d8\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gbxhl" Nov 02 06:33:45.553106 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:45.552948 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/807aa567-b901-4c00-a31d-198216fe94d8-workload-certs\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gbxhl\" (UID: \"807aa567-b901-4c00-a31d-198216fe94d8\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gbxhl" Nov 02 06:33:45.553106 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:45.553014 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/807aa567-b901-4c00-a31d-198216fe94d8-credential-socket\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gbxhl\" (UID: \"807aa567-b901-4c00-a31d-198216fe94d8\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gbxhl" Nov 02 06:33:45.553106 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:45.553077 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/807aa567-b901-4c00-a31d-198216fe94d8-workload-socket\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gbxhl\" (UID: \"807aa567-b901-4c00-a31d-198216fe94d8\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gbxhl" Nov 02 06:33:45.553249 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:45.553124 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/807aa567-b901-4c00-a31d-198216fe94d8-istio-data\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gbxhl\" (UID: \"807aa567-b901-4c00-a31d-198216fe94d8\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gbxhl" Nov 02 06:33:45.553249 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:45.553130 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/807aa567-b901-4c00-a31d-198216fe94d8-istio-envoy\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gbxhl\" (UID: \"807aa567-b901-4c00-a31d-198216fe94d8\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gbxhl" Nov 02 06:33:45.553314 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:45.553296 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/807aa567-b901-4c00-a31d-198216fe94d8-workload-certs\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gbxhl\" (UID: \"807aa567-b901-4c00-a31d-198216fe94d8\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gbxhl" Nov 02 06:33:45.553379 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:45.553358 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/807aa567-b901-4c00-a31d-198216fe94d8-credential-socket\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gbxhl\" (UID: \"807aa567-b901-4c00-a31d-198216fe94d8\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gbxhl" Nov 02 06:33:45.553474 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:45.553460 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/807aa567-b901-4c00-a31d-198216fe94d8-istiod-ca-cert\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gbxhl\" (UID: \"807aa567-b901-4c00-a31d-198216fe94d8\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gbxhl" Nov 02 06:33:45.555122 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:45.555101 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/807aa567-b901-4c00-a31d-198216fe94d8-istio-envoy\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gbxhl\" (UID: \"807aa567-b901-4c00-a31d-198216fe94d8\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gbxhl" Nov 02 06:33:45.555330 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:45.555314 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/807aa567-b901-4c00-a31d-198216fe94d8-istio-podinfo\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gbxhl\" (UID: \"807aa567-b901-4c00-a31d-198216fe94d8\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gbxhl" Nov 02 06:33:45.605658 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:45.605596 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/807aa567-b901-4c00-a31d-198216fe94d8-istio-token\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gbxhl\" (UID: \"807aa567-b901-4c00-a31d-198216fe94d8\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gbxhl" Nov 02 06:33:45.605758 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:45.605722 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s972w\" (UniqueName: \"kubernetes.io/projected/807aa567-b901-4c00-a31d-198216fe94d8-kube-api-access-s972w\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gbxhl\" (UID: \"807aa567-b901-4c00-a31d-198216fe94d8\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gbxhl" Nov 02 06:33:45.625309 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:45.625286 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gbxhl" Nov 02 06:33:46.060255 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:46.058867 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gbxhl"] Nov 02 06:33:46.062402 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:33:46.062375 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod807aa567_b901_4c00_a31d_198216fe94d8.slice/crio-dea9ea43a33cb838faa16bb8567468463743868754bceedd02ce8b667fff0903 WatchSource:0}: Error finding container dea9ea43a33cb838faa16bb8567468463743868754bceedd02ce8b667fff0903: Status 404 returned error can't find the container with id dea9ea43a33cb838faa16bb8567468463743868754bceedd02ce8b667fff0903 Nov 02 06:33:46.064633 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:46.064601 5916 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 06:33:46.064708 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:46.064667 5916 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 06:33:46.064708 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:46.064695 5916 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 06:33:46.713895 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:46.713853 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gbxhl" event={"ID":"807aa567-b901-4c00-a31d-198216fe94d8","Type":"ContainerStarted","Data":"1e11a229f24f64cce7469d4adb32c5b0140d66d494966db2cee5a02ad524b41f"} Nov 02 06:33:46.713895 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:46.713924 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gbxhl" event={"ID":"807aa567-b901-4c00-a31d-198216fe94d8","Type":"ContainerStarted","Data":"dea9ea43a33cb838faa16bb8567468463743868754bceedd02ce8b667fff0903"} Nov 02 06:33:46.839439 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:46.825101 5916 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gbxhl" podStartSLOduration=1.8250835300000001 podStartE2EDuration="1.82508353s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:33:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:33:46.823221807 +0000 UTC m=+1015.095042201" watchObservedRunningTime="2026-11-02 06:33:46.82508353 +0000 UTC m=+1015.096903924" Nov 02 06:33:47.629023 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:47.628991 5916 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gbxhl" Nov 02 06:33:47.629387 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:47.629363 5916 patch_prober.go:29] interesting pod/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gbxhl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Nov 02 06:33:47.629536 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:47.629511 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gbxhl" podUID="807aa567-b901-4c00-a31d-198216fe94d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Nov 02 06:33:48.640930 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:48.630263 5916 patch_prober.go:29] interesting pod/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gbxhl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Nov 02 06:33:48.640930 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:48.630330 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gbxhl" podUID="807aa567-b901-4c00-a31d-198216fe94d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Nov 02 06:33:49.632930 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:49.629228 5916 patch_prober.go:29] interesting pod/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gbxhl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Nov 02 06:33:49.632930 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:49.629298 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gbxhl" podUID="807aa567-b901-4c00-a31d-198216fe94d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Nov 02 06:33:49.893133 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:33:49.893099 5916 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p98cq" containerName="extractor" Nov 02 06:33:50.629281 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:50.629244 5916 patch_prober.go:29] interesting pod/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gbxhl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Nov 02 06:33:50.629591 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:50.629556 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gbxhl" podUID="807aa567-b901-4c00-a31d-198216fe94d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Nov 02 06:33:51.629928 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:51.629240 5916 patch_prober.go:29] interesting pod/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gbxhl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Nov 02 06:33:51.629928 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:51.629311 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gbxhl" podUID="807aa567-b901-4c00-a31d-198216fe94d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Nov 02 06:33:52.882638 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:52.882602 5916 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gbxhl" Nov 02 06:33:52.883345 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:52.883325 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gbxhl" Nov 02 06:33:52.884067 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:33:52.884047 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gbxhl" Nov 02 06:34:10.037251 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:34:10.037221 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-581608/distinct-w2-581608-kserve-6bb6bb7b65-v7kfj"] Nov 02 06:34:10.041733 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:34:10.041711 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-581608/distinct-w2-581608-kserve-6bb6bb7b65-v7kfj" Nov 02 06:34:10.108773 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:34:10.108743 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-581608/distinct-w2-581608-kserve-6bb6bb7b65-v7kfj"] Nov 02 06:34:10.124284 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:34:10.124260 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dcf2f562-3c50-4de7-9c84-cb659d3bc25b-dshm\") pod \"distinct-w2-581608-kserve-6bb6bb7b65-v7kfj\" (UID: \"dcf2f562-3c50-4de7-9c84-cb659d3bc25b\") " pod="e2e-models-e2e-worker-w2-581608/distinct-w2-581608-kserve-6bb6bb7b65-v7kfj" Nov 02 06:34:10.124406 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:34:10.124294 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dcf2f562-3c50-4de7-9c84-cb659d3bc25b-home\") pod \"distinct-w2-581608-kserve-6bb6bb7b65-v7kfj\" (UID: \"dcf2f562-3c50-4de7-9c84-cb659d3bc25b\") " pod="e2e-models-e2e-worker-w2-581608/distinct-w2-581608-kserve-6bb6bb7b65-v7kfj" Nov 02 06:34:10.124459 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:34:10.124437 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dcf2f562-3c50-4de7-9c84-cb659d3bc25b-tls-certs\") pod \"distinct-w2-581608-kserve-6bb6bb7b65-v7kfj\" (UID: \"dcf2f562-3c50-4de7-9c84-cb659d3bc25b\") " pod="e2e-models-e2e-worker-w2-581608/distinct-w2-581608-kserve-6bb6bb7b65-v7kfj" Nov 02 06:34:10.124499 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:34:10.124488 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dcf2f562-3c50-4de7-9c84-cb659d3bc25b-tmp-dir\") pod \"distinct-w2-581608-kserve-6bb6bb7b65-v7kfj\" (UID: \"dcf2f562-3c50-4de7-9c84-cb659d3bc25b\") " pod="e2e-models-e2e-worker-w2-581608/distinct-w2-581608-kserve-6bb6bb7b65-v7kfj" Nov 02 06:34:10.124580 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:34:10.124568 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4j8g2\" (UniqueName: \"kubernetes.io/projected/dcf2f562-3c50-4de7-9c84-cb659d3bc25b-kube-api-access-4j8g2\") pod \"distinct-w2-581608-kserve-6bb6bb7b65-v7kfj\" (UID: \"dcf2f562-3c50-4de7-9c84-cb659d3bc25b\") " pod="e2e-models-e2e-worker-w2-581608/distinct-w2-581608-kserve-6bb6bb7b65-v7kfj" Nov 02 06:34:10.124615 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:34:10.124589 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dcf2f562-3c50-4de7-9c84-cb659d3bc25b-model-cache\") pod \"distinct-w2-581608-kserve-6bb6bb7b65-v7kfj\" (UID: \"dcf2f562-3c50-4de7-9c84-cb659d3bc25b\") " pod="e2e-models-e2e-worker-w2-581608/distinct-w2-581608-kserve-6bb6bb7b65-v7kfj" Nov 02 06:34:10.136496 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:34:10.136464 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-581608\"/\"distinct-w2-581608-kserve-self-signed-certs\"" Nov 02 06:34:10.225646 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:34:10.225607 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4j8g2\" (UniqueName: \"kubernetes.io/projected/dcf2f562-3c50-4de7-9c84-cb659d3bc25b-kube-api-access-4j8g2\") pod \"distinct-w2-581608-kserve-6bb6bb7b65-v7kfj\" (UID: \"dcf2f562-3c50-4de7-9c84-cb659d3bc25b\") " pod="e2e-models-e2e-worker-w2-581608/distinct-w2-581608-kserve-6bb6bb7b65-v7kfj" Nov 02 06:34:10.225861 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:34:10.225652 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dcf2f562-3c50-4de7-9c84-cb659d3bc25b-model-cache\") pod \"distinct-w2-581608-kserve-6bb6bb7b65-v7kfj\" (UID: \"dcf2f562-3c50-4de7-9c84-cb659d3bc25b\") " pod="e2e-models-e2e-worker-w2-581608/distinct-w2-581608-kserve-6bb6bb7b65-v7kfj" Nov 02 06:34:10.225861 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:34:10.225729 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dcf2f562-3c50-4de7-9c84-cb659d3bc25b-dshm\") pod \"distinct-w2-581608-kserve-6bb6bb7b65-v7kfj\" (UID: \"dcf2f562-3c50-4de7-9c84-cb659d3bc25b\") " pod="e2e-models-e2e-worker-w2-581608/distinct-w2-581608-kserve-6bb6bb7b65-v7kfj" Nov 02 06:34:10.225861 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:34:10.225758 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dcf2f562-3c50-4de7-9c84-cb659d3bc25b-home\") pod \"distinct-w2-581608-kserve-6bb6bb7b65-v7kfj\" (UID: \"dcf2f562-3c50-4de7-9c84-cb659d3bc25b\") " pod="e2e-models-e2e-worker-w2-581608/distinct-w2-581608-kserve-6bb6bb7b65-v7kfj" Nov 02 06:34:10.225861 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:34:10.225841 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dcf2f562-3c50-4de7-9c84-cb659d3bc25b-tls-certs\") pod \"distinct-w2-581608-kserve-6bb6bb7b65-v7kfj\" (UID: \"dcf2f562-3c50-4de7-9c84-cb659d3bc25b\") " pod="e2e-models-e2e-worker-w2-581608/distinct-w2-581608-kserve-6bb6bb7b65-v7kfj" Nov 02 06:34:10.226157 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:34:10.225892 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dcf2f562-3c50-4de7-9c84-cb659d3bc25b-tmp-dir\") pod \"distinct-w2-581608-kserve-6bb6bb7b65-v7kfj\" (UID: \"dcf2f562-3c50-4de7-9c84-cb659d3bc25b\") " pod="e2e-models-e2e-worker-w2-581608/distinct-w2-581608-kserve-6bb6bb7b65-v7kfj" Nov 02 06:34:10.226157 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:34:10.226122 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dcf2f562-3c50-4de7-9c84-cb659d3bc25b-model-cache\") pod \"distinct-w2-581608-kserve-6bb6bb7b65-v7kfj\" (UID: \"dcf2f562-3c50-4de7-9c84-cb659d3bc25b\") " pod="e2e-models-e2e-worker-w2-581608/distinct-w2-581608-kserve-6bb6bb7b65-v7kfj" Nov 02 06:34:10.226455 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:34:10.226432 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dcf2f562-3c50-4de7-9c84-cb659d3bc25b-tmp-dir\") pod \"distinct-w2-581608-kserve-6bb6bb7b65-v7kfj\" (UID: \"dcf2f562-3c50-4de7-9c84-cb659d3bc25b\") " pod="e2e-models-e2e-worker-w2-581608/distinct-w2-581608-kserve-6bb6bb7b65-v7kfj" Nov 02 06:34:10.226589 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:34:10.226565 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dcf2f562-3c50-4de7-9c84-cb659d3bc25b-home\") pod \"distinct-w2-581608-kserve-6bb6bb7b65-v7kfj\" (UID: \"dcf2f562-3c50-4de7-9c84-cb659d3bc25b\") " pod="e2e-models-e2e-worker-w2-581608/distinct-w2-581608-kserve-6bb6bb7b65-v7kfj" Nov 02 06:34:10.228989 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:34:10.228962 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dcf2f562-3c50-4de7-9c84-cb659d3bc25b-dshm\") pod \"distinct-w2-581608-kserve-6bb6bb7b65-v7kfj\" (UID: \"dcf2f562-3c50-4de7-9c84-cb659d3bc25b\") " pod="e2e-models-e2e-worker-w2-581608/distinct-w2-581608-kserve-6bb6bb7b65-v7kfj" Nov 02 06:34:10.229143 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:34:10.229129 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dcf2f562-3c50-4de7-9c84-cb659d3bc25b-tls-certs\") pod \"distinct-w2-581608-kserve-6bb6bb7b65-v7kfj\" (UID: \"dcf2f562-3c50-4de7-9c84-cb659d3bc25b\") " pod="e2e-models-e2e-worker-w2-581608/distinct-w2-581608-kserve-6bb6bb7b65-v7kfj" Nov 02 06:34:10.279824 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:34:10.279759 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4j8g2\" (UniqueName: \"kubernetes.io/projected/dcf2f562-3c50-4de7-9c84-cb659d3bc25b-kube-api-access-4j8g2\") pod \"distinct-w2-581608-kserve-6bb6bb7b65-v7kfj\" (UID: \"dcf2f562-3c50-4de7-9c84-cb659d3bc25b\") " pod="e2e-models-e2e-worker-w2-581608/distinct-w2-581608-kserve-6bb6bb7b65-v7kfj" Nov 02 06:34:10.386951 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:34:10.386892 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-581608/distinct-w2-581608-kserve-6bb6bb7b65-v7kfj" Nov 02 06:34:10.816215 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:34:10.816016 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-581608/distinct-w2-581608-kserve-6bb6bb7b65-v7kfj"] Nov 02 06:34:10.819303 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:34:10.819277 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddcf2f562_3c50_4de7_9c84_cb659d3bc25b.slice/crio-be49cc29f4950957803648d757499df4f7fc2b10ab2a8b8cb5a4ec13c3bf2997 WatchSource:0}: Error finding container be49cc29f4950957803648d757499df4f7fc2b10ab2a8b8cb5a4ec13c3bf2997: Status 404 returned error can't find the container with id be49cc29f4950957803648d757499df4f7fc2b10ab2a8b8cb5a4ec13c3bf2997 Nov 02 06:34:10.928083 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:34:10.928051 5916 generic.go:356] "Generic (PLEG): container finished" podID="e4fb4f2e-7ef9-4223-973f-494bbcb98a82" containerID="7fe13de0cea50638b78dd0af0926f417675b17e76904acc1af1a4679c9624d2f" exitCode=0 Nov 02 06:34:10.928264 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:34:10.928133 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-lvn8x-gtslh" event={"ID":"e4fb4f2e-7ef9-4223-973f-494bbcb98a82","Type":"ContainerDied","Data":"7fe13de0cea50638b78dd0af0926f417675b17e76904acc1af1a4679c9624d2f"} Nov 02 06:34:10.930008 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:34:10.929986 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-581608/distinct-w2-581608-kserve-6bb6bb7b65-v7kfj" event={"ID":"dcf2f562-3c50-4de7-9c84-cb659d3bc25b","Type":"ContainerStarted","Data":"7faa43095cc60bd7ff99255ae17c318ec6ccea9860454a6b281c5f57b2e65baf"} Nov 02 06:34:10.930106 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:34:10.930014 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-581608/distinct-w2-581608-kserve-6bb6bb7b65-v7kfj" event={"ID":"dcf2f562-3c50-4de7-9c84-cb659d3bc25b","Type":"ContainerStarted","Data":"be49cc29f4950957803648d757499df4f7fc2b10ab2a8b8cb5a4ec13c3bf2997"} Nov 02 06:34:11.141184 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:34:11.141135 5916 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-581608/distinct-w2-581608-kserve-6bb6bb7b65-v7kfj" podStartSLOduration=2.141120919 podStartE2EDuration="2.141120919s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:34:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:34:11.138971512 +0000 UTC m=+1039.410791907" watchObservedRunningTime="2026-11-02 06:34:11.141120919 +0000 UTC m=+1039.412941312" Nov 02 06:34:11.387965 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:34:11.387922 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-581608/distinct-w2-581608-kserve-6bb6bb7b65-v7kfj" Nov 02 06:34:12.095421 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:34:12.095391 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-lvn8x-gtslh" Nov 02 06:34:12.251719 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:34:12.251623 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4fb4f2e-7ef9-4223-973f-494bbcb98a82-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/e4fb4f2e-7ef9-4223-973f-494bbcb98a82-archives-path\") pod \"e4fb4f2e-7ef9-4223-973f-494bbcb98a82\" (UID: \"e4fb4f2e-7ef9-4223-973f-494bbcb98a82\") " Nov 02 06:34:12.251719 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:34:12.251710 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4fb4f2e-7ef9-4223-973f-494bbcb98a82-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4fb4f2e-7ef9-4223-973f-494bbcb98a82-service-ca-bundle\") pod \"e4fb4f2e-7ef9-4223-973f-494bbcb98a82\" (UID: \"e4fb4f2e-7ef9-4223-973f-494bbcb98a82\") " Nov 02 06:34:12.252281 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:34:12.251757 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4fb4f2e-7ef9-4223-973f-494bbcb98a82-kube-api-access-lszfj\" (UniqueName: \"kubernetes.io/projected/e4fb4f2e-7ef9-4223-973f-494bbcb98a82-kube-api-access-lszfj\") pod \"e4fb4f2e-7ef9-4223-973f-494bbcb98a82\" (UID: \"e4fb4f2e-7ef9-4223-973f-494bbcb98a82\") " Nov 02 06:34:12.252281 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:34:12.251925 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4fb4f2e-7ef9-4223-973f-494bbcb98a82-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4fb4f2e-7ef9-4223-973f-494bbcb98a82-trusted-ca-bundle\") pod \"e4fb4f2e-7ef9-4223-973f-494bbcb98a82\" (UID: \"e4fb4f2e-7ef9-4223-973f-494bbcb98a82\") " Nov 02 06:34:12.252281 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:34:12.252016 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4fb4f2e-7ef9-4223-973f-494bbcb98a82-archives-path" pod "e4fb4f2e-7ef9-4223-973f-494bbcb98a82" (UID: "e4fb4f2e-7ef9-4223-973f-494bbcb98a82"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:34:12.252281 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:34:12.252124 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4fb4f2e-7ef9-4223-973f-494bbcb98a82-service-ca-bundle" pod "e4fb4f2e-7ef9-4223-973f-494bbcb98a82" (UID: "e4fb4f2e-7ef9-4223-973f-494bbcb98a82"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:34:12.252501 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:34:12.252427 5916 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/e4fb4f2e-7ef9-4223-973f-494bbcb98a82-archives-path\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:34:12.252501 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:34:12.252452 5916 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4fb4f2e-7ef9-4223-973f-494bbcb98a82-service-ca-bundle\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:34:12.252595 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:34:12.252494 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4fb4f2e-7ef9-4223-973f-494bbcb98a82-trusted-ca-bundle" pod "e4fb4f2e-7ef9-4223-973f-494bbcb98a82" (UID: "e4fb4f2e-7ef9-4223-973f-494bbcb98a82"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:34:12.254568 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:34:12.254537 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4fb4f2e-7ef9-4223-973f-494bbcb98a82-kube-api-access-lszfj" pod "e4fb4f2e-7ef9-4223-973f-494bbcb98a82" (UID: "e4fb4f2e-7ef9-4223-973f-494bbcb98a82"). InnerVolumeSpecName "kube-api-access-lszfj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:34:12.353320 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:34:12.353281 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lszfj\" (UniqueName: \"kubernetes.io/projected/e4fb4f2e-7ef9-4223-973f-494bbcb98a82-kube-api-access-lszfj\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:34:12.353320 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:34:12.353309 5916 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4fb4f2e-7ef9-4223-973f-494bbcb98a82-trusted-ca-bundle\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:34:12.949011 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:34:12.948981 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-lvn8x-gtslh" Nov 02 06:34:12.949222 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:34:12.949026 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-lvn8x-gtslh" event={"ID":"e4fb4f2e-7ef9-4223-973f-494bbcb98a82","Type":"ContainerDied","Data":"40b31d3013011aab2807d354beb5340ceb06f052eb8e223f4faa894679b07721"} Nov 02 06:34:12.949222 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:34:12.949078 5916 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="40b31d3013011aab2807d354beb5340ceb06f052eb8e223f4faa894679b07721" Nov 02 06:34:19.893542 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:34:19.893505 5916 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p98cq" containerName="extractor" Nov 02 06:34:20.387646 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:34:20.387612 5916 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-581608/distinct-w2-581608-kserve-6bb6bb7b65-v7kfj" Nov 02 06:34:20.400286 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:34:20.400254 5916 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-581608/distinct-w2-581608-kserve-6bb6bb7b65-v7kfj" Nov 02 06:34:21.019127 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:34:21.019095 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-581608/distinct-w2-581608-kserve-6bb6bb7b65-v7kfj" Nov 02 06:34:49.893845 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:34:49.893805 5916 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p98cq" containerName="extractor" Nov 02 06:35:15.613937 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:35:15.613891 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-jr76m"] Nov 02 06:35:15.614623 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:35:15.614602 5916 cpu_manager.go:414] "containerMap: removing container" podUID="e4fb4f2e-7ef9-4223-973f-494bbcb98a82" containerName="insights-gathering" Nov 02 06:35:15.614723 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:35:15.614623 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4fb4f2e-7ef9-4223-973f-494bbcb98a82" containerName="insights-gathering" Nov 02 06:35:15.614795 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:35:15.614781 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4fb4f2e-7ef9-4223-973f-494bbcb98a82" containerName="insights-gathering" Nov 02 06:35:15.618471 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:35:15.618452 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-jr76m" Nov 02 06:35:15.667135 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:35:15.667108 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-bdcd80-openshift-default-dockercfg-h4jqn\"" Nov 02 06:35:15.717352 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:35:15.717325 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-jr76m"] Nov 02 06:35:15.749211 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:35:15.749178 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20a4dcea-b779-47d9-a38b-5ede52dc0076-istio-token\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-jr76m\" (UID: \"20a4dcea-b779-47d9-a38b-5ede52dc0076\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-jr76m" Nov 02 06:35:15.749358 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:35:15.749225 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20a4dcea-b779-47d9-a38b-5ede52dc0076-workload-socket\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-jr76m\" (UID: \"20a4dcea-b779-47d9-a38b-5ede52dc0076\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-jr76m" Nov 02 06:35:15.749358 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:35:15.749249 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20a4dcea-b779-47d9-a38b-5ede52dc0076-workload-certs\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-jr76m\" (UID: \"20a4dcea-b779-47d9-a38b-5ede52dc0076\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-jr76m" Nov 02 06:35:15.749358 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:35:15.749290 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20a4dcea-b779-47d9-a38b-5ede52dc0076-istiod-ca-cert\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-jr76m\" (UID: \"20a4dcea-b779-47d9-a38b-5ede52dc0076\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-jr76m" Nov 02 06:35:15.749484 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:35:15.749394 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20a4dcea-b779-47d9-a38b-5ede52dc0076-istio-data\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-jr76m\" (UID: \"20a4dcea-b779-47d9-a38b-5ede52dc0076\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-jr76m" Nov 02 06:35:15.749484 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:35:15.749433 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20a4dcea-b779-47d9-a38b-5ede52dc0076-istio-envoy\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-jr76m\" (UID: \"20a4dcea-b779-47d9-a38b-5ede52dc0076\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-jr76m" Nov 02 06:35:15.749614 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:35:15.749592 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20a4dcea-b779-47d9-a38b-5ede52dc0076-istio-podinfo\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-jr76m\" (UID: \"20a4dcea-b779-47d9-a38b-5ede52dc0076\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-jr76m" Nov 02 06:35:15.749667 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:35:15.749638 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20a4dcea-b779-47d9-a38b-5ede52dc0076-credential-socket\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-jr76m\" (UID: \"20a4dcea-b779-47d9-a38b-5ede52dc0076\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-jr76m" Nov 02 06:35:15.749717 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:35:15.749703 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5d22b\" (UniqueName: \"kubernetes.io/projected/20a4dcea-b779-47d9-a38b-5ede52dc0076-kube-api-access-5d22b\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-jr76m\" (UID: \"20a4dcea-b779-47d9-a38b-5ede52dc0076\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-jr76m" Nov 02 06:35:15.850802 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:35:15.850722 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20a4dcea-b779-47d9-a38b-5ede52dc0076-istio-data\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-jr76m\" (UID: \"20a4dcea-b779-47d9-a38b-5ede52dc0076\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-jr76m" Nov 02 06:35:15.850802 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:35:15.850763 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20a4dcea-b779-47d9-a38b-5ede52dc0076-istio-envoy\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-jr76m\" (UID: \"20a4dcea-b779-47d9-a38b-5ede52dc0076\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-jr76m" Nov 02 06:35:15.851083 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:35:15.850813 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20a4dcea-b779-47d9-a38b-5ede52dc0076-istio-podinfo\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-jr76m\" (UID: \"20a4dcea-b779-47d9-a38b-5ede52dc0076\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-jr76m" Nov 02 06:35:15.851083 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:35:15.850833 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20a4dcea-b779-47d9-a38b-5ede52dc0076-credential-socket\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-jr76m\" (UID: \"20a4dcea-b779-47d9-a38b-5ede52dc0076\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-jr76m" Nov 02 06:35:15.851083 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:35:15.850865 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5d22b\" (UniqueName: \"kubernetes.io/projected/20a4dcea-b779-47d9-a38b-5ede52dc0076-kube-api-access-5d22b\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-jr76m\" (UID: \"20a4dcea-b779-47d9-a38b-5ede52dc0076\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-jr76m" Nov 02 06:35:15.851083 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:35:15.850947 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20a4dcea-b779-47d9-a38b-5ede52dc0076-istio-token\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-jr76m\" (UID: \"20a4dcea-b779-47d9-a38b-5ede52dc0076\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-jr76m" Nov 02 06:35:15.851083 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:35:15.850976 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20a4dcea-b779-47d9-a38b-5ede52dc0076-workload-socket\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-jr76m\" (UID: \"20a4dcea-b779-47d9-a38b-5ede52dc0076\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-jr76m" Nov 02 06:35:15.851083 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:35:15.851000 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20a4dcea-b779-47d9-a38b-5ede52dc0076-workload-certs\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-jr76m\" (UID: \"20a4dcea-b779-47d9-a38b-5ede52dc0076\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-jr76m" Nov 02 06:35:15.851083 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:35:15.851023 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20a4dcea-b779-47d9-a38b-5ede52dc0076-istiod-ca-cert\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-jr76m\" (UID: \"20a4dcea-b779-47d9-a38b-5ede52dc0076\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-jr76m" Nov 02 06:35:15.851444 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:35:15.851211 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20a4dcea-b779-47d9-a38b-5ede52dc0076-credential-socket\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-jr76m\" (UID: \"20a4dcea-b779-47d9-a38b-5ede52dc0076\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-jr76m" Nov 02 06:35:15.851444 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:35:15.851231 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20a4dcea-b779-47d9-a38b-5ede52dc0076-istio-data\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-jr76m\" (UID: \"20a4dcea-b779-47d9-a38b-5ede52dc0076\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-jr76m" Nov 02 06:35:15.851556 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:35:15.851475 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20a4dcea-b779-47d9-a38b-5ede52dc0076-workload-socket\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-jr76m\" (UID: \"20a4dcea-b779-47d9-a38b-5ede52dc0076\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-jr76m" Nov 02 06:35:15.851612 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:35:15.851552 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20a4dcea-b779-47d9-a38b-5ede52dc0076-workload-certs\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-jr76m\" (UID: \"20a4dcea-b779-47d9-a38b-5ede52dc0076\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-jr76m" Nov 02 06:35:15.851851 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:35:15.851831 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20a4dcea-b779-47d9-a38b-5ede52dc0076-istiod-ca-cert\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-jr76m\" (UID: \"20a4dcea-b779-47d9-a38b-5ede52dc0076\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-jr76m" Nov 02 06:35:15.853278 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:35:15.853259 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20a4dcea-b779-47d9-a38b-5ede52dc0076-istio-envoy\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-jr76m\" (UID: \"20a4dcea-b779-47d9-a38b-5ede52dc0076\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-jr76m" Nov 02 06:35:15.853405 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:35:15.853389 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20a4dcea-b779-47d9-a38b-5ede52dc0076-istio-podinfo\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-jr76m\" (UID: \"20a4dcea-b779-47d9-a38b-5ede52dc0076\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-jr76m" Nov 02 06:35:15.904096 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:35:15.904070 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20a4dcea-b779-47d9-a38b-5ede52dc0076-istio-token\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-jr76m\" (UID: \"20a4dcea-b779-47d9-a38b-5ede52dc0076\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-jr76m" Nov 02 06:35:15.904231 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:35:15.904185 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5d22b\" (UniqueName: \"kubernetes.io/projected/20a4dcea-b779-47d9-a38b-5ede52dc0076-kube-api-access-5d22b\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-jr76m\" (UID: \"20a4dcea-b779-47d9-a38b-5ede52dc0076\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-jr76m" Nov 02 06:35:15.932140 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:35:15.932113 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-jr76m" Nov 02 06:35:16.355264 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:35:16.355226 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-jr76m"] Nov 02 06:35:16.361470 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:35:16.361444 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20a4dcea_b779_47d9_a38b_5ede52dc0076.slice/crio-754948647f69d2d90c4453f2afd5676ad923a9379476c4aea74e4de6c19c052c WatchSource:0}: Error finding container 754948647f69d2d90c4453f2afd5676ad923a9379476c4aea74e4de6c19c052c: Status 404 returned error can't find the container with id 754948647f69d2d90c4453f2afd5676ad923a9379476c4aea74e4de6c19c052c Nov 02 06:35:16.363753 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:35:16.363723 5916 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 06:35:16.363836 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:35:16.363791 5916 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 06:35:16.363836 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:35:16.363818 5916 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 06:35:16.429239 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:35:16.429212 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-jr76m" event={"ID":"20a4dcea-b779-47d9-a38b-5ede52dc0076","Type":"ContainerStarted","Data":"754948647f69d2d90c4453f2afd5676ad923a9379476c4aea74e4de6c19c052c"} Nov 02 06:35:17.456674 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:35:17.456634 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-jr76m" event={"ID":"20a4dcea-b779-47d9-a38b-5ede52dc0076","Type":"ContainerStarted","Data":"2c69bf27890cf6e7f0f1459a17e5aa476b117401b7d27701f3e343659e42eeed"} Nov 02 06:35:17.935010 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:35:17.934969 5916 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-jr76m" Nov 02 06:35:17.941066 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:35:17.941037 5916 patch_prober.go:29] interesting pod/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-jr76m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Nov 02 06:35:17.941312 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:35:17.941277 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-jr76m" podUID="20a4dcea-b779-47d9-a38b-5ede52dc0076" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Nov 02 06:35:18.950296 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:35:18.950257 5916 patch_prober.go:29] interesting pod/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-jr76m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Nov 02 06:35:18.950721 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:35:18.950312 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-jr76m" podUID="20a4dcea-b779-47d9-a38b-5ede52dc0076" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Nov 02 06:35:19.894215 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:35:19.894179 5916 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p98cq" containerName="extractor" Nov 02 06:35:19.946929 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:35:19.936298 5916 patch_prober.go:29] interesting pod/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-jr76m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Nov 02 06:35:19.946929 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:35:19.936359 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-jr76m" podUID="20a4dcea-b779-47d9-a38b-5ede52dc0076" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Nov 02 06:35:20.933360 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:35:20.933311 5916 patch_prober.go:29] interesting pod/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-jr76m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Nov 02 06:35:20.934118 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:35:20.934072 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-jr76m" podUID="20a4dcea-b779-47d9-a38b-5ede52dc0076" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Nov 02 06:35:21.936357 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:35:21.936314 5916 patch_prober.go:29] interesting pod/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-jr76m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Nov 02 06:35:21.936991 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:35:21.936955 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-jr76m" podUID="20a4dcea-b779-47d9-a38b-5ede52dc0076" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Nov 02 06:35:22.932700 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:35:22.932657 5916 patch_prober.go:29] interesting pod/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-jr76m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Nov 02 06:35:22.932882 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:35:22.932730 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-jr76m" podUID="20a4dcea-b779-47d9-a38b-5ede52dc0076" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Nov 02 06:35:23.938718 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:35:23.938679 5916 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-jr76m" Nov 02 06:35:23.939936 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:35:23.939899 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-jr76m" Nov 02 06:35:23.940587 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:35:23.940571 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-jr76m" Nov 02 06:35:24.095742 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:35:24.095675 5916 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-jr76m" podStartSLOduration=9.095658318 podStartE2EDuration="9.095658318s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:35:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:35:17.662724278 +0000 UTC m=+1105.934544671" watchObservedRunningTime="2026-11-02 06:35:24.095658318 +0000 UTC m=+1112.367478714" Nov 02 06:35:24.518929 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:35:24.518890 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-fb1398/distinct-2-w3-fb1398-kserve-597b7dc99b-dp4jf"] Nov 02 06:35:24.522985 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:35:24.522968 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-fb1398/distinct-2-w3-fb1398-kserve-597b7dc99b-dp4jf" Nov 02 06:35:24.549867 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:35:24.549835 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/523a31c4-4aea-4e9f-9ec4-7c28faefbf7f-dshm\") pod \"distinct-2-w3-fb1398-kserve-597b7dc99b-dp4jf\" (UID: \"523a31c4-4aea-4e9f-9ec4-7c28faefbf7f\") " pod="e2e-models-e2e-worker-w3-fb1398/distinct-2-w3-fb1398-kserve-597b7dc99b-dp4jf" Nov 02 06:35:24.550070 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:35:24.550049 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/523a31c4-4aea-4e9f-9ec4-7c28faefbf7f-tmp-dir\") pod \"distinct-2-w3-fb1398-kserve-597b7dc99b-dp4jf\" (UID: \"523a31c4-4aea-4e9f-9ec4-7c28faefbf7f\") " pod="e2e-models-e2e-worker-w3-fb1398/distinct-2-w3-fb1398-kserve-597b7dc99b-dp4jf" Nov 02 06:35:24.550153 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:35:24.550136 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/523a31c4-4aea-4e9f-9ec4-7c28faefbf7f-tls-certs\") pod \"distinct-2-w3-fb1398-kserve-597b7dc99b-dp4jf\" (UID: \"523a31c4-4aea-4e9f-9ec4-7c28faefbf7f\") " pod="e2e-models-e2e-worker-w3-fb1398/distinct-2-w3-fb1398-kserve-597b7dc99b-dp4jf" Nov 02 06:35:24.550212 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:35:24.550164 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppnvp\" (UniqueName: \"kubernetes.io/projected/523a31c4-4aea-4e9f-9ec4-7c28faefbf7f-kube-api-access-ppnvp\") pod \"distinct-2-w3-fb1398-kserve-597b7dc99b-dp4jf\" (UID: \"523a31c4-4aea-4e9f-9ec4-7c28faefbf7f\") " pod="e2e-models-e2e-worker-w3-fb1398/distinct-2-w3-fb1398-kserve-597b7dc99b-dp4jf" Nov 02 06:35:24.550267 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:35:24.550209 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/523a31c4-4aea-4e9f-9ec4-7c28faefbf7f-model-cache\") pod \"distinct-2-w3-fb1398-kserve-597b7dc99b-dp4jf\" (UID: \"523a31c4-4aea-4e9f-9ec4-7c28faefbf7f\") " pod="e2e-models-e2e-worker-w3-fb1398/distinct-2-w3-fb1398-kserve-597b7dc99b-dp4jf" Nov 02 06:35:24.550325 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:35:24.550271 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/523a31c4-4aea-4e9f-9ec4-7c28faefbf7f-home\") pod \"distinct-2-w3-fb1398-kserve-597b7dc99b-dp4jf\" (UID: \"523a31c4-4aea-4e9f-9ec4-7c28faefbf7f\") " pod="e2e-models-e2e-worker-w3-fb1398/distinct-2-w3-fb1398-kserve-597b7dc99b-dp4jf" Nov 02 06:35:24.573142 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:35:24.573091 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-fb1398\"/\"default-dockercfg-mf5zh\"" Nov 02 06:35:24.573142 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:35:24.573108 5916 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-fb1398\"/\"kube-root-ca.crt\"" Nov 02 06:35:24.573367 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:35:24.573195 5916 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-fb1398\"/\"openshift-service-ca.crt\"" Nov 02 06:35:24.573367 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:35:24.573201 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-fb1398\"/\"distinct-2-w3-fb1398-kserve-self-signed-certs\"" Nov 02 06:35:24.622710 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:35:24.622681 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-fb1398/distinct-2-w3-fb1398-kserve-597b7dc99b-dp4jf"] Nov 02 06:35:24.650820 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:35:24.650791 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/523a31c4-4aea-4e9f-9ec4-7c28faefbf7f-home\") pod \"distinct-2-w3-fb1398-kserve-597b7dc99b-dp4jf\" (UID: \"523a31c4-4aea-4e9f-9ec4-7c28faefbf7f\") " pod="e2e-models-e2e-worker-w3-fb1398/distinct-2-w3-fb1398-kserve-597b7dc99b-dp4jf" Nov 02 06:35:24.651021 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:35:24.650844 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/523a31c4-4aea-4e9f-9ec4-7c28faefbf7f-dshm\") pod \"distinct-2-w3-fb1398-kserve-597b7dc99b-dp4jf\" (UID: \"523a31c4-4aea-4e9f-9ec4-7c28faefbf7f\") " pod="e2e-models-e2e-worker-w3-fb1398/distinct-2-w3-fb1398-kserve-597b7dc99b-dp4jf" Nov 02 06:35:24.651021 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:35:24.650974 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/523a31c4-4aea-4e9f-9ec4-7c28faefbf7f-tmp-dir\") pod \"distinct-2-w3-fb1398-kserve-597b7dc99b-dp4jf\" (UID: \"523a31c4-4aea-4e9f-9ec4-7c28faefbf7f\") " pod="e2e-models-e2e-worker-w3-fb1398/distinct-2-w3-fb1398-kserve-597b7dc99b-dp4jf" Nov 02 06:35:24.651143 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:35:24.651034 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/523a31c4-4aea-4e9f-9ec4-7c28faefbf7f-tls-certs\") pod \"distinct-2-w3-fb1398-kserve-597b7dc99b-dp4jf\" (UID: \"523a31c4-4aea-4e9f-9ec4-7c28faefbf7f\") " pod="e2e-models-e2e-worker-w3-fb1398/distinct-2-w3-fb1398-kserve-597b7dc99b-dp4jf" Nov 02 06:35:24.651143 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:35:24.651061 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppnvp\" (UniqueName: \"kubernetes.io/projected/523a31c4-4aea-4e9f-9ec4-7c28faefbf7f-kube-api-access-ppnvp\") pod \"distinct-2-w3-fb1398-kserve-597b7dc99b-dp4jf\" (UID: \"523a31c4-4aea-4e9f-9ec4-7c28faefbf7f\") " pod="e2e-models-e2e-worker-w3-fb1398/distinct-2-w3-fb1398-kserve-597b7dc99b-dp4jf" Nov 02 06:35:24.651143 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:35:24.651107 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/523a31c4-4aea-4e9f-9ec4-7c28faefbf7f-model-cache\") pod \"distinct-2-w3-fb1398-kserve-597b7dc99b-dp4jf\" (UID: \"523a31c4-4aea-4e9f-9ec4-7c28faefbf7f\") " pod="e2e-models-e2e-worker-w3-fb1398/distinct-2-w3-fb1398-kserve-597b7dc99b-dp4jf" Nov 02 06:35:24.651364 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:35:24.651341 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/523a31c4-4aea-4e9f-9ec4-7c28faefbf7f-home\") pod \"distinct-2-w3-fb1398-kserve-597b7dc99b-dp4jf\" (UID: \"523a31c4-4aea-4e9f-9ec4-7c28faefbf7f\") " pod="e2e-models-e2e-worker-w3-fb1398/distinct-2-w3-fb1398-kserve-597b7dc99b-dp4jf" Nov 02 06:35:24.651639 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:35:24.651616 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/523a31c4-4aea-4e9f-9ec4-7c28faefbf7f-model-cache\") pod \"distinct-2-w3-fb1398-kserve-597b7dc99b-dp4jf\" (UID: \"523a31c4-4aea-4e9f-9ec4-7c28faefbf7f\") " pod="e2e-models-e2e-worker-w3-fb1398/distinct-2-w3-fb1398-kserve-597b7dc99b-dp4jf" Nov 02 06:35:24.651778 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:35:24.651633 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/523a31c4-4aea-4e9f-9ec4-7c28faefbf7f-tmp-dir\") pod \"distinct-2-w3-fb1398-kserve-597b7dc99b-dp4jf\" (UID: \"523a31c4-4aea-4e9f-9ec4-7c28faefbf7f\") " pod="e2e-models-e2e-worker-w3-fb1398/distinct-2-w3-fb1398-kserve-597b7dc99b-dp4jf" Nov 02 06:35:24.654293 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:35:24.654269 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/523a31c4-4aea-4e9f-9ec4-7c28faefbf7f-tls-certs\") pod \"distinct-2-w3-fb1398-kserve-597b7dc99b-dp4jf\" (UID: \"523a31c4-4aea-4e9f-9ec4-7c28faefbf7f\") " pod="e2e-models-e2e-worker-w3-fb1398/distinct-2-w3-fb1398-kserve-597b7dc99b-dp4jf" Nov 02 06:35:24.654494 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:35:24.654473 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/523a31c4-4aea-4e9f-9ec4-7c28faefbf7f-dshm\") pod \"distinct-2-w3-fb1398-kserve-597b7dc99b-dp4jf\" (UID: \"523a31c4-4aea-4e9f-9ec4-7c28faefbf7f\") " pod="e2e-models-e2e-worker-w3-fb1398/distinct-2-w3-fb1398-kserve-597b7dc99b-dp4jf" Nov 02 06:35:24.704513 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:35:24.704485 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppnvp\" (UniqueName: \"kubernetes.io/projected/523a31c4-4aea-4e9f-9ec4-7c28faefbf7f-kube-api-access-ppnvp\") pod \"distinct-2-w3-fb1398-kserve-597b7dc99b-dp4jf\" (UID: \"523a31c4-4aea-4e9f-9ec4-7c28faefbf7f\") " pod="e2e-models-e2e-worker-w3-fb1398/distinct-2-w3-fb1398-kserve-597b7dc99b-dp4jf" Nov 02 06:35:24.838715 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:35:24.838625 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-fb1398/distinct-2-w3-fb1398-kserve-597b7dc99b-dp4jf" Nov 02 06:35:25.256227 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:35:25.255863 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-fb1398/distinct-2-w3-fb1398-kserve-597b7dc99b-dp4jf"] Nov 02 06:35:25.267715 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:35:25.267687 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod523a31c4_4aea_4e9f_9ec4_7c28faefbf7f.slice/crio-138c7b1ee60e1b768650e701a52e3f2e54990dc8cb9ff0b6799d283e121791df WatchSource:0}: Error finding container 138c7b1ee60e1b768650e701a52e3f2e54990dc8cb9ff0b6799d283e121791df: Status 404 returned error can't find the container with id 138c7b1ee60e1b768650e701a52e3f2e54990dc8cb9ff0b6799d283e121791df Nov 02 06:35:25.519269 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:35:25.519177 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-fb1398/distinct-2-w3-fb1398-kserve-597b7dc99b-dp4jf" event={"ID":"523a31c4-4aea-4e9f-9ec4-7c28faefbf7f","Type":"ContainerStarted","Data":"60af55daf28a4fcff0cc0f411c7c1c80e6a9a5429faa7b85469067dfdea55d0e"} Nov 02 06:35:25.519269 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:35:25.519222 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-fb1398/distinct-2-w3-fb1398-kserve-597b7dc99b-dp4jf" event={"ID":"523a31c4-4aea-4e9f-9ec4-7c28faefbf7f","Type":"ContainerStarted","Data":"138c7b1ee60e1b768650e701a52e3f2e54990dc8cb9ff0b6799d283e121791df"} Nov 02 06:35:25.631123 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:35:25.631062 5916 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-fb1398/distinct-2-w3-fb1398-kserve-597b7dc99b-dp4jf" podStartSLOduration=1.631044529 podStartE2EDuration="1.631044529s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:35:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:35:25.626128959 +0000 UTC m=+1113.897949352" watchObservedRunningTime="2026-11-02 06:35:25.631044529 +0000 UTC m=+1113.902864923" Nov 02 06:35:25.847876 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:35:25.847762 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-fb1398/distinct-2-w3-fb1398-kserve-597b7dc99b-dp4jf" Nov 02 06:35:34.839251 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:35:34.839214 5916 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-fb1398/distinct-2-w3-fb1398-kserve-597b7dc99b-dp4jf" Nov 02 06:35:34.852069 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:35:34.852044 5916 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-fb1398/distinct-2-w3-fb1398-kserve-597b7dc99b-dp4jf" Nov 02 06:35:35.607789 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:35:35.607762 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-fb1398/distinct-2-w3-fb1398-kserve-597b7dc99b-dp4jf" Nov 02 06:35:43.051539 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:35:43.051502 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-640421b6-z5df5"] Nov 02 06:35:43.097546 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:35:43.097505 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-640421b6-z5df5"] Nov 02 06:35:43.842857 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:35:43.842822 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e6672c5-8cec-458c-916c-74f5e417d387" path="/var/lib/kubelet/pods/9e6672c5-8cec-458c-916c-74f5e417d387/volumes" Nov 02 06:35:49.894869 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:35:49.894832 5916 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p98cq" containerName="extractor" Nov 02 06:35:53.113601 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:35:53.113566 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-dcf9c3/distinct-2-w5-dcf9c3-kserve-64747dbd8f-wlwng"] Nov 02 06:35:53.119859 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:35:53.119836 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-dcf9c3/distinct-2-w5-dcf9c3-kserve-64747dbd8f-wlwng" Nov 02 06:35:53.162636 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:35:53.162601 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e37d42ab-7b62-4f07-8a29-cdedeb0fe89a-model-cache\") pod \"distinct-2-w5-dcf9c3-kserve-64747dbd8f-wlwng\" (UID: \"e37d42ab-7b62-4f07-8a29-cdedeb0fe89a\") " pod="e2e-models-e2e-worker-w5-dcf9c3/distinct-2-w5-dcf9c3-kserve-64747dbd8f-wlwng" Nov 02 06:35:53.162802 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:35:53.162644 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e37d42ab-7b62-4f07-8a29-cdedeb0fe89a-tmp-dir\") pod \"distinct-2-w5-dcf9c3-kserve-64747dbd8f-wlwng\" (UID: \"e37d42ab-7b62-4f07-8a29-cdedeb0fe89a\") " pod="e2e-models-e2e-worker-w5-dcf9c3/distinct-2-w5-dcf9c3-kserve-64747dbd8f-wlwng" Nov 02 06:35:53.162852 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:35:53.162796 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5xvl\" (UniqueName: \"kubernetes.io/projected/e37d42ab-7b62-4f07-8a29-cdedeb0fe89a-kube-api-access-g5xvl\") pod \"distinct-2-w5-dcf9c3-kserve-64747dbd8f-wlwng\" (UID: \"e37d42ab-7b62-4f07-8a29-cdedeb0fe89a\") " pod="e2e-models-e2e-worker-w5-dcf9c3/distinct-2-w5-dcf9c3-kserve-64747dbd8f-wlwng" Nov 02 06:35:53.162953 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:35:53.162894 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e37d42ab-7b62-4f07-8a29-cdedeb0fe89a-tls-certs\") pod \"distinct-2-w5-dcf9c3-kserve-64747dbd8f-wlwng\" (UID: \"e37d42ab-7b62-4f07-8a29-cdedeb0fe89a\") " pod="e2e-models-e2e-worker-w5-dcf9c3/distinct-2-w5-dcf9c3-kserve-64747dbd8f-wlwng" Nov 02 06:35:53.162997 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:35:53.162983 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e37d42ab-7b62-4f07-8a29-cdedeb0fe89a-dshm\") pod \"distinct-2-w5-dcf9c3-kserve-64747dbd8f-wlwng\" (UID: \"e37d42ab-7b62-4f07-8a29-cdedeb0fe89a\") " pod="e2e-models-e2e-worker-w5-dcf9c3/distinct-2-w5-dcf9c3-kserve-64747dbd8f-wlwng" Nov 02 06:35:53.163034 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:35:53.163021 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e37d42ab-7b62-4f07-8a29-cdedeb0fe89a-home\") pod \"distinct-2-w5-dcf9c3-kserve-64747dbd8f-wlwng\" (UID: \"e37d42ab-7b62-4f07-8a29-cdedeb0fe89a\") " pod="e2e-models-e2e-worker-w5-dcf9c3/distinct-2-w5-dcf9c3-kserve-64747dbd8f-wlwng" Nov 02 06:35:53.178997 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:35:53.178969 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-dcf9c3/distinct-2-w5-dcf9c3-kserve-64747dbd8f-wlwng"] Nov 02 06:35:53.215731 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:35:53.215708 5916 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-dcf9c3\"/\"openshift-service-ca.crt\"" Nov 02 06:35:53.215731 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:35:53.215726 5916 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-dcf9c3\"/\"kube-root-ca.crt\"" Nov 02 06:35:53.220598 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:35:53.220580 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-dcf9c3\"/\"default-dockercfg-hc4f4\"" Nov 02 06:35:53.222892 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:35:53.222875 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-dcf9c3\"/\"distinct-2-w5-dcf9c3-kserve-self-signed-certs\"" Nov 02 06:35:53.268137 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:35:53.268094 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5xvl\" (UniqueName: \"kubernetes.io/projected/e37d42ab-7b62-4f07-8a29-cdedeb0fe89a-kube-api-access-g5xvl\") pod \"distinct-2-w5-dcf9c3-kserve-64747dbd8f-wlwng\" (UID: \"e37d42ab-7b62-4f07-8a29-cdedeb0fe89a\") " pod="e2e-models-e2e-worker-w5-dcf9c3/distinct-2-w5-dcf9c3-kserve-64747dbd8f-wlwng" Nov 02 06:35:53.268328 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:35:53.268176 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e37d42ab-7b62-4f07-8a29-cdedeb0fe89a-tls-certs\") pod \"distinct-2-w5-dcf9c3-kserve-64747dbd8f-wlwng\" (UID: \"e37d42ab-7b62-4f07-8a29-cdedeb0fe89a\") " pod="e2e-models-e2e-worker-w5-dcf9c3/distinct-2-w5-dcf9c3-kserve-64747dbd8f-wlwng" Nov 02 06:35:53.268328 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:35:53.268214 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e37d42ab-7b62-4f07-8a29-cdedeb0fe89a-dshm\") pod \"distinct-2-w5-dcf9c3-kserve-64747dbd8f-wlwng\" (UID: \"e37d42ab-7b62-4f07-8a29-cdedeb0fe89a\") " pod="e2e-models-e2e-worker-w5-dcf9c3/distinct-2-w5-dcf9c3-kserve-64747dbd8f-wlwng" Nov 02 06:35:53.268328 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:35:53.268269 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e37d42ab-7b62-4f07-8a29-cdedeb0fe89a-home\") pod \"distinct-2-w5-dcf9c3-kserve-64747dbd8f-wlwng\" (UID: \"e37d42ab-7b62-4f07-8a29-cdedeb0fe89a\") " pod="e2e-models-e2e-worker-w5-dcf9c3/distinct-2-w5-dcf9c3-kserve-64747dbd8f-wlwng" Nov 02 06:35:53.268486 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:35:53.268435 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e37d42ab-7b62-4f07-8a29-cdedeb0fe89a-model-cache\") pod \"distinct-2-w5-dcf9c3-kserve-64747dbd8f-wlwng\" (UID: \"e37d42ab-7b62-4f07-8a29-cdedeb0fe89a\") " pod="e2e-models-e2e-worker-w5-dcf9c3/distinct-2-w5-dcf9c3-kserve-64747dbd8f-wlwng" Nov 02 06:35:53.268486 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:35:53.268466 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e37d42ab-7b62-4f07-8a29-cdedeb0fe89a-tmp-dir\") pod \"distinct-2-w5-dcf9c3-kserve-64747dbd8f-wlwng\" (UID: \"e37d42ab-7b62-4f07-8a29-cdedeb0fe89a\") " pod="e2e-models-e2e-worker-w5-dcf9c3/distinct-2-w5-dcf9c3-kserve-64747dbd8f-wlwng" Nov 02 06:35:53.269037 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:35:53.268999 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e37d42ab-7b62-4f07-8a29-cdedeb0fe89a-tmp-dir\") pod \"distinct-2-w5-dcf9c3-kserve-64747dbd8f-wlwng\" (UID: \"e37d42ab-7b62-4f07-8a29-cdedeb0fe89a\") " pod="e2e-models-e2e-worker-w5-dcf9c3/distinct-2-w5-dcf9c3-kserve-64747dbd8f-wlwng" Nov 02 06:35:53.270055 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:35:53.270028 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e37d42ab-7b62-4f07-8a29-cdedeb0fe89a-home\") pod \"distinct-2-w5-dcf9c3-kserve-64747dbd8f-wlwng\" (UID: \"e37d42ab-7b62-4f07-8a29-cdedeb0fe89a\") " pod="e2e-models-e2e-worker-w5-dcf9c3/distinct-2-w5-dcf9c3-kserve-64747dbd8f-wlwng" Nov 02 06:35:53.270201 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:35:53.270167 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e37d42ab-7b62-4f07-8a29-cdedeb0fe89a-model-cache\") pod \"distinct-2-w5-dcf9c3-kserve-64747dbd8f-wlwng\" (UID: \"e37d42ab-7b62-4f07-8a29-cdedeb0fe89a\") " pod="e2e-models-e2e-worker-w5-dcf9c3/distinct-2-w5-dcf9c3-kserve-64747dbd8f-wlwng" Nov 02 06:35:53.272895 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:35:53.272870 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e37d42ab-7b62-4f07-8a29-cdedeb0fe89a-dshm\") pod \"distinct-2-w5-dcf9c3-kserve-64747dbd8f-wlwng\" (UID: \"e37d42ab-7b62-4f07-8a29-cdedeb0fe89a\") " pod="e2e-models-e2e-worker-w5-dcf9c3/distinct-2-w5-dcf9c3-kserve-64747dbd8f-wlwng" Nov 02 06:35:53.273778 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:35:53.273756 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e37d42ab-7b62-4f07-8a29-cdedeb0fe89a-tls-certs\") pod \"distinct-2-w5-dcf9c3-kserve-64747dbd8f-wlwng\" (UID: \"e37d42ab-7b62-4f07-8a29-cdedeb0fe89a\") " pod="e2e-models-e2e-worker-w5-dcf9c3/distinct-2-w5-dcf9c3-kserve-64747dbd8f-wlwng" Nov 02 06:35:53.322610 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:35:53.322574 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5xvl\" (UniqueName: \"kubernetes.io/projected/e37d42ab-7b62-4f07-8a29-cdedeb0fe89a-kube-api-access-g5xvl\") pod \"distinct-2-w5-dcf9c3-kserve-64747dbd8f-wlwng\" (UID: \"e37d42ab-7b62-4f07-8a29-cdedeb0fe89a\") " pod="e2e-models-e2e-worker-w5-dcf9c3/distinct-2-w5-dcf9c3-kserve-64747dbd8f-wlwng" Nov 02 06:35:53.435350 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:35:53.435313 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-dcf9c3/distinct-2-w5-dcf9c3-kserve-64747dbd8f-wlwng" Nov 02 06:35:53.864013 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:35:53.863920 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-dcf9c3/distinct-2-w5-dcf9c3-kserve-64747dbd8f-wlwng"] Nov 02 06:35:53.868975 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:35:53.868940 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode37d42ab_7b62_4f07_8a29_cdedeb0fe89a.slice/crio-cb536d243ae5318c718f7a4792a345d35a101dec52019ff7febed153e464a89c WatchSource:0}: Error finding container cb536d243ae5318c718f7a4792a345d35a101dec52019ff7febed153e464a89c: Status 404 returned error can't find the container with id cb536d243ae5318c718f7a4792a345d35a101dec52019ff7febed153e464a89c Nov 02 06:35:54.760979 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:35:54.760938 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-dcf9c3/distinct-2-w5-dcf9c3-kserve-64747dbd8f-wlwng" event={"ID":"e37d42ab-7b62-4f07-8a29-cdedeb0fe89a","Type":"ContainerStarted","Data":"5099831e5bb3125d1e33570a3097a3425678fb23b95c4a134e7e94c45db9ed52"} Nov 02 06:35:54.761395 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:35:54.760993 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-dcf9c3/distinct-2-w5-dcf9c3-kserve-64747dbd8f-wlwng" event={"ID":"e37d42ab-7b62-4f07-8a29-cdedeb0fe89a","Type":"ContainerStarted","Data":"cb536d243ae5318c718f7a4792a345d35a101dec52019ff7febed153e464a89c"} Nov 02 06:35:54.873528 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:35:54.873476 5916 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-dcf9c3/distinct-2-w5-dcf9c3-kserve-64747dbd8f-wlwng" podStartSLOduration=1.873462003 podStartE2EDuration="1.873462003s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:35:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:35:54.870372422 +0000 UTC m=+1143.142192815" watchObservedRunningTime="2026-11-02 06:35:54.873462003 +0000 UTC m=+1143.145282417" Nov 02 06:35:55.436263 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:35:55.436228 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-dcf9c3/distinct-2-w5-dcf9c3-kserve-64747dbd8f-wlwng" Nov 02 06:35:58.048806 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:35:58.048763 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-094ee0f4-9g8wx"] Nov 02 06:35:58.093240 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:35:58.093212 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-094ee0f4-9g8wx"] Nov 02 06:35:59.844024 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:35:59.843983 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98c8af93-2990-4e95-82e1-e8f980cb2b9e" path="/var/lib/kubelet/pods/98c8af93-2990-4e95-82e1-e8f980cb2b9e/volumes" Nov 02 06:35:59.940346 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:35:59.940314 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a418ce9-tf7rb"] Nov 02 06:35:59.944571 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:35:59.944550 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a418ce9-tf7rb" Nov 02 06:35:59.997264 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:35:59.997235 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a418ce9-tf7rb"] Nov 02 06:36:00.038780 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:00.038750 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 06:36:00.038942 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:00.038798 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-68l4r\"" Nov 02 06:36:00.048612 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:00.048560 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdl2t\" (UniqueName: \"kubernetes.io/projected/7d838d23-c56b-4439-ab37-a6b95e3317db-kube-api-access-zdl2t\") pod \"maas-api-revoke-sub-5a418ce9-tf7rb\" (UID: \"7d838d23-c56b-4439-ab37-a6b95e3317db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a418ce9-tf7rb" Nov 02 06:36:00.048805 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:00.048784 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d838d23-c56b-4439-ab37-a6b95e3317db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a418ce9-tf7rb\" (UID: \"7d838d23-c56b-4439-ab37-a6b95e3317db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a418ce9-tf7rb" Nov 02 06:36:00.048931 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:00.048873 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d838d23-c56b-4439-ab37-a6b95e3317db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a418ce9-tf7rb\" (UID: \"7d838d23-c56b-4439-ab37-a6b95e3317db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a418ce9-tf7rb" Nov 02 06:36:00.150028 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:00.149995 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d838d23-c56b-4439-ab37-a6b95e3317db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a418ce9-tf7rb\" (UID: \"7d838d23-c56b-4439-ab37-a6b95e3317db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a418ce9-tf7rb" Nov 02 06:36:00.150217 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:00.150057 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d838d23-c56b-4439-ab37-a6b95e3317db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a418ce9-tf7rb\" (UID: \"7d838d23-c56b-4439-ab37-a6b95e3317db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a418ce9-tf7rb" Nov 02 06:36:00.150217 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:00.150122 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdl2t\" (UniqueName: \"kubernetes.io/projected/7d838d23-c56b-4439-ab37-a6b95e3317db-kube-api-access-zdl2t\") pod \"maas-api-revoke-sub-5a418ce9-tf7rb\" (UID: \"7d838d23-c56b-4439-ab37-a6b95e3317db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a418ce9-tf7rb" Nov 02 06:36:00.150679 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:00.150649 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d838d23-c56b-4439-ab37-a6b95e3317db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a418ce9-tf7rb\" (UID: \"7d838d23-c56b-4439-ab37-a6b95e3317db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a418ce9-tf7rb" Nov 02 06:36:00.152667 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:00.152635 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d838d23-c56b-4439-ab37-a6b95e3317db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a418ce9-tf7rb\" (UID: \"7d838d23-c56b-4439-ab37-a6b95e3317db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a418ce9-tf7rb" Nov 02 06:36:00.203066 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:00.203042 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdl2t\" (UniqueName: \"kubernetes.io/projected/7d838d23-c56b-4439-ab37-a6b95e3317db-kube-api-access-zdl2t\") pod \"maas-api-revoke-sub-5a418ce9-tf7rb\" (UID: \"7d838d23-c56b-4439-ab37-a6b95e3317db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a418ce9-tf7rb" Nov 02 06:36:00.258312 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:00.258275 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a418ce9-tf7rb" Nov 02 06:36:00.636099 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:00.636062 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a418ce9-tf7rb"] Nov 02 06:36:00.645717 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:36:00.645684 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d838d23_c56b_4439_ab37_a6b95e3317db.slice/crio-b81ca74461306daac867832460e0eb0fcf5ffd1226b3f57546b85868a523ad9c WatchSource:0}: Error finding container b81ca74461306daac867832460e0eb0fcf5ffd1226b3f57546b85868a523ad9c: Status 404 returned error can't find the container with id b81ca74461306daac867832460e0eb0fcf5ffd1226b3f57546b85868a523ad9c Nov 02 06:36:00.805790 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:00.805756 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a418ce9-tf7rb" event={"ID":"7d838d23-c56b-4439-ab37-a6b95e3317db","Type":"ContainerStarted","Data":"b81ca74461306daac867832460e0eb0fcf5ffd1226b3f57546b85868a523ad9c"} Nov 02 06:36:01.832569 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:01.832435 5916 generic.go:356] "Generic (PLEG): container finished" podID="7d838d23-c56b-4439-ab37-a6b95e3317db" containerID="497801801e4f758868dce5b928ae62a470511d8345a8d58fed4437426f964286" exitCode=0 Nov 02 06:36:01.832988 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:01.832569 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a418ce9-tf7rb" event={"ID":"7d838d23-c56b-4439-ab37-a6b95e3317db","Type":"ContainerDied","Data":"497801801e4f758868dce5b928ae62a470511d8345a8d58fed4437426f964286"} Nov 02 06:36:03.082456 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:03.082245 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a418ce9-tf7rb" Nov 02 06:36:03.198427 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:03.198395 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d838d23-c56b-4439-ab37-a6b95e3317db-kube-api-access-zdl2t\" (UniqueName: \"kubernetes.io/projected/7d838d23-c56b-4439-ab37-a6b95e3317db-kube-api-access-zdl2t\") pod \"7d838d23-c56b-4439-ab37-a6b95e3317db\" (UID: \"7d838d23-c56b-4439-ab37-a6b95e3317db\") " Nov 02 06:36:03.198621 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:03.198466 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d838d23-c56b-4439-ab37-a6b95e3317db-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d838d23-c56b-4439-ab37-a6b95e3317db-maas-api-xks-ca\") pod \"7d838d23-c56b-4439-ab37-a6b95e3317db\" (UID: \"7d838d23-c56b-4439-ab37-a6b95e3317db\") " Nov 02 06:36:03.198621 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:03.198528 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d838d23-c56b-4439-ab37-a6b95e3317db-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d838d23-c56b-4439-ab37-a6b95e3317db-maas-api-openshift-ca\") pod \"7d838d23-c56b-4439-ab37-a6b95e3317db\" (UID: \"7d838d23-c56b-4439-ab37-a6b95e3317db\") " Nov 02 06:36:03.199317 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:03.199286 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d838d23-c56b-4439-ab37-a6b95e3317db-maas-api-openshift-ca" pod "7d838d23-c56b-4439-ab37-a6b95e3317db" (UID: "7d838d23-c56b-4439-ab37-a6b95e3317db"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:36:03.200933 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:03.200874 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d838d23-c56b-4439-ab37-a6b95e3317db-kube-api-access-zdl2t" pod "7d838d23-c56b-4439-ab37-a6b95e3317db" (UID: "7d838d23-c56b-4439-ab37-a6b95e3317db"). InnerVolumeSpecName "kube-api-access-zdl2t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:36:03.201293 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:03.201268 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d838d23-c56b-4439-ab37-a6b95e3317db-maas-api-xks-ca" pod "7d838d23-c56b-4439-ab37-a6b95e3317db" (UID: "7d838d23-c56b-4439-ab37-a6b95e3317db"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:36:03.300417 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:03.300387 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zdl2t\" (UniqueName: \"kubernetes.io/projected/7d838d23-c56b-4439-ab37-a6b95e3317db-kube-api-access-zdl2t\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:36:03.300417 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:03.300415 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d838d23-c56b-4439-ab37-a6b95e3317db-maas-api-xks-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:36:03.300417 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:03.300425 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d838d23-c56b-4439-ab37-a6b95e3317db-maas-api-openshift-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:36:03.436240 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:03.436206 5916 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-dcf9c3/distinct-2-w5-dcf9c3-kserve-64747dbd8f-wlwng" Nov 02 06:36:03.448842 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:03.448756 5916 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-dcf9c3/distinct-2-w5-dcf9c3-kserve-64747dbd8f-wlwng" Nov 02 06:36:03.562465 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:03.562432 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-581608/distinct-2-w2-581608-kserve-57dd49c4b-qnt9k"] Nov 02 06:36:03.562962 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:03.562948 5916 cpu_manager.go:414] "containerMap: removing container" podUID="7d838d23-c56b-4439-ab37-a6b95e3317db" containerName="revoke-keys" Nov 02 06:36:03.563033 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:03.562963 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d838d23-c56b-4439-ab37-a6b95e3317db" containerName="revoke-keys" Nov 02 06:36:03.563090 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:03.563079 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d838d23-c56b-4439-ab37-a6b95e3317db" containerName="revoke-keys" Nov 02 06:36:03.566527 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:03.566505 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-581608/distinct-2-w2-581608-kserve-57dd49c4b-qnt9k" Nov 02 06:36:03.615207 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:03.615179 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-581608\"/\"distinct-2-w2-581608-kserve-self-signed-certs\"" Nov 02 06:36:03.617507 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:03.617471 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-581608/distinct-2-w2-581608-kserve-57dd49c4b-qnt9k"] Nov 02 06:36:03.707633 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:03.707535 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/837780ad-8ea3-4ed2-922f-917cffc7bb93-model-cache\") pod \"distinct-2-w2-581608-kserve-57dd49c4b-qnt9k\" (UID: \"837780ad-8ea3-4ed2-922f-917cffc7bb93\") " pod="e2e-models-e2e-worker-w2-581608/distinct-2-w2-581608-kserve-57dd49c4b-qnt9k" Nov 02 06:36:03.707633 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:03.707597 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/837780ad-8ea3-4ed2-922f-917cffc7bb93-tls-certs\") pod \"distinct-2-w2-581608-kserve-57dd49c4b-qnt9k\" (UID: \"837780ad-8ea3-4ed2-922f-917cffc7bb93\") " pod="e2e-models-e2e-worker-w2-581608/distinct-2-w2-581608-kserve-57dd49c4b-qnt9k" Nov 02 06:36:03.707867 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:03.707700 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/837780ad-8ea3-4ed2-922f-917cffc7bb93-tmp-dir\") pod \"distinct-2-w2-581608-kserve-57dd49c4b-qnt9k\" (UID: \"837780ad-8ea3-4ed2-922f-917cffc7bb93\") " pod="e2e-models-e2e-worker-w2-581608/distinct-2-w2-581608-kserve-57dd49c4b-qnt9k" Nov 02 06:36:03.707867 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:03.707731 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/837780ad-8ea3-4ed2-922f-917cffc7bb93-dshm\") pod \"distinct-2-w2-581608-kserve-57dd49c4b-qnt9k\" (UID: \"837780ad-8ea3-4ed2-922f-917cffc7bb93\") " pod="e2e-models-e2e-worker-w2-581608/distinct-2-w2-581608-kserve-57dd49c4b-qnt9k" Nov 02 06:36:03.707867 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:03.707757 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4jsm\" (UniqueName: \"kubernetes.io/projected/837780ad-8ea3-4ed2-922f-917cffc7bb93-kube-api-access-w4jsm\") pod \"distinct-2-w2-581608-kserve-57dd49c4b-qnt9k\" (UID: \"837780ad-8ea3-4ed2-922f-917cffc7bb93\") " pod="e2e-models-e2e-worker-w2-581608/distinct-2-w2-581608-kserve-57dd49c4b-qnt9k" Nov 02 06:36:03.708039 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:03.707862 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/837780ad-8ea3-4ed2-922f-917cffc7bb93-home\") pod \"distinct-2-w2-581608-kserve-57dd49c4b-qnt9k\" (UID: \"837780ad-8ea3-4ed2-922f-917cffc7bb93\") " pod="e2e-models-e2e-worker-w2-581608/distinct-2-w2-581608-kserve-57dd49c4b-qnt9k" Nov 02 06:36:03.809314 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:03.809280 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/837780ad-8ea3-4ed2-922f-917cffc7bb93-model-cache\") pod \"distinct-2-w2-581608-kserve-57dd49c4b-qnt9k\" (UID: \"837780ad-8ea3-4ed2-922f-917cffc7bb93\") " pod="e2e-models-e2e-worker-w2-581608/distinct-2-w2-581608-kserve-57dd49c4b-qnt9k" Nov 02 06:36:03.809528 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:03.809338 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/837780ad-8ea3-4ed2-922f-917cffc7bb93-tls-certs\") pod \"distinct-2-w2-581608-kserve-57dd49c4b-qnt9k\" (UID: \"837780ad-8ea3-4ed2-922f-917cffc7bb93\") " pod="e2e-models-e2e-worker-w2-581608/distinct-2-w2-581608-kserve-57dd49c4b-qnt9k" Nov 02 06:36:03.809528 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:03.809385 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/837780ad-8ea3-4ed2-922f-917cffc7bb93-tmp-dir\") pod \"distinct-2-w2-581608-kserve-57dd49c4b-qnt9k\" (UID: \"837780ad-8ea3-4ed2-922f-917cffc7bb93\") " pod="e2e-models-e2e-worker-w2-581608/distinct-2-w2-581608-kserve-57dd49c4b-qnt9k" Nov 02 06:36:03.809528 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:03.809410 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/837780ad-8ea3-4ed2-922f-917cffc7bb93-dshm\") pod \"distinct-2-w2-581608-kserve-57dd49c4b-qnt9k\" (UID: \"837780ad-8ea3-4ed2-922f-917cffc7bb93\") " pod="e2e-models-e2e-worker-w2-581608/distinct-2-w2-581608-kserve-57dd49c4b-qnt9k" Nov 02 06:36:03.809528 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:03.809443 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4jsm\" (UniqueName: \"kubernetes.io/projected/837780ad-8ea3-4ed2-922f-917cffc7bb93-kube-api-access-w4jsm\") pod \"distinct-2-w2-581608-kserve-57dd49c4b-qnt9k\" (UID: \"837780ad-8ea3-4ed2-922f-917cffc7bb93\") " pod="e2e-models-e2e-worker-w2-581608/distinct-2-w2-581608-kserve-57dd49c4b-qnt9k" Nov 02 06:36:03.809528 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:03.809501 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/837780ad-8ea3-4ed2-922f-917cffc7bb93-home\") pod \"distinct-2-w2-581608-kserve-57dd49c4b-qnt9k\" (UID: \"837780ad-8ea3-4ed2-922f-917cffc7bb93\") " pod="e2e-models-e2e-worker-w2-581608/distinct-2-w2-581608-kserve-57dd49c4b-qnt9k" Nov 02 06:36:03.810037 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:03.810005 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/837780ad-8ea3-4ed2-922f-917cffc7bb93-model-cache\") pod \"distinct-2-w2-581608-kserve-57dd49c4b-qnt9k\" (UID: \"837780ad-8ea3-4ed2-922f-917cffc7bb93\") " pod="e2e-models-e2e-worker-w2-581608/distinct-2-w2-581608-kserve-57dd49c4b-qnt9k" Nov 02 06:36:03.810037 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:03.810036 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/837780ad-8ea3-4ed2-922f-917cffc7bb93-tmp-dir\") pod \"distinct-2-w2-581608-kserve-57dd49c4b-qnt9k\" (UID: \"837780ad-8ea3-4ed2-922f-917cffc7bb93\") " pod="e2e-models-e2e-worker-w2-581608/distinct-2-w2-581608-kserve-57dd49c4b-qnt9k" Nov 02 06:36:03.810221 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:03.810062 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/837780ad-8ea3-4ed2-922f-917cffc7bb93-home\") pod \"distinct-2-w2-581608-kserve-57dd49c4b-qnt9k\" (UID: \"837780ad-8ea3-4ed2-922f-917cffc7bb93\") " pod="e2e-models-e2e-worker-w2-581608/distinct-2-w2-581608-kserve-57dd49c4b-qnt9k" Nov 02 06:36:03.812020 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:03.811991 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/837780ad-8ea3-4ed2-922f-917cffc7bb93-dshm\") pod \"distinct-2-w2-581608-kserve-57dd49c4b-qnt9k\" (UID: \"837780ad-8ea3-4ed2-922f-917cffc7bb93\") " pod="e2e-models-e2e-worker-w2-581608/distinct-2-w2-581608-kserve-57dd49c4b-qnt9k" Nov 02 06:36:03.812134 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:03.812090 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/837780ad-8ea3-4ed2-922f-917cffc7bb93-tls-certs\") pod \"distinct-2-w2-581608-kserve-57dd49c4b-qnt9k\" (UID: \"837780ad-8ea3-4ed2-922f-917cffc7bb93\") " pod="e2e-models-e2e-worker-w2-581608/distinct-2-w2-581608-kserve-57dd49c4b-qnt9k" Nov 02 06:36:03.854322 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:03.854292 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a418ce9-tf7rb" Nov 02 06:36:03.854457 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:03.854327 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a418ce9-tf7rb" event={"ID":"7d838d23-c56b-4439-ab37-a6b95e3317db","Type":"ContainerDied","Data":"b81ca74461306daac867832460e0eb0fcf5ffd1226b3f57546b85868a523ad9c"} Nov 02 06:36:03.854457 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:03.854386 5916 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b81ca74461306daac867832460e0eb0fcf5ffd1226b3f57546b85868a523ad9c" Nov 02 06:36:03.863514 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:03.863487 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4jsm\" (UniqueName: \"kubernetes.io/projected/837780ad-8ea3-4ed2-922f-917cffc7bb93-kube-api-access-w4jsm\") pod \"distinct-2-w2-581608-kserve-57dd49c4b-qnt9k\" (UID: \"837780ad-8ea3-4ed2-922f-917cffc7bb93\") " pod="e2e-models-e2e-worker-w2-581608/distinct-2-w2-581608-kserve-57dd49c4b-qnt9k" Nov 02 06:36:03.870713 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:03.870684 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-dcf9c3/distinct-2-w5-dcf9c3-kserve-64747dbd8f-wlwng" Nov 02 06:36:03.882666 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:03.882636 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-581608/distinct-2-w2-581608-kserve-57dd49c4b-qnt9k" Nov 02 06:36:04.329100 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:04.329065 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-581608/distinct-2-w2-581608-kserve-57dd49c4b-qnt9k"] Nov 02 06:36:04.330382 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:36:04.330357 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod837780ad_8ea3_4ed2_922f_917cffc7bb93.slice/crio-652c8f205729f878d461dd7345d43a37798fcf44b58127210f03d068b9e05346 WatchSource:0}: Error finding container 652c8f205729f878d461dd7345d43a37798fcf44b58127210f03d068b9e05346: Status 404 returned error can't find the container with id 652c8f205729f878d461dd7345d43a37798fcf44b58127210f03d068b9e05346 Nov 02 06:36:04.867753 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:04.867716 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-581608/distinct-2-w2-581608-kserve-57dd49c4b-qnt9k" event={"ID":"837780ad-8ea3-4ed2-922f-917cffc7bb93","Type":"ContainerStarted","Data":"ae464271d3cd0b7c4574877c5e265e528ac0a69b46ab0602818df9ca80517383"} Nov 02 06:36:04.867753 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:04.867753 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-581608/distinct-2-w2-581608-kserve-57dd49c4b-qnt9k" event={"ID":"837780ad-8ea3-4ed2-922f-917cffc7bb93","Type":"ContainerStarted","Data":"652c8f205729f878d461dd7345d43a37798fcf44b58127210f03d068b9e05346"} Nov 02 06:36:04.883516 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:04.883480 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-581608/distinct-2-w2-581608-kserve-57dd49c4b-qnt9k" Nov 02 06:36:04.980119 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:04.980067 5916 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-581608/distinct-2-w2-581608-kserve-57dd49c4b-qnt9k" podStartSLOduration=1.9800516639999999 podStartE2EDuration="1.980051664s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:36:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:36:04.974771418 +0000 UTC m=+1153.246591811" watchObservedRunningTime="2026-11-02 06:36:04.980051664 +0000 UTC m=+1153.251872057" Nov 02 06:36:07.284064 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:07.284026 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0fe9b4ab-984fr"] Nov 02 06:36:07.288182 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:07.288157 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fe9b4ab-984fr" Nov 02 06:36:07.337169 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:07.337141 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 06:36:07.341837 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:07.341814 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0fe9b4ab-984fr"] Nov 02 06:36:07.351835 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:07.351798 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ac2c97b-96e6-421c-8503-06eb94532e6a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0fe9b4ab-984fr\" (UID: \"7ac2c97b-96e6-421c-8503-06eb94532e6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fe9b4ab-984fr" Nov 02 06:36:07.352129 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:07.352101 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ac2c97b-96e6-421c-8503-06eb94532e6a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0fe9b4ab-984fr\" (UID: \"7ac2c97b-96e6-421c-8503-06eb94532e6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fe9b4ab-984fr" Nov 02 06:36:07.352226 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:07.352201 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmh8v\" (UniqueName: \"kubernetes.io/projected/7ac2c97b-96e6-421c-8503-06eb94532e6a-kube-api-access-hmh8v\") pod \"maas-api-revoke-sub-0fe9b4ab-984fr\" (UID: \"7ac2c97b-96e6-421c-8503-06eb94532e6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fe9b4ab-984fr" Nov 02 06:36:07.382796 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:07.382769 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-68l4r\"" Nov 02 06:36:07.452961 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:07.452895 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmh8v\" (UniqueName: \"kubernetes.io/projected/7ac2c97b-96e6-421c-8503-06eb94532e6a-kube-api-access-hmh8v\") pod \"maas-api-revoke-sub-0fe9b4ab-984fr\" (UID: \"7ac2c97b-96e6-421c-8503-06eb94532e6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fe9b4ab-984fr" Nov 02 06:36:07.453119 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:07.452985 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ac2c97b-96e6-421c-8503-06eb94532e6a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0fe9b4ab-984fr\" (UID: \"7ac2c97b-96e6-421c-8503-06eb94532e6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fe9b4ab-984fr" Nov 02 06:36:07.453219 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:07.453200 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ac2c97b-96e6-421c-8503-06eb94532e6a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0fe9b4ab-984fr\" (UID: \"7ac2c97b-96e6-421c-8503-06eb94532e6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fe9b4ab-984fr" Nov 02 06:36:07.453665 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:07.453647 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ac2c97b-96e6-421c-8503-06eb94532e6a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0fe9b4ab-984fr\" (UID: \"7ac2c97b-96e6-421c-8503-06eb94532e6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fe9b4ab-984fr" Nov 02 06:36:07.455641 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:07.455615 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ac2c97b-96e6-421c-8503-06eb94532e6a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0fe9b4ab-984fr\" (UID: \"7ac2c97b-96e6-421c-8503-06eb94532e6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fe9b4ab-984fr" Nov 02 06:36:07.505925 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:07.505884 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmh8v\" (UniqueName: \"kubernetes.io/projected/7ac2c97b-96e6-421c-8503-06eb94532e6a-kube-api-access-hmh8v\") pod \"maas-api-revoke-sub-0fe9b4ab-984fr\" (UID: \"7ac2c97b-96e6-421c-8503-06eb94532e6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fe9b4ab-984fr" Nov 02 06:36:07.604272 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:07.604183 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fe9b4ab-984fr" Nov 02 06:36:07.990219 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:36:07.990186 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ac2c97b_96e6_421c_8503_06eb94532e6a.slice/crio-f4728a071d8177345a039911caa95c4adff1b2383233cda9800c22e9b54a7b8b WatchSource:0}: Error finding container f4728a071d8177345a039911caa95c4adff1b2383233cda9800c22e9b54a7b8b: Status 404 returned error can't find the container with id f4728a071d8177345a039911caa95c4adff1b2383233cda9800c22e9b54a7b8b Nov 02 06:36:07.991547 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:07.991520 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0fe9b4ab-984fr"] Nov 02 06:36:08.900892 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:08.900864 5916 generic.go:356] "Generic (PLEG): container finished" podID="7ac2c97b-96e6-421c-8503-06eb94532e6a" containerID="28fa798fb1562c22d16eb5a93a232befd6a3e42dac3cdc5cfe144b58eff42ded" exitCode=0 Nov 02 06:36:08.901226 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:08.900937 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fe9b4ab-984fr" event={"ID":"7ac2c97b-96e6-421c-8503-06eb94532e6a","Type":"ContainerDied","Data":"28fa798fb1562c22d16eb5a93a232befd6a3e42dac3cdc5cfe144b58eff42ded"} Nov 02 06:36:08.901226 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:08.900984 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fe9b4ab-984fr" event={"ID":"7ac2c97b-96e6-421c-8503-06eb94532e6a","Type":"ContainerStarted","Data":"f4728a071d8177345a039911caa95c4adff1b2383233cda9800c22e9b54a7b8b"} Nov 02 06:36:10.064282 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:10.064256 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fe9b4ab-984fr" Nov 02 06:36:10.193540 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:10.193453 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ac2c97b-96e6-421c-8503-06eb94532e6a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ac2c97b-96e6-421c-8503-06eb94532e6a-maas-api-openshift-ca\") pod \"7ac2c97b-96e6-421c-8503-06eb94532e6a\" (UID: \"7ac2c97b-96e6-421c-8503-06eb94532e6a\") " Nov 02 06:36:10.193728 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:10.193709 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ac2c97b-96e6-421c-8503-06eb94532e6a-kube-api-access-hmh8v\" (UniqueName: \"kubernetes.io/projected/7ac2c97b-96e6-421c-8503-06eb94532e6a-kube-api-access-hmh8v\") pod \"7ac2c97b-96e6-421c-8503-06eb94532e6a\" (UID: \"7ac2c97b-96e6-421c-8503-06eb94532e6a\") " Nov 02 06:36:10.193788 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:10.193775 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ac2c97b-96e6-421c-8503-06eb94532e6a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ac2c97b-96e6-421c-8503-06eb94532e6a-maas-api-xks-ca\") pod \"7ac2c97b-96e6-421c-8503-06eb94532e6a\" (UID: \"7ac2c97b-96e6-421c-8503-06eb94532e6a\") " Nov 02 06:36:10.193855 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:10.193827 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ac2c97b-96e6-421c-8503-06eb94532e6a-maas-api-openshift-ca" pod "7ac2c97b-96e6-421c-8503-06eb94532e6a" (UID: "7ac2c97b-96e6-421c-8503-06eb94532e6a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:36:10.194198 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:10.194176 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ac2c97b-96e6-421c-8503-06eb94532e6a-maas-api-openshift-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:36:10.195773 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:10.195748 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ac2c97b-96e6-421c-8503-06eb94532e6a-kube-api-access-hmh8v" pod "7ac2c97b-96e6-421c-8503-06eb94532e6a" (UID: "7ac2c97b-96e6-421c-8503-06eb94532e6a"). InnerVolumeSpecName "kube-api-access-hmh8v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:36:10.195893 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:10.195875 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ac2c97b-96e6-421c-8503-06eb94532e6a-maas-api-xks-ca" pod "7ac2c97b-96e6-421c-8503-06eb94532e6a" (UID: "7ac2c97b-96e6-421c-8503-06eb94532e6a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:36:10.295392 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:10.295355 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmh8v\" (UniqueName: \"kubernetes.io/projected/7ac2c97b-96e6-421c-8503-06eb94532e6a-kube-api-access-hmh8v\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:36:10.295392 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:10.295387 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ac2c97b-96e6-421c-8503-06eb94532e6a-maas-api-xks-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:36:10.917531 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:10.917500 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fe9b4ab-984fr" Nov 02 06:36:10.917711 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:10.917535 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fe9b4ab-984fr" event={"ID":"7ac2c97b-96e6-421c-8503-06eb94532e6a","Type":"ContainerDied","Data":"f4728a071d8177345a039911caa95c4adff1b2383233cda9800c22e9b54a7b8b"} Nov 02 06:36:10.917711 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:10.917592 5916 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f4728a071d8177345a039911caa95c4adff1b2383233cda9800c22e9b54a7b8b" Nov 02 06:36:13.045738 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:13.045705 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98d79b6f-hkh6r"] Nov 02 06:36:13.091960 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:13.091923 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98d79b6f-hkh6r"] Nov 02 06:36:13.844410 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:13.844380 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab436b1a-64b1-4527-b5c3-b9d9bd0a5883" path="/var/lib/kubelet/pods/ab436b1a-64b1-4527-b5c3-b9d9bd0a5883/volumes" Nov 02 06:36:13.883031 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:13.883001 5916 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-581608/distinct-2-w2-581608-kserve-57dd49c4b-qnt9k" Nov 02 06:36:13.895776 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:13.895752 5916 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-581608/distinct-2-w2-581608-kserve-57dd49c4b-qnt9k" Nov 02 06:36:13.951505 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:13.951479 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-581608/distinct-2-w2-581608-kserve-57dd49c4b-qnt9k" Nov 02 06:36:15.363309 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:15.363259 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7f7deb8-rql6s"] Nov 02 06:36:15.364000 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:15.363979 5916 cpu_manager.go:414] "containerMap: removing container" podUID="7ac2c97b-96e6-421c-8503-06eb94532e6a" containerName="revoke-keys" Nov 02 06:36:15.364000 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:15.364000 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ac2c97b-96e6-421c-8503-06eb94532e6a" containerName="revoke-keys" Nov 02 06:36:15.364207 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:15.364195 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ac2c97b-96e6-421c-8503-06eb94532e6a" containerName="revoke-keys" Nov 02 06:36:15.368073 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:15.368052 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7f7deb8-rql6s" Nov 02 06:36:15.422735 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:15.422700 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7f7deb8-rql6s"] Nov 02 06:36:15.452719 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:15.452683 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fef5332f-da6e-4a02-9ee5-b50499a1c373-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b7f7deb8-rql6s\" (UID: \"fef5332f-da6e-4a02-9ee5-b50499a1c373\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7f7deb8-rql6s" Nov 02 06:36:15.452882 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:15.452733 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f66dk\" (UniqueName: \"kubernetes.io/projected/fef5332f-da6e-4a02-9ee5-b50499a1c373-kube-api-access-f66dk\") pod \"maas-api-revoke-sub-b7f7deb8-rql6s\" (UID: \"fef5332f-da6e-4a02-9ee5-b50499a1c373\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7f7deb8-rql6s" Nov 02 06:36:15.452979 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:15.452885 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fef5332f-da6e-4a02-9ee5-b50499a1c373-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b7f7deb8-rql6s\" (UID: \"fef5332f-da6e-4a02-9ee5-b50499a1c373\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7f7deb8-rql6s" Nov 02 06:36:15.462471 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:15.462445 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 06:36:15.462471 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:15.462468 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-68l4r\"" Nov 02 06:36:15.553651 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:15.553620 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fef5332f-da6e-4a02-9ee5-b50499a1c373-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b7f7deb8-rql6s\" (UID: \"fef5332f-da6e-4a02-9ee5-b50499a1c373\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7f7deb8-rql6s" Nov 02 06:36:15.553824 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:15.553663 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f66dk\" (UniqueName: \"kubernetes.io/projected/fef5332f-da6e-4a02-9ee5-b50499a1c373-kube-api-access-f66dk\") pod \"maas-api-revoke-sub-b7f7deb8-rql6s\" (UID: \"fef5332f-da6e-4a02-9ee5-b50499a1c373\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7f7deb8-rql6s" Nov 02 06:36:15.553824 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:15.553725 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fef5332f-da6e-4a02-9ee5-b50499a1c373-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b7f7deb8-rql6s\" (UID: \"fef5332f-da6e-4a02-9ee5-b50499a1c373\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7f7deb8-rql6s" Nov 02 06:36:15.554418 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:15.554388 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fef5332f-da6e-4a02-9ee5-b50499a1c373-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b7f7deb8-rql6s\" (UID: \"fef5332f-da6e-4a02-9ee5-b50499a1c373\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7f7deb8-rql6s" Nov 02 06:36:15.555939 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:15.555919 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fef5332f-da6e-4a02-9ee5-b50499a1c373-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b7f7deb8-rql6s\" (UID: \"fef5332f-da6e-4a02-9ee5-b50499a1c373\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7f7deb8-rql6s" Nov 02 06:36:15.612176 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:15.612143 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f66dk\" (UniqueName: \"kubernetes.io/projected/fef5332f-da6e-4a02-9ee5-b50499a1c373-kube-api-access-f66dk\") pod \"maas-api-revoke-sub-b7f7deb8-rql6s\" (UID: \"fef5332f-da6e-4a02-9ee5-b50499a1c373\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7f7deb8-rql6s" Nov 02 06:36:15.644691 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:15.644655 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-c955m"] Nov 02 06:36:15.648889 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:15.648869 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-c955m" Nov 02 06:36:15.654473 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:15.654439 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12eeae94-c1bf-4f75-bc8d-2170fc882d8e-workload-socket\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-c955m\" (UID: \"12eeae94-c1bf-4f75-bc8d-2170fc882d8e\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-c955m" Nov 02 06:36:15.654576 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:15.654540 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12eeae94-c1bf-4f75-bc8d-2170fc882d8e-istiod-ca-cert\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-c955m\" (UID: \"12eeae94-c1bf-4f75-bc8d-2170fc882d8e\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-c955m" Nov 02 06:36:15.654637 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:15.654576 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12eeae94-c1bf-4f75-bc8d-2170fc882d8e-istio-envoy\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-c955m\" (UID: \"12eeae94-c1bf-4f75-bc8d-2170fc882d8e\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-c955m" Nov 02 06:36:15.654709 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:15.654661 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tkr7\" (UniqueName: \"kubernetes.io/projected/12eeae94-c1bf-4f75-bc8d-2170fc882d8e-kube-api-access-9tkr7\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-c955m\" (UID: \"12eeae94-c1bf-4f75-bc8d-2170fc882d8e\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-c955m" Nov 02 06:36:15.654709 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:15.654703 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12eeae94-c1bf-4f75-bc8d-2170fc882d8e-istio-data\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-c955m\" (UID: \"12eeae94-c1bf-4f75-bc8d-2170fc882d8e\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-c955m" Nov 02 06:36:15.654814 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:15.654770 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12eeae94-c1bf-4f75-bc8d-2170fc882d8e-istio-podinfo\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-c955m\" (UID: \"12eeae94-c1bf-4f75-bc8d-2170fc882d8e\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-c955m" Nov 02 06:36:15.654814 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:15.654793 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12eeae94-c1bf-4f75-bc8d-2170fc882d8e-credential-socket\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-c955m\" (UID: \"12eeae94-c1bf-4f75-bc8d-2170fc882d8e\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-c955m" Nov 02 06:36:15.654962 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:15.654816 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12eeae94-c1bf-4f75-bc8d-2170fc882d8e-workload-certs\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-c955m\" (UID: \"12eeae94-c1bf-4f75-bc8d-2170fc882d8e\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-c955m" Nov 02 06:36:15.654962 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:15.654871 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12eeae94-c1bf-4f75-bc8d-2170fc882d8e-istio-token\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-c955m\" (UID: \"12eeae94-c1bf-4f75-bc8d-2170fc882d8e\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-c955m" Nov 02 06:36:15.681228 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:15.681202 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7f7deb8-rql6s" Nov 02 06:36:15.700077 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:15.700052 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-cdfce9-openshift-default-dockercfg-j4bz4\"" Nov 02 06:36:15.706962 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:15.706931 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-c955m"] Nov 02 06:36:15.756221 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:15.756183 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tkr7\" (UniqueName: \"kubernetes.io/projected/12eeae94-c1bf-4f75-bc8d-2170fc882d8e-kube-api-access-9tkr7\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-c955m\" (UID: \"12eeae94-c1bf-4f75-bc8d-2170fc882d8e\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-c955m" Nov 02 06:36:15.756367 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:15.756257 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12eeae94-c1bf-4f75-bc8d-2170fc882d8e-istio-data\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-c955m\" (UID: \"12eeae94-c1bf-4f75-bc8d-2170fc882d8e\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-c955m" Nov 02 06:36:15.756367 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:15.756301 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12eeae94-c1bf-4f75-bc8d-2170fc882d8e-istio-podinfo\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-c955m\" (UID: \"12eeae94-c1bf-4f75-bc8d-2170fc882d8e\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-c955m" Nov 02 06:36:15.756367 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:15.756327 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12eeae94-c1bf-4f75-bc8d-2170fc882d8e-credential-socket\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-c955m\" (UID: \"12eeae94-c1bf-4f75-bc8d-2170fc882d8e\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-c955m" Nov 02 06:36:15.756530 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:15.756364 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12eeae94-c1bf-4f75-bc8d-2170fc882d8e-workload-certs\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-c955m\" (UID: \"12eeae94-c1bf-4f75-bc8d-2170fc882d8e\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-c955m" Nov 02 06:36:15.756530 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:15.756421 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12eeae94-c1bf-4f75-bc8d-2170fc882d8e-istio-token\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-c955m\" (UID: \"12eeae94-c1bf-4f75-bc8d-2170fc882d8e\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-c955m" Nov 02 06:36:15.756530 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:15.756465 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12eeae94-c1bf-4f75-bc8d-2170fc882d8e-workload-socket\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-c955m\" (UID: \"12eeae94-c1bf-4f75-bc8d-2170fc882d8e\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-c955m" Nov 02 06:36:15.756710 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:15.756546 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12eeae94-c1bf-4f75-bc8d-2170fc882d8e-istiod-ca-cert\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-c955m\" (UID: \"12eeae94-c1bf-4f75-bc8d-2170fc882d8e\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-c955m" Nov 02 06:36:15.756710 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:15.756587 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12eeae94-c1bf-4f75-bc8d-2170fc882d8e-istio-envoy\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-c955m\" (UID: \"12eeae94-c1bf-4f75-bc8d-2170fc882d8e\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-c955m" Nov 02 06:36:15.756710 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:15.756695 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12eeae94-c1bf-4f75-bc8d-2170fc882d8e-istio-data\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-c955m\" (UID: \"12eeae94-c1bf-4f75-bc8d-2170fc882d8e\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-c955m" Nov 02 06:36:15.756872 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:15.756731 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12eeae94-c1bf-4f75-bc8d-2170fc882d8e-credential-socket\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-c955m\" (UID: \"12eeae94-c1bf-4f75-bc8d-2170fc882d8e\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-c955m" Nov 02 06:36:15.756872 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:15.756766 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12eeae94-c1bf-4f75-bc8d-2170fc882d8e-workload-certs\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-c955m\" (UID: \"12eeae94-c1bf-4f75-bc8d-2170fc882d8e\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-c955m" Nov 02 06:36:15.757038 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:15.757020 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12eeae94-c1bf-4f75-bc8d-2170fc882d8e-workload-socket\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-c955m\" (UID: \"12eeae94-c1bf-4f75-bc8d-2170fc882d8e\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-c955m" Nov 02 06:36:15.757589 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:15.757568 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12eeae94-c1bf-4f75-bc8d-2170fc882d8e-istiod-ca-cert\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-c955m\" (UID: \"12eeae94-c1bf-4f75-bc8d-2170fc882d8e\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-c955m" Nov 02 06:36:15.761160 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:15.760323 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12eeae94-c1bf-4f75-bc8d-2170fc882d8e-istio-envoy\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-c955m\" (UID: \"12eeae94-c1bf-4f75-bc8d-2170fc882d8e\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-c955m" Nov 02 06:36:15.770951 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:15.770613 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12eeae94-c1bf-4f75-bc8d-2170fc882d8e-istio-podinfo\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-c955m\" (UID: \"12eeae94-c1bf-4f75-bc8d-2170fc882d8e\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-c955m" Nov 02 06:36:15.809672 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:15.809630 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12eeae94-c1bf-4f75-bc8d-2170fc882d8e-istio-token\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-c955m\" (UID: \"12eeae94-c1bf-4f75-bc8d-2170fc882d8e\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-c955m" Nov 02 06:36:15.809989 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:15.809791 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tkr7\" (UniqueName: \"kubernetes.io/projected/12eeae94-c1bf-4f75-bc8d-2170fc882d8e-kube-api-access-9tkr7\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-c955m\" (UID: \"12eeae94-c1bf-4f75-bc8d-2170fc882d8e\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-c955m" Nov 02 06:36:15.963251 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:15.963178 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-c955m" Nov 02 06:36:16.070585 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:16.070553 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7f7deb8-rql6s"] Nov 02 06:36:16.074394 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:36:16.074356 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfef5332f_da6e_4a02_9ee5_b50499a1c373.slice/crio-c94bb584077ba54f40d7c0134e9cac8ca22043a2f665723866fd729df0c45121 WatchSource:0}: Error finding container c94bb584077ba54f40d7c0134e9cac8ca22043a2f665723866fd729df0c45121: Status 404 returned error can't find the container with id c94bb584077ba54f40d7c0134e9cac8ca22043a2f665723866fd729df0c45121 Nov 02 06:36:16.341097 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:16.341062 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-c955m"] Nov 02 06:36:16.345159 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:36:16.345129 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12eeae94_c1bf_4f75_bc8d_2170fc882d8e.slice/crio-76504f59f711f2dac6a5d3d28a88dc430029508e43a65936cac525f4e961136d WatchSource:0}: Error finding container 76504f59f711f2dac6a5d3d28a88dc430029508e43a65936cac525f4e961136d: Status 404 returned error can't find the container with id 76504f59f711f2dac6a5d3d28a88dc430029508e43a65936cac525f4e961136d Nov 02 06:36:16.347577 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:16.347546 5916 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 06:36:16.347663 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:16.347608 5916 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 06:36:16.347663 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:16.347638 5916 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 06:36:16.974620 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:16.974580 5916 generic.go:356] "Generic (PLEG): container finished" podID="fef5332f-da6e-4a02-9ee5-b50499a1c373" containerID="22a25b082795494c52451bcf2b9604520fb23aea4f936d6e8a94236d2bcbb261" exitCode=0 Nov 02 06:36:16.975110 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:16.974669 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7f7deb8-rql6s" event={"ID":"fef5332f-da6e-4a02-9ee5-b50499a1c373","Type":"ContainerDied","Data":"22a25b082795494c52451bcf2b9604520fb23aea4f936d6e8a94236d2bcbb261"} Nov 02 06:36:16.975110 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:16.974733 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7f7deb8-rql6s" event={"ID":"fef5332f-da6e-4a02-9ee5-b50499a1c373","Type":"ContainerStarted","Data":"c94bb584077ba54f40d7c0134e9cac8ca22043a2f665723866fd729df0c45121"} Nov 02 06:36:16.980940 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:16.976973 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-c955m" event={"ID":"12eeae94-c1bf-4f75-bc8d-2170fc882d8e","Type":"ContainerStarted","Data":"f5a5d7460a73766554bf55fd4276489711bbea9edda84644c823835082903343"} Nov 02 06:36:16.980940 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:16.977028 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-c955m" event={"ID":"12eeae94-c1bf-4f75-bc8d-2170fc882d8e","Type":"ContainerStarted","Data":"76504f59f711f2dac6a5d3d28a88dc430029508e43a65936cac525f4e961136d"} Nov 02 06:36:17.205407 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:17.205297 5916 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-c955m" podStartSLOduration=2.205282484 podStartE2EDuration="2.205282484s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:36:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:36:17.20305869 +0000 UTC m=+1165.474879097" watchObservedRunningTime="2026-11-02 06:36:17.205282484 +0000 UTC m=+1165.477102878" Nov 02 06:36:17.964321 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:17.964287 5916 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-c955m" Nov 02 06:36:17.965991 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:17.965959 5916 patch_prober.go:29] interesting pod/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-c955m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Nov 02 06:36:17.966129 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:17.966008 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-c955m" podUID="12eeae94-c1bf-4f75-bc8d-2170fc882d8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Nov 02 06:36:18.183816 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:18.183789 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7f7deb8-rql6s" Nov 02 06:36:18.286808 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:18.286716 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fef5332f-da6e-4a02-9ee5-b50499a1c373-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fef5332f-da6e-4a02-9ee5-b50499a1c373-maas-api-openshift-ca\") pod \"fef5332f-da6e-4a02-9ee5-b50499a1c373\" (UID: \"fef5332f-da6e-4a02-9ee5-b50499a1c373\") " Nov 02 06:36:18.286997 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:18.286838 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fef5332f-da6e-4a02-9ee5-b50499a1c373-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fef5332f-da6e-4a02-9ee5-b50499a1c373-maas-api-xks-ca\") pod \"fef5332f-da6e-4a02-9ee5-b50499a1c373\" (UID: \"fef5332f-da6e-4a02-9ee5-b50499a1c373\") " Nov 02 06:36:18.286997 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:18.286893 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fef5332f-da6e-4a02-9ee5-b50499a1c373-kube-api-access-f66dk\" (UniqueName: \"kubernetes.io/projected/fef5332f-da6e-4a02-9ee5-b50499a1c373-kube-api-access-f66dk\") pod \"fef5332f-da6e-4a02-9ee5-b50499a1c373\" (UID: \"fef5332f-da6e-4a02-9ee5-b50499a1c373\") " Nov 02 06:36:18.287153 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:18.287131 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fef5332f-da6e-4a02-9ee5-b50499a1c373-maas-api-openshift-ca" pod "fef5332f-da6e-4a02-9ee5-b50499a1c373" (UID: "fef5332f-da6e-4a02-9ee5-b50499a1c373"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:36:18.287406 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:18.287375 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fef5332f-da6e-4a02-9ee5-b50499a1c373-maas-api-openshift-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:36:18.289066 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:18.289042 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fef5332f-da6e-4a02-9ee5-b50499a1c373-maas-api-xks-ca" pod "fef5332f-da6e-4a02-9ee5-b50499a1c373" (UID: "fef5332f-da6e-4a02-9ee5-b50499a1c373"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:36:18.289155 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:18.289075 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fef5332f-da6e-4a02-9ee5-b50499a1c373-kube-api-access-f66dk" pod "fef5332f-da6e-4a02-9ee5-b50499a1c373" (UID: "fef5332f-da6e-4a02-9ee5-b50499a1c373"). InnerVolumeSpecName "kube-api-access-f66dk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:36:18.388792 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:18.388763 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fef5332f-da6e-4a02-9ee5-b50499a1c373-maas-api-xks-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:36:18.388792 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:18.388791 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f66dk\" (UniqueName: \"kubernetes.io/projected/fef5332f-da6e-4a02-9ee5-b50499a1c373-kube-api-access-f66dk\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:36:18.964087 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:18.964051 5916 patch_prober.go:29] interesting pod/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-c955m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Nov 02 06:36:18.964268 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:18.964112 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-c955m" podUID="12eeae94-c1bf-4f75-bc8d-2170fc882d8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Nov 02 06:36:18.995894 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:18.995865 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7f7deb8-rql6s" Nov 02 06:36:18.996099 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:18.995898 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7f7deb8-rql6s" event={"ID":"fef5332f-da6e-4a02-9ee5-b50499a1c373","Type":"ContainerDied","Data":"c94bb584077ba54f40d7c0134e9cac8ca22043a2f665723866fd729df0c45121"} Nov 02 06:36:18.996099 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:18.995960 5916 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c94bb584077ba54f40d7c0134e9cac8ca22043a2f665723866fd729df0c45121" Nov 02 06:36:19.895870 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:36:19.895829 5916 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p98cq" containerName="extractor" Nov 02 06:36:19.964204 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:19.964162 5916 patch_prober.go:29] interesting pod/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-c955m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Nov 02 06:36:19.964406 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:19.964223 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-c955m" podUID="12eeae94-c1bf-4f75-bc8d-2170fc882d8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Nov 02 06:36:20.964043 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:20.964008 5916 patch_prober.go:29] interesting pod/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-c955m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Nov 02 06:36:20.964579 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:20.964062 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-c955m" podUID="12eeae94-c1bf-4f75-bc8d-2170fc882d8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Nov 02 06:36:21.249990 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:21.248117 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10585146-mkldd"] Nov 02 06:36:21.249990 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:21.248991 5916 cpu_manager.go:414] "containerMap: removing container" podUID="fef5332f-da6e-4a02-9ee5-b50499a1c373" containerName="revoke-keys" Nov 02 06:36:21.249990 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:21.249011 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fef5332f-da6e-4a02-9ee5-b50499a1c373" containerName="revoke-keys" Nov 02 06:36:21.249990 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:21.249184 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="fef5332f-da6e-4a02-9ee5-b50499a1c373" containerName="revoke-keys" Nov 02 06:36:21.254571 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:21.254544 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10585146-mkldd" Nov 02 06:36:21.284709 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:21.284677 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10585146-mkldd"] Nov 02 06:36:21.303412 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:21.303360 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 06:36:21.350955 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:21.336863 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fvlp\" (UniqueName: \"kubernetes.io/projected/fd3cc14b-9cd3-4500-bf18-8a0832e357cd-kube-api-access-2fvlp\") pod \"maas-api-revoke-sub-10585146-mkldd\" (UID: \"fd3cc14b-9cd3-4500-bf18-8a0832e357cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10585146-mkldd" Nov 02 06:36:21.350955 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:21.336936 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd3cc14b-9cd3-4500-bf18-8a0832e357cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-10585146-mkldd\" (UID: \"fd3cc14b-9cd3-4500-bf18-8a0832e357cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10585146-mkldd" Nov 02 06:36:21.350955 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:21.337052 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd3cc14b-9cd3-4500-bf18-8a0832e357cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-10585146-mkldd\" (UID: \"fd3cc14b-9cd3-4500-bf18-8a0832e357cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10585146-mkldd" Nov 02 06:36:21.350955 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:21.337207 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-68l4r\"" Nov 02 06:36:21.450695 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:21.450654 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd3cc14b-9cd3-4500-bf18-8a0832e357cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-10585146-mkldd\" (UID: \"fd3cc14b-9cd3-4500-bf18-8a0832e357cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10585146-mkldd" Nov 02 06:36:21.450939 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:21.450832 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fvlp\" (UniqueName: \"kubernetes.io/projected/fd3cc14b-9cd3-4500-bf18-8a0832e357cd-kube-api-access-2fvlp\") pod \"maas-api-revoke-sub-10585146-mkldd\" (UID: \"fd3cc14b-9cd3-4500-bf18-8a0832e357cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10585146-mkldd" Nov 02 06:36:21.450939 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:21.450863 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd3cc14b-9cd3-4500-bf18-8a0832e357cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-10585146-mkldd\" (UID: \"fd3cc14b-9cd3-4500-bf18-8a0832e357cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10585146-mkldd" Nov 02 06:36:21.451413 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:21.451388 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd3cc14b-9cd3-4500-bf18-8a0832e357cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-10585146-mkldd\" (UID: \"fd3cc14b-9cd3-4500-bf18-8a0832e357cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10585146-mkldd" Nov 02 06:36:21.453188 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:21.453167 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd3cc14b-9cd3-4500-bf18-8a0832e357cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-10585146-mkldd\" (UID: \"fd3cc14b-9cd3-4500-bf18-8a0832e357cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10585146-mkldd" Nov 02 06:36:21.504355 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:21.504289 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fvlp\" (UniqueName: \"kubernetes.io/projected/fd3cc14b-9cd3-4500-bf18-8a0832e357cd-kube-api-access-2fvlp\") pod \"maas-api-revoke-sub-10585146-mkldd\" (UID: \"fd3cc14b-9cd3-4500-bf18-8a0832e357cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10585146-mkldd" Nov 02 06:36:21.569022 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:21.568988 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10585146-mkldd" Nov 02 06:36:21.956121 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:21.956072 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10585146-mkldd"] Nov 02 06:36:21.957002 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:36:21.956969 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd3cc14b_9cd3_4500_bf18_8a0832e357cd.slice/crio-5be7db14cfb116492faa5fe5ed3c76042305e8330011c9e9e108719c21e909f1 WatchSource:0}: Error finding container 5be7db14cfb116492faa5fe5ed3c76042305e8330011c9e9e108719c21e909f1: Status 404 returned error can't find the container with id 5be7db14cfb116492faa5fe5ed3c76042305e8330011c9e9e108719c21e909f1 Nov 02 06:36:21.964210 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:21.964188 5916 patch_prober.go:29] interesting pod/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-c955m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Nov 02 06:36:21.964509 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:21.964229 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-c955m" podUID="12eeae94-c1bf-4f75-bc8d-2170fc882d8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Nov 02 06:36:22.019288 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:22.019261 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10585146-mkldd" event={"ID":"fd3cc14b-9cd3-4500-bf18-8a0832e357cd","Type":"ContainerStarted","Data":"5be7db14cfb116492faa5fe5ed3c76042305e8330011c9e9e108719c21e909f1"} Nov 02 06:36:22.964004 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:22.963971 5916 patch_prober.go:29] interesting pod/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-c955m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Nov 02 06:36:22.964146 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:22.964020 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-c955m" podUID="12eeae94-c1bf-4f75-bc8d-2170fc882d8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Nov 02 06:36:23.055467 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:23.055427 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10585146-mkldd" event={"ID":"fd3cc14b-9cd3-4500-bf18-8a0832e357cd","Type":"ContainerDied","Data":"901080ed9502e689e4599eb9bc470b13f3e9df7944498af9251b998aa617cdd9"} Nov 02 06:36:23.056031 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:23.055406 5916 generic.go:356] "Generic (PLEG): container finished" podID="fd3cc14b-9cd3-4500-bf18-8a0832e357cd" containerID="901080ed9502e689e4599eb9bc470b13f3e9df7944498af9251b998aa617cdd9" exitCode=0 Nov 02 06:36:23.968309 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:23.968276 5916 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-c955m" Nov 02 06:36:23.968801 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:23.968782 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-c955m" Nov 02 06:36:23.969602 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:23.969585 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-c955m" Nov 02 06:36:24.215715 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:24.215683 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10585146-mkldd" Nov 02 06:36:24.295710 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:24.295637 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd3cc14b-9cd3-4500-bf18-8a0832e357cd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd3cc14b-9cd3-4500-bf18-8a0832e357cd-maas-api-xks-ca\") pod \"fd3cc14b-9cd3-4500-bf18-8a0832e357cd\" (UID: \"fd3cc14b-9cd3-4500-bf18-8a0832e357cd\") " Nov 02 06:36:24.295855 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:24.295724 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd3cc14b-9cd3-4500-bf18-8a0832e357cd-kube-api-access-2fvlp\" (UniqueName: \"kubernetes.io/projected/fd3cc14b-9cd3-4500-bf18-8a0832e357cd-kube-api-access-2fvlp\") pod \"fd3cc14b-9cd3-4500-bf18-8a0832e357cd\" (UID: \"fd3cc14b-9cd3-4500-bf18-8a0832e357cd\") " Nov 02 06:36:24.295898 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:24.295870 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd3cc14b-9cd3-4500-bf18-8a0832e357cd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd3cc14b-9cd3-4500-bf18-8a0832e357cd-maas-api-openshift-ca\") pod \"fd3cc14b-9cd3-4500-bf18-8a0832e357cd\" (UID: \"fd3cc14b-9cd3-4500-bf18-8a0832e357cd\") " Nov 02 06:36:24.296420 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:24.296391 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd3cc14b-9cd3-4500-bf18-8a0832e357cd-maas-api-openshift-ca" pod "fd3cc14b-9cd3-4500-bf18-8a0832e357cd" (UID: "fd3cc14b-9cd3-4500-bf18-8a0832e357cd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:36:24.298364 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:24.298335 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd3cc14b-9cd3-4500-bf18-8a0832e357cd-maas-api-xks-ca" pod "fd3cc14b-9cd3-4500-bf18-8a0832e357cd" (UID: "fd3cc14b-9cd3-4500-bf18-8a0832e357cd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:36:24.298621 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:24.298593 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd3cc14b-9cd3-4500-bf18-8a0832e357cd-kube-api-access-2fvlp" pod "fd3cc14b-9cd3-4500-bf18-8a0832e357cd" (UID: "fd3cc14b-9cd3-4500-bf18-8a0832e357cd"). InnerVolumeSpecName "kube-api-access-2fvlp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:36:24.397887 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:24.397854 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2fvlp\" (UniqueName: \"kubernetes.io/projected/fd3cc14b-9cd3-4500-bf18-8a0832e357cd-kube-api-access-2fvlp\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:36:24.397887 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:24.397885 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd3cc14b-9cd3-4500-bf18-8a0832e357cd-maas-api-openshift-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:36:24.397887 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:24.397897 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd3cc14b-9cd3-4500-bf18-8a0832e357cd-maas-api-xks-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:36:25.073779 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:25.073746 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10585146-mkldd" Nov 02 06:36:25.073779 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:25.073759 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10585146-mkldd" event={"ID":"fd3cc14b-9cd3-4500-bf18-8a0832e357cd","Type":"ContainerDied","Data":"5be7db14cfb116492faa5fe5ed3c76042305e8330011c9e9e108719c21e909f1"} Nov 02 06:36:25.074010 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:25.073808 5916 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5be7db14cfb116492faa5fe5ed3c76042305e8330011c9e9e108719c21e909f1" Nov 02 06:36:33.062261 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:33.062165 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc8a56e2-v5l4q"] Nov 02 06:36:33.103952 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:33.103892 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc8a56e2-v5l4q"] Nov 02 06:36:33.848714 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:33.848673 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc7f508b-03c7-47a7-9750-a346b915b9f8" path="/var/lib/kubelet/pods/dc7f508b-03c7-47a7-9750-a346b915b9f8/volumes" Nov 02 06:36:36.556490 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:36.556457 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27a288b1-m49qv"] Nov 02 06:36:36.557931 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:36.557846 5916 cpu_manager.go:414] "containerMap: removing container" podUID="fd3cc14b-9cd3-4500-bf18-8a0832e357cd" containerName="revoke-keys" Nov 02 06:36:36.557931 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:36.557875 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd3cc14b-9cd3-4500-bf18-8a0832e357cd" containerName="revoke-keys" Nov 02 06:36:36.558144 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:36.558046 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd3cc14b-9cd3-4500-bf18-8a0832e357cd" containerName="revoke-keys" Nov 02 06:36:36.563843 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:36.563820 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27a288b1-m49qv" Nov 02 06:36:36.619333 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:36.619291 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27a288b1-m49qv"] Nov 02 06:36:36.653323 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:36.653291 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 06:36:36.653543 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:36.653340 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-68l4r\"" Nov 02 06:36:36.660410 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:36.660381 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fs8h2\" (UniqueName: \"kubernetes.io/projected/6b98be7f-f248-4ab5-9a80-f9d0f27cd6a4-kube-api-access-fs8h2\") pod \"maas-api-revoke-sub-27a288b1-m49qv\" (UID: \"6b98be7f-f248-4ab5-9a80-f9d0f27cd6a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27a288b1-m49qv" Nov 02 06:36:36.660548 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:36.660520 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b98be7f-f248-4ab5-9a80-f9d0f27cd6a4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-27a288b1-m49qv\" (UID: \"6b98be7f-f248-4ab5-9a80-f9d0f27cd6a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27a288b1-m49qv" Nov 02 06:36:36.660604 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:36.660589 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b98be7f-f248-4ab5-9a80-f9d0f27cd6a4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-27a288b1-m49qv\" (UID: \"6b98be7f-f248-4ab5-9a80-f9d0f27cd6a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27a288b1-m49qv" Nov 02 06:36:36.762474 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:36.762435 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fs8h2\" (UniqueName: \"kubernetes.io/projected/6b98be7f-f248-4ab5-9a80-f9d0f27cd6a4-kube-api-access-fs8h2\") pod \"maas-api-revoke-sub-27a288b1-m49qv\" (UID: \"6b98be7f-f248-4ab5-9a80-f9d0f27cd6a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27a288b1-m49qv" Nov 02 06:36:36.762686 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:36.762573 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b98be7f-f248-4ab5-9a80-f9d0f27cd6a4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-27a288b1-m49qv\" (UID: \"6b98be7f-f248-4ab5-9a80-f9d0f27cd6a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27a288b1-m49qv" Nov 02 06:36:36.762686 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:36.762670 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b98be7f-f248-4ab5-9a80-f9d0f27cd6a4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-27a288b1-m49qv\" (UID: \"6b98be7f-f248-4ab5-9a80-f9d0f27cd6a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27a288b1-m49qv" Nov 02 06:36:36.763444 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:36.763406 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b98be7f-f248-4ab5-9a80-f9d0f27cd6a4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-27a288b1-m49qv\" (UID: \"6b98be7f-f248-4ab5-9a80-f9d0f27cd6a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27a288b1-m49qv" Nov 02 06:36:36.765183 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:36.765158 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b98be7f-f248-4ab5-9a80-f9d0f27cd6a4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-27a288b1-m49qv\" (UID: \"6b98be7f-f248-4ab5-9a80-f9d0f27cd6a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27a288b1-m49qv" Nov 02 06:36:36.816790 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:36.816729 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fs8h2\" (UniqueName: \"kubernetes.io/projected/6b98be7f-f248-4ab5-9a80-f9d0f27cd6a4-kube-api-access-fs8h2\") pod \"maas-api-revoke-sub-27a288b1-m49qv\" (UID: \"6b98be7f-f248-4ab5-9a80-f9d0f27cd6a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27a288b1-m49qv" Nov 02 06:36:36.883727 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:36.883166 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27a288b1-m49qv" Nov 02 06:36:37.046657 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:37.046618 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0453c67-lct9k"] Nov 02 06:36:37.052209 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:37.052180 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0453c67-lct9k"] Nov 02 06:36:37.265724 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:37.265535 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27a288b1-m49qv"] Nov 02 06:36:37.268499 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:36:37.268458 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b98be7f_f248_4ab5_9a80_f9d0f27cd6a4.slice/crio-9e4625d095b25ce7c66a07d3c1d3d7264f830059606523c8eba14e8fc9e2b8c0 WatchSource:0}: Error finding container 9e4625d095b25ce7c66a07d3c1d3d7264f830059606523c8eba14e8fc9e2b8c0: Status 404 returned error can't find the container with id 9e4625d095b25ce7c66a07d3c1d3d7264f830059606523c8eba14e8fc9e2b8c0 Nov 02 06:36:37.775226 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:37.775192 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03165c46-z897v"] Nov 02 06:36:37.780127 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:37.780107 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03165c46-z897v" Nov 02 06:36:37.846753 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:37.846717 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9a183ab-1d0c-4b4b-92d5-b051c389aa53" path="/var/lib/kubelet/pods/e9a183ab-1d0c-4b4b-92d5-b051c389aa53/volumes" Nov 02 06:36:37.847173 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:37.847155 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03165c46-z897v"] Nov 02 06:36:37.882481 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:37.882451 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1acf763f-268c-4bd3-ac4a-3f701c0e2dbd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-03165c46-z897v\" (UID: \"1acf763f-268c-4bd3-ac4a-3f701c0e2dbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03165c46-z897v" Nov 02 06:36:37.882644 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:37.882502 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7vhs\" (UniqueName: \"kubernetes.io/projected/1acf763f-268c-4bd3-ac4a-3f701c0e2dbd-kube-api-access-x7vhs\") pod \"maas-api-revoke-sub-03165c46-z897v\" (UID: \"1acf763f-268c-4bd3-ac4a-3f701c0e2dbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03165c46-z897v" Nov 02 06:36:37.882719 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:37.882663 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1acf763f-268c-4bd3-ac4a-3f701c0e2dbd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-03165c46-z897v\" (UID: \"1acf763f-268c-4bd3-ac4a-3f701c0e2dbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03165c46-z897v" Nov 02 06:36:37.983293 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:37.983249 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1acf763f-268c-4bd3-ac4a-3f701c0e2dbd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-03165c46-z897v\" (UID: \"1acf763f-268c-4bd3-ac4a-3f701c0e2dbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03165c46-z897v" Nov 02 06:36:37.983467 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:37.983314 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7vhs\" (UniqueName: \"kubernetes.io/projected/1acf763f-268c-4bd3-ac4a-3f701c0e2dbd-kube-api-access-x7vhs\") pod \"maas-api-revoke-sub-03165c46-z897v\" (UID: \"1acf763f-268c-4bd3-ac4a-3f701c0e2dbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03165c46-z897v" Nov 02 06:36:37.983467 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:37.983406 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1acf763f-268c-4bd3-ac4a-3f701c0e2dbd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-03165c46-z897v\" (UID: \"1acf763f-268c-4bd3-ac4a-3f701c0e2dbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03165c46-z897v" Nov 02 06:36:37.984021 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:37.983999 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1acf763f-268c-4bd3-ac4a-3f701c0e2dbd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-03165c46-z897v\" (UID: \"1acf763f-268c-4bd3-ac4a-3f701c0e2dbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03165c46-z897v" Nov 02 06:36:37.985637 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:37.985610 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1acf763f-268c-4bd3-ac4a-3f701c0e2dbd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-03165c46-z897v\" (UID: \"1acf763f-268c-4bd3-ac4a-3f701c0e2dbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03165c46-z897v" Nov 02 06:36:38.038498 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:38.038434 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7vhs\" (UniqueName: \"kubernetes.io/projected/1acf763f-268c-4bd3-ac4a-3f701c0e2dbd-kube-api-access-x7vhs\") pod \"maas-api-revoke-sub-03165c46-z897v\" (UID: \"1acf763f-268c-4bd3-ac4a-3f701c0e2dbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03165c46-z897v" Nov 02 06:36:38.094278 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:38.094251 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03165c46-z897v" Nov 02 06:36:38.187335 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:38.187228 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27a288b1-m49qv" event={"ID":"6b98be7f-f248-4ab5-9a80-f9d0f27cd6a4","Type":"ContainerStarted","Data":"581a47bfcc87ad237583822cdb5786c083da41991a20596ec700f4c3e68fccc0"} Nov 02 06:36:38.187335 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:38.187289 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27a288b1-m49qv" event={"ID":"6b98be7f-f248-4ab5-9a80-f9d0f27cd6a4","Type":"ContainerStarted","Data":"9e4625d095b25ce7c66a07d3c1d3d7264f830059606523c8eba14e8fc9e2b8c0"} Nov 02 06:36:38.298634 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:38.298513 5916 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27a288b1-m49qv" podStartSLOduration=2.298494047 podStartE2EDuration="2.298494047s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:36:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:36:38.292813118 +0000 UTC m=+1186.564633512" watchObservedRunningTime="2026-11-02 06:36:38.298494047 +0000 UTC m=+1186.570314447" Nov 02 06:36:38.479769 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:38.479732 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03165c46-z897v"] Nov 02 06:36:38.484879 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:36:38.484853 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1acf763f_268c_4bd3_ac4a_3f701c0e2dbd.slice/crio-412c4b2ec4a7de28bd88f4e756a7f882b6afc5ea8611613a6da15e4aadeb4856 WatchSource:0}: Error finding container 412c4b2ec4a7de28bd88f4e756a7f882b6afc5ea8611613a6da15e4aadeb4856: Status 404 returned error can't find the container with id 412c4b2ec4a7de28bd88f4e756a7f882b6afc5ea8611613a6da15e4aadeb4856 Nov 02 06:36:39.196372 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:39.196330 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03165c46-z897v" event={"ID":"1acf763f-268c-4bd3-ac4a-3f701c0e2dbd","Type":"ContainerStarted","Data":"71bfeb7ce9a65b585643e1c948c075d2f7674d61d3a33856a176061345d85164"} Nov 02 06:36:39.196799 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:39.196392 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03165c46-z897v" event={"ID":"1acf763f-268c-4bd3-ac4a-3f701c0e2dbd","Type":"ContainerStarted","Data":"412c4b2ec4a7de28bd88f4e756a7f882b6afc5ea8611613a6da15e4aadeb4856"} Nov 02 06:36:39.198011 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:39.197982 5916 generic.go:356] "Generic (PLEG): container finished" podID="6b98be7f-f248-4ab5-9a80-f9d0f27cd6a4" containerID="581a47bfcc87ad237583822cdb5786c083da41991a20596ec700f4c3e68fccc0" exitCode=0 Nov 02 06:36:39.198104 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:39.198047 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27a288b1-m49qv" event={"ID":"6b98be7f-f248-4ab5-9a80-f9d0f27cd6a4","Type":"ContainerDied","Data":"581a47bfcc87ad237583822cdb5786c083da41991a20596ec700f4c3e68fccc0"} Nov 02 06:36:39.307746 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:39.307701 5916 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03165c46-z897v" podStartSLOduration=2.307688929 podStartE2EDuration="2.307688929s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:36:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:36:39.303309628 +0000 UTC m=+1187.575130022" watchObservedRunningTime="2026-11-02 06:36:39.307688929 +0000 UTC m=+1187.579509355" Nov 02 06:36:40.207085 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:40.207046 5916 generic.go:356] "Generic (PLEG): container finished" podID="1acf763f-268c-4bd3-ac4a-3f701c0e2dbd" containerID="71bfeb7ce9a65b585643e1c948c075d2f7674d61d3a33856a176061345d85164" exitCode=0 Nov 02 06:36:40.207514 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:40.207122 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03165c46-z897v" event={"ID":"1acf763f-268c-4bd3-ac4a-3f701c0e2dbd","Type":"ContainerDied","Data":"71bfeb7ce9a65b585643e1c948c075d2f7674d61d3a33856a176061345d85164"} Nov 02 06:36:40.362746 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:40.362714 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27a288b1-m49qv" Nov 02 06:36:40.406954 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:40.406925 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b98be7f-f248-4ab5-9a80-f9d0f27cd6a4-kube-api-access-fs8h2\" (UniqueName: \"kubernetes.io/projected/6b98be7f-f248-4ab5-9a80-f9d0f27cd6a4-kube-api-access-fs8h2\") pod \"6b98be7f-f248-4ab5-9a80-f9d0f27cd6a4\" (UID: \"6b98be7f-f248-4ab5-9a80-f9d0f27cd6a4\") " Nov 02 06:36:40.407123 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:40.407020 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b98be7f-f248-4ab5-9a80-f9d0f27cd6a4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b98be7f-f248-4ab5-9a80-f9d0f27cd6a4-maas-api-openshift-ca\") pod \"6b98be7f-f248-4ab5-9a80-f9d0f27cd6a4\" (UID: \"6b98be7f-f248-4ab5-9a80-f9d0f27cd6a4\") " Nov 02 06:36:40.407123 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:40.407083 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b98be7f-f248-4ab5-9a80-f9d0f27cd6a4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b98be7f-f248-4ab5-9a80-f9d0f27cd6a4-maas-api-xks-ca\") pod \"6b98be7f-f248-4ab5-9a80-f9d0f27cd6a4\" (UID: \"6b98be7f-f248-4ab5-9a80-f9d0f27cd6a4\") " Nov 02 06:36:40.407425 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:40.407399 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b98be7f-f248-4ab5-9a80-f9d0f27cd6a4-maas-api-openshift-ca" pod "6b98be7f-f248-4ab5-9a80-f9d0f27cd6a4" (UID: "6b98be7f-f248-4ab5-9a80-f9d0f27cd6a4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:36:40.409031 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:40.409001 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b98be7f-f248-4ab5-9a80-f9d0f27cd6a4-kube-api-access-fs8h2" pod "6b98be7f-f248-4ab5-9a80-f9d0f27cd6a4" (UID: "6b98be7f-f248-4ab5-9a80-f9d0f27cd6a4"). InnerVolumeSpecName "kube-api-access-fs8h2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:36:40.409167 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:40.409045 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b98be7f-f248-4ab5-9a80-f9d0f27cd6a4-maas-api-xks-ca" pod "6b98be7f-f248-4ab5-9a80-f9d0f27cd6a4" (UID: "6b98be7f-f248-4ab5-9a80-f9d0f27cd6a4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:36:40.508652 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:40.508580 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b98be7f-f248-4ab5-9a80-f9d0f27cd6a4-maas-api-xks-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:36:40.508652 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:40.508607 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fs8h2\" (UniqueName: \"kubernetes.io/projected/6b98be7f-f248-4ab5-9a80-f9d0f27cd6a4-kube-api-access-fs8h2\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:36:40.508652 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:40.508618 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b98be7f-f248-4ab5-9a80-f9d0f27cd6a4-maas-api-openshift-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:36:41.215435 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:41.215399 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27a288b1-m49qv" Nov 02 06:36:41.215949 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:41.215434 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27a288b1-m49qv" event={"ID":"6b98be7f-f248-4ab5-9a80-f9d0f27cd6a4","Type":"ContainerDied","Data":"9e4625d095b25ce7c66a07d3c1d3d7264f830059606523c8eba14e8fc9e2b8c0"} Nov 02 06:36:41.215949 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:41.215479 5916 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9e4625d095b25ce7c66a07d3c1d3d7264f830059606523c8eba14e8fc9e2b8c0" Nov 02 06:36:41.364060 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:41.364036 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03165c46-z897v" Nov 02 06:36:41.420220 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:41.420192 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1acf763f-268c-4bd3-ac4a-3f701c0e2dbd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1acf763f-268c-4bd3-ac4a-3f701c0e2dbd-maas-api-xks-ca\") pod \"1acf763f-268c-4bd3-ac4a-3f701c0e2dbd\" (UID: \"1acf763f-268c-4bd3-ac4a-3f701c0e2dbd\") " Nov 02 06:36:41.420368 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:41.420255 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1acf763f-268c-4bd3-ac4a-3f701c0e2dbd-kube-api-access-x7vhs\" (UniqueName: \"kubernetes.io/projected/1acf763f-268c-4bd3-ac4a-3f701c0e2dbd-kube-api-access-x7vhs\") pod \"1acf763f-268c-4bd3-ac4a-3f701c0e2dbd\" (UID: \"1acf763f-268c-4bd3-ac4a-3f701c0e2dbd\") " Nov 02 06:36:41.420368 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:41.420315 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1acf763f-268c-4bd3-ac4a-3f701c0e2dbd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1acf763f-268c-4bd3-ac4a-3f701c0e2dbd-maas-api-openshift-ca\") pod \"1acf763f-268c-4bd3-ac4a-3f701c0e2dbd\" (UID: \"1acf763f-268c-4bd3-ac4a-3f701c0e2dbd\") " Nov 02 06:36:41.420708 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:41.420680 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1acf763f-268c-4bd3-ac4a-3f701c0e2dbd-maas-api-openshift-ca" pod "1acf763f-268c-4bd3-ac4a-3f701c0e2dbd" (UID: "1acf763f-268c-4bd3-ac4a-3f701c0e2dbd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:36:41.420867 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:41.420846 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1acf763f-268c-4bd3-ac4a-3f701c0e2dbd-maas-api-openshift-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:36:41.422266 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:41.422244 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1acf763f-268c-4bd3-ac4a-3f701c0e2dbd-maas-api-xks-ca" pod "1acf763f-268c-4bd3-ac4a-3f701c0e2dbd" (UID: "1acf763f-268c-4bd3-ac4a-3f701c0e2dbd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:36:41.422381 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:41.422360 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1acf763f-268c-4bd3-ac4a-3f701c0e2dbd-kube-api-access-x7vhs" pod "1acf763f-268c-4bd3-ac4a-3f701c0e2dbd" (UID: "1acf763f-268c-4bd3-ac4a-3f701c0e2dbd"). InnerVolumeSpecName "kube-api-access-x7vhs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:36:41.521628 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:41.521550 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1acf763f-268c-4bd3-ac4a-3f701c0e2dbd-maas-api-xks-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:36:41.521628 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:41.521580 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7vhs\" (UniqueName: \"kubernetes.io/projected/1acf763f-268c-4bd3-ac4a-3f701c0e2dbd-kube-api-access-x7vhs\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:36:42.224929 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:42.224873 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03165c46-z897v" Nov 02 06:36:42.225317 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:42.224934 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03165c46-z897v" event={"ID":"1acf763f-268c-4bd3-ac4a-3f701c0e2dbd","Type":"ContainerDied","Data":"412c4b2ec4a7de28bd88f4e756a7f882b6afc5ea8611613a6da15e4aadeb4856"} Nov 02 06:36:42.225317 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:42.224989 5916 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="412c4b2ec4a7de28bd88f4e756a7f882b6afc5ea8611613a6da15e4aadeb4856" Nov 02 06:36:43.046392 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:43.046351 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eeebfe4d-2pfpp"] Nov 02 06:36:43.092474 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:43.092443 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eeebfe4d-2pfpp"] Nov 02 06:36:43.598765 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:43.598731 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f9f9a772-sgd5d"] Nov 02 06:36:43.599480 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:43.599451 5916 cpu_manager.go:414] "containerMap: removing container" podUID="6b98be7f-f248-4ab5-9a80-f9d0f27cd6a4" containerName="revoke-keys" Nov 02 06:36:43.599480 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:43.599471 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b98be7f-f248-4ab5-9a80-f9d0f27cd6a4" containerName="revoke-keys" Nov 02 06:36:43.599480 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:43.599480 5916 cpu_manager.go:414] "containerMap: removing container" podUID="1acf763f-268c-4bd3-ac4a-3f701c0e2dbd" containerName="revoke-keys" Nov 02 06:36:43.599480 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:43.599486 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1acf763f-268c-4bd3-ac4a-3f701c0e2dbd" containerName="revoke-keys" Nov 02 06:36:43.599740 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:43.599607 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b98be7f-f248-4ab5-9a80-f9d0f27cd6a4" containerName="revoke-keys" Nov 02 06:36:43.599740 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:43.599614 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="1acf763f-268c-4bd3-ac4a-3f701c0e2dbd" containerName="revoke-keys" Nov 02 06:36:43.603267 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:43.603246 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9f9a772-sgd5d" Nov 02 06:36:43.647566 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:43.647532 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sk5kx\" (UniqueName: \"kubernetes.io/projected/253c37b4-7bd4-4310-b8be-b37c64f6b57a-kube-api-access-sk5kx\") pod \"maas-api-revoke-sub-f9f9a772-sgd5d\" (UID: \"253c37b4-7bd4-4310-b8be-b37c64f6b57a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9f9a772-sgd5d" Nov 02 06:36:43.647731 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:43.647620 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/253c37b4-7bd4-4310-b8be-b37c64f6b57a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f9f9a772-sgd5d\" (UID: \"253c37b4-7bd4-4310-b8be-b37c64f6b57a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9f9a772-sgd5d" Nov 02 06:36:43.647731 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:43.647700 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/253c37b4-7bd4-4310-b8be-b37c64f6b57a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f9f9a772-sgd5d\" (UID: \"253c37b4-7bd4-4310-b8be-b37c64f6b57a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9f9a772-sgd5d" Nov 02 06:36:43.655696 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:43.655669 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f9f9a772-sgd5d"] Nov 02 06:36:43.696596 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:43.696566 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 06:36:43.696773 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:43.696615 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-68l4r\"" Nov 02 06:36:43.748846 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:43.748816 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sk5kx\" (UniqueName: \"kubernetes.io/projected/253c37b4-7bd4-4310-b8be-b37c64f6b57a-kube-api-access-sk5kx\") pod \"maas-api-revoke-sub-f9f9a772-sgd5d\" (UID: \"253c37b4-7bd4-4310-b8be-b37c64f6b57a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9f9a772-sgd5d" Nov 02 06:36:43.749076 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:43.748864 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/253c37b4-7bd4-4310-b8be-b37c64f6b57a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f9f9a772-sgd5d\" (UID: \"253c37b4-7bd4-4310-b8be-b37c64f6b57a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9f9a772-sgd5d" Nov 02 06:36:43.749076 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:43.749011 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/253c37b4-7bd4-4310-b8be-b37c64f6b57a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f9f9a772-sgd5d\" (UID: \"253c37b4-7bd4-4310-b8be-b37c64f6b57a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9f9a772-sgd5d" Nov 02 06:36:43.749669 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:43.749646 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/253c37b4-7bd4-4310-b8be-b37c64f6b57a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f9f9a772-sgd5d\" (UID: \"253c37b4-7bd4-4310-b8be-b37c64f6b57a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9f9a772-sgd5d" Nov 02 06:36:43.751149 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:43.751125 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/253c37b4-7bd4-4310-b8be-b37c64f6b57a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f9f9a772-sgd5d\" (UID: \"253c37b4-7bd4-4310-b8be-b37c64f6b57a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9f9a772-sgd5d" Nov 02 06:36:43.802076 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:43.802039 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sk5kx\" (UniqueName: \"kubernetes.io/projected/253c37b4-7bd4-4310-b8be-b37c64f6b57a-kube-api-access-sk5kx\") pod \"maas-api-revoke-sub-f9f9a772-sgd5d\" (UID: \"253c37b4-7bd4-4310-b8be-b37c64f6b57a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9f9a772-sgd5d" Nov 02 06:36:43.842409 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:43.842376 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f21f10e-9b76-4a63-9e50-ff819e919b70" path="/var/lib/kubelet/pods/7f21f10e-9b76-4a63-9e50-ff819e919b70/volumes" Nov 02 06:36:43.916167 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:43.916140 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9f9a772-sgd5d" Nov 02 06:36:44.291741 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:44.291663 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f9f9a772-sgd5d"] Nov 02 06:36:44.295794 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:36:44.295767 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod253c37b4_7bd4_4310_b8be_b37c64f6b57a.slice/crio-5afecd858ff30f56c950a2d85306620a332a5d8e548f18d4b6b527334eda2dc4 WatchSource:0}: Error finding container 5afecd858ff30f56c950a2d85306620a332a5d8e548f18d4b6b527334eda2dc4: Status 404 returned error can't find the container with id 5afecd858ff30f56c950a2d85306620a332a5d8e548f18d4b6b527334eda2dc4 Nov 02 06:36:45.259607 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:45.259573 5916 generic.go:356] "Generic (PLEG): container finished" podID="253c37b4-7bd4-4310-b8be-b37c64f6b57a" containerID="4c494a6cb9eae74e1c14a0e9d6e5ae9e4c9571384b881727c76f9397b5c815af" exitCode=0 Nov 02 06:36:45.259999 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:45.259633 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9f9a772-sgd5d" event={"ID":"253c37b4-7bd4-4310-b8be-b37c64f6b57a","Type":"ContainerDied","Data":"4c494a6cb9eae74e1c14a0e9d6e5ae9e4c9571384b881727c76f9397b5c815af"} Nov 02 06:36:45.259999 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:45.259682 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9f9a772-sgd5d" event={"ID":"253c37b4-7bd4-4310-b8be-b37c64f6b57a","Type":"ContainerStarted","Data":"5afecd858ff30f56c950a2d85306620a332a5d8e548f18d4b6b527334eda2dc4"} Nov 02 06:36:46.013984 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:46.013952 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-9mj97"] Nov 02 06:36:46.018276 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:46.018254 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-9mj97" Nov 02 06:36:46.066776 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:46.066743 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-581608-openshift-default-dockercfg-bm4lm\"" Nov 02 06:36:46.076359 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:46.076333 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ee541ef-78e0-4075-9780-eb53b607ddcc-workload-certs\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-9mj97\" (UID: \"9ee541ef-78e0-4075-9780-eb53b607ddcc\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-9mj97" Nov 02 06:36:46.076498 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:46.076396 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ee541ef-78e0-4075-9780-eb53b607ddcc-istio-data\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-9mj97\" (UID: \"9ee541ef-78e0-4075-9780-eb53b607ddcc\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-9mj97" Nov 02 06:36:46.076540 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:46.076491 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmdzl\" (UniqueName: \"kubernetes.io/projected/9ee541ef-78e0-4075-9780-eb53b607ddcc-kube-api-access-zmdzl\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-9mj97\" (UID: \"9ee541ef-78e0-4075-9780-eb53b607ddcc\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-9mj97" Nov 02 06:36:46.076600 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:46.076584 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ee541ef-78e0-4075-9780-eb53b607ddcc-istio-podinfo\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-9mj97\" (UID: \"9ee541ef-78e0-4075-9780-eb53b607ddcc\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-9mj97" Nov 02 06:36:46.076633 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:46.076611 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ee541ef-78e0-4075-9780-eb53b607ddcc-credential-socket\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-9mj97\" (UID: \"9ee541ef-78e0-4075-9780-eb53b607ddcc\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-9mj97" Nov 02 06:36:46.076714 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:46.076701 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ee541ef-78e0-4075-9780-eb53b607ddcc-workload-socket\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-9mj97\" (UID: \"9ee541ef-78e0-4075-9780-eb53b607ddcc\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-9mj97" Nov 02 06:36:46.076755 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:46.076745 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ee541ef-78e0-4075-9780-eb53b607ddcc-istio-envoy\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-9mj97\" (UID: \"9ee541ef-78e0-4075-9780-eb53b607ddcc\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-9mj97" Nov 02 06:36:46.076809 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:46.076796 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ee541ef-78e0-4075-9780-eb53b607ddcc-istio-token\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-9mj97\" (UID: \"9ee541ef-78e0-4075-9780-eb53b607ddcc\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-9mj97" Nov 02 06:36:46.076843 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:46.076817 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ee541ef-78e0-4075-9780-eb53b607ddcc-istiod-ca-cert\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-9mj97\" (UID: \"9ee541ef-78e0-4075-9780-eb53b607ddcc\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-9mj97" Nov 02 06:36:46.119949 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:46.119898 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-9mj97"] Nov 02 06:36:46.178194 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:46.178156 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ee541ef-78e0-4075-9780-eb53b607ddcc-workload-socket\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-9mj97\" (UID: \"9ee541ef-78e0-4075-9780-eb53b607ddcc\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-9mj97" Nov 02 06:36:46.178404 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:46.178220 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ee541ef-78e0-4075-9780-eb53b607ddcc-istio-envoy\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-9mj97\" (UID: \"9ee541ef-78e0-4075-9780-eb53b607ddcc\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-9mj97" Nov 02 06:36:46.178404 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:46.178255 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ee541ef-78e0-4075-9780-eb53b607ddcc-istio-token\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-9mj97\" (UID: \"9ee541ef-78e0-4075-9780-eb53b607ddcc\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-9mj97" Nov 02 06:36:46.178404 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:46.178269 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ee541ef-78e0-4075-9780-eb53b607ddcc-istiod-ca-cert\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-9mj97\" (UID: \"9ee541ef-78e0-4075-9780-eb53b607ddcc\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-9mj97" Nov 02 06:36:46.178404 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:46.178302 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ee541ef-78e0-4075-9780-eb53b607ddcc-workload-certs\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-9mj97\" (UID: \"9ee541ef-78e0-4075-9780-eb53b607ddcc\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-9mj97" Nov 02 06:36:46.178404 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:46.178385 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ee541ef-78e0-4075-9780-eb53b607ddcc-istio-data\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-9mj97\" (UID: \"9ee541ef-78e0-4075-9780-eb53b607ddcc\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-9mj97" Nov 02 06:36:46.178696 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:46.178428 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmdzl\" (UniqueName: \"kubernetes.io/projected/9ee541ef-78e0-4075-9780-eb53b607ddcc-kube-api-access-zmdzl\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-9mj97\" (UID: \"9ee541ef-78e0-4075-9780-eb53b607ddcc\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-9mj97" Nov 02 06:36:46.178696 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:46.178504 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ee541ef-78e0-4075-9780-eb53b607ddcc-istio-podinfo\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-9mj97\" (UID: \"9ee541ef-78e0-4075-9780-eb53b607ddcc\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-9mj97" Nov 02 06:36:46.178696 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:46.178530 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ee541ef-78e0-4075-9780-eb53b607ddcc-credential-socket\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-9mj97\" (UID: \"9ee541ef-78e0-4075-9780-eb53b607ddcc\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-9mj97" Nov 02 06:36:46.178696 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:46.178639 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ee541ef-78e0-4075-9780-eb53b607ddcc-workload-socket\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-9mj97\" (UID: \"9ee541ef-78e0-4075-9780-eb53b607ddcc\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-9mj97" Nov 02 06:36:46.178939 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:46.178738 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ee541ef-78e0-4075-9780-eb53b607ddcc-workload-certs\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-9mj97\" (UID: \"9ee541ef-78e0-4075-9780-eb53b607ddcc\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-9mj97" Nov 02 06:36:46.179119 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:46.179081 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ee541ef-78e0-4075-9780-eb53b607ddcc-istio-data\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-9mj97\" (UID: \"9ee541ef-78e0-4075-9780-eb53b607ddcc\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-9mj97" Nov 02 06:36:46.179195 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:46.179113 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ee541ef-78e0-4075-9780-eb53b607ddcc-credential-socket\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-9mj97\" (UID: \"9ee541ef-78e0-4075-9780-eb53b607ddcc\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-9mj97" Nov 02 06:36:46.179254 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:46.179212 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ee541ef-78e0-4075-9780-eb53b607ddcc-istiod-ca-cert\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-9mj97\" (UID: \"9ee541ef-78e0-4075-9780-eb53b607ddcc\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-9mj97" Nov 02 06:36:46.181011 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:46.180985 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ee541ef-78e0-4075-9780-eb53b607ddcc-istio-envoy\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-9mj97\" (UID: \"9ee541ef-78e0-4075-9780-eb53b607ddcc\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-9mj97" Nov 02 06:36:46.181304 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:46.181281 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ee541ef-78e0-4075-9780-eb53b607ddcc-istio-podinfo\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-9mj97\" (UID: \"9ee541ef-78e0-4075-9780-eb53b607ddcc\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-9mj97" Nov 02 06:36:46.233136 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:46.233107 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ee541ef-78e0-4075-9780-eb53b607ddcc-istio-token\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-9mj97\" (UID: \"9ee541ef-78e0-4075-9780-eb53b607ddcc\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-9mj97" Nov 02 06:36:46.233263 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:46.233226 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmdzl\" (UniqueName: \"kubernetes.io/projected/9ee541ef-78e0-4075-9780-eb53b607ddcc-kube-api-access-zmdzl\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-9mj97\" (UID: \"9ee541ef-78e0-4075-9780-eb53b607ddcc\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-9mj97" Nov 02 06:36:46.332742 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:46.332709 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-9mj97" Nov 02 06:36:46.403089 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:46.403062 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9f9a772-sgd5d" Nov 02 06:36:46.481507 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:46.481468 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/253c37b4-7bd4-4310-b8be-b37c64f6b57a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/253c37b4-7bd4-4310-b8be-b37c64f6b57a-maas-api-openshift-ca\") pod \"253c37b4-7bd4-4310-b8be-b37c64f6b57a\" (UID: \"253c37b4-7bd4-4310-b8be-b37c64f6b57a\") " Nov 02 06:36:46.481693 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:46.481584 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/253c37b4-7bd4-4310-b8be-b37c64f6b57a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/253c37b4-7bd4-4310-b8be-b37c64f6b57a-maas-api-xks-ca\") pod \"253c37b4-7bd4-4310-b8be-b37c64f6b57a\" (UID: \"253c37b4-7bd4-4310-b8be-b37c64f6b57a\") " Nov 02 06:36:46.481795 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:46.481775 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/253c37b4-7bd4-4310-b8be-b37c64f6b57a-kube-api-access-sk5kx\" (UniqueName: \"kubernetes.io/projected/253c37b4-7bd4-4310-b8be-b37c64f6b57a-kube-api-access-sk5kx\") pod \"253c37b4-7bd4-4310-b8be-b37c64f6b57a\" (UID: \"253c37b4-7bd4-4310-b8be-b37c64f6b57a\") " Nov 02 06:36:46.481916 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:46.481874 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/253c37b4-7bd4-4310-b8be-b37c64f6b57a-maas-api-openshift-ca" pod "253c37b4-7bd4-4310-b8be-b37c64f6b57a" (UID: "253c37b4-7bd4-4310-b8be-b37c64f6b57a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:36:46.482165 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:46.482148 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/253c37b4-7bd4-4310-b8be-b37c64f6b57a-maas-api-openshift-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:36:46.483761 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:46.483738 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/253c37b4-7bd4-4310-b8be-b37c64f6b57a-maas-api-xks-ca" pod "253c37b4-7bd4-4310-b8be-b37c64f6b57a" (UID: "253c37b4-7bd4-4310-b8be-b37c64f6b57a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:36:46.483814 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:46.483760 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/253c37b4-7bd4-4310-b8be-b37c64f6b57a-kube-api-access-sk5kx" pod "253c37b4-7bd4-4310-b8be-b37c64f6b57a" (UID: "253c37b4-7bd4-4310-b8be-b37c64f6b57a"). InnerVolumeSpecName "kube-api-access-sk5kx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:36:46.583815 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:46.583709 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/253c37b4-7bd4-4310-b8be-b37c64f6b57a-maas-api-xks-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:36:46.583815 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:46.583749 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sk5kx\" (UniqueName: \"kubernetes.io/projected/253c37b4-7bd4-4310-b8be-b37c64f6b57a-kube-api-access-sk5kx\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:36:46.591048 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:46.591014 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-wzvj7"] Nov 02 06:36:46.592960 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:46.592930 5916 cpu_manager.go:414] "containerMap: removing container" podUID="253c37b4-7bd4-4310-b8be-b37c64f6b57a" containerName="revoke-keys" Nov 02 06:36:46.593111 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:46.592962 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="253c37b4-7bd4-4310-b8be-b37c64f6b57a" containerName="revoke-keys" Nov 02 06:36:46.593268 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:46.593252 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="253c37b4-7bd4-4310-b8be-b37c64f6b57a" containerName="revoke-keys" Nov 02 06:36:46.602781 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:46.602758 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-wzvj7" Nov 02 06:36:46.650357 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:46.650326 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-wzvj7"] Nov 02 06:36:46.684518 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:46.684488 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/beb17737-f6a0-47e1-8524-e56e8d267502-workload-socket\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-wzvj7\" (UID: \"beb17737-f6a0-47e1-8524-e56e8d267502\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-wzvj7" Nov 02 06:36:46.684707 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:46.684543 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/beb17737-f6a0-47e1-8524-e56e8d267502-istiod-ca-cert\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-wzvj7\" (UID: \"beb17737-f6a0-47e1-8524-e56e8d267502\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-wzvj7" Nov 02 06:36:46.684798 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:46.684723 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/beb17737-f6a0-47e1-8524-e56e8d267502-credential-socket\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-wzvj7\" (UID: \"beb17737-f6a0-47e1-8524-e56e8d267502\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-wzvj7" Nov 02 06:36:46.684864 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:46.684800 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/beb17737-f6a0-47e1-8524-e56e8d267502-istio-envoy\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-wzvj7\" (UID: \"beb17737-f6a0-47e1-8524-e56e8d267502\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-wzvj7" Nov 02 06:36:46.684934 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:46.684860 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/beb17737-f6a0-47e1-8524-e56e8d267502-istio-podinfo\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-wzvj7\" (UID: \"beb17737-f6a0-47e1-8524-e56e8d267502\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-wzvj7" Nov 02 06:36:46.684998 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:46.684972 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hf4j6\" (UniqueName: \"kubernetes.io/projected/beb17737-f6a0-47e1-8524-e56e8d267502-kube-api-access-hf4j6\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-wzvj7\" (UID: \"beb17737-f6a0-47e1-8524-e56e8d267502\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-wzvj7" Nov 02 06:36:46.685140 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:46.685114 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/beb17737-f6a0-47e1-8524-e56e8d267502-istio-data\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-wzvj7\" (UID: \"beb17737-f6a0-47e1-8524-e56e8d267502\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-wzvj7" Nov 02 06:36:46.685303 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:46.685166 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/beb17737-f6a0-47e1-8524-e56e8d267502-workload-certs\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-wzvj7\" (UID: \"beb17737-f6a0-47e1-8524-e56e8d267502\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-wzvj7" Nov 02 06:36:46.685424 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:46.685288 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/beb17737-f6a0-47e1-8524-e56e8d267502-istio-token\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-wzvj7\" (UID: \"beb17737-f6a0-47e1-8524-e56e8d267502\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-wzvj7" Nov 02 06:36:46.786454 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:46.786405 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/beb17737-f6a0-47e1-8524-e56e8d267502-istio-envoy\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-wzvj7\" (UID: \"beb17737-f6a0-47e1-8524-e56e8d267502\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-wzvj7" Nov 02 06:36:46.786454 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:46.786452 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/beb17737-f6a0-47e1-8524-e56e8d267502-istio-podinfo\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-wzvj7\" (UID: \"beb17737-f6a0-47e1-8524-e56e8d267502\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-wzvj7" Nov 02 06:36:46.786682 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:46.786479 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hf4j6\" (UniqueName: \"kubernetes.io/projected/beb17737-f6a0-47e1-8524-e56e8d267502-kube-api-access-hf4j6\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-wzvj7\" (UID: \"beb17737-f6a0-47e1-8524-e56e8d267502\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-wzvj7" Nov 02 06:36:46.786682 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:46.786556 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/beb17737-f6a0-47e1-8524-e56e8d267502-istio-data\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-wzvj7\" (UID: \"beb17737-f6a0-47e1-8524-e56e8d267502\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-wzvj7" Nov 02 06:36:46.786682 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:46.786600 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/beb17737-f6a0-47e1-8524-e56e8d267502-workload-certs\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-wzvj7\" (UID: \"beb17737-f6a0-47e1-8524-e56e8d267502\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-wzvj7" Nov 02 06:36:46.786682 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:46.786633 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/beb17737-f6a0-47e1-8524-e56e8d267502-istio-token\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-wzvj7\" (UID: \"beb17737-f6a0-47e1-8524-e56e8d267502\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-wzvj7" Nov 02 06:36:46.786893 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:46.786695 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/beb17737-f6a0-47e1-8524-e56e8d267502-workload-socket\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-wzvj7\" (UID: \"beb17737-f6a0-47e1-8524-e56e8d267502\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-wzvj7" Nov 02 06:36:46.786893 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:46.786736 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/beb17737-f6a0-47e1-8524-e56e8d267502-istiod-ca-cert\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-wzvj7\" (UID: \"beb17737-f6a0-47e1-8524-e56e8d267502\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-wzvj7" Nov 02 06:36:46.786893 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:46.786836 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/beb17737-f6a0-47e1-8524-e56e8d267502-credential-socket\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-wzvj7\" (UID: \"beb17737-f6a0-47e1-8524-e56e8d267502\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-wzvj7" Nov 02 06:36:46.787426 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:46.787156 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/beb17737-f6a0-47e1-8524-e56e8d267502-workload-certs\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-wzvj7\" (UID: \"beb17737-f6a0-47e1-8524-e56e8d267502\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-wzvj7" Nov 02 06:36:46.787426 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:46.787213 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/beb17737-f6a0-47e1-8524-e56e8d267502-workload-socket\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-wzvj7\" (UID: \"beb17737-f6a0-47e1-8524-e56e8d267502\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-wzvj7" Nov 02 06:36:46.787426 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:46.787233 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/beb17737-f6a0-47e1-8524-e56e8d267502-credential-socket\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-wzvj7\" (UID: \"beb17737-f6a0-47e1-8524-e56e8d267502\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-wzvj7" Nov 02 06:36:46.787426 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:46.787396 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/beb17737-f6a0-47e1-8524-e56e8d267502-istio-data\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-wzvj7\" (UID: \"beb17737-f6a0-47e1-8524-e56e8d267502\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-wzvj7" Nov 02 06:36:46.788101 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:46.788073 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/beb17737-f6a0-47e1-8524-e56e8d267502-istiod-ca-cert\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-wzvj7\" (UID: \"beb17737-f6a0-47e1-8524-e56e8d267502\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-wzvj7" Nov 02 06:36:46.789168 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:46.789148 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/beb17737-f6a0-47e1-8524-e56e8d267502-istio-podinfo\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-wzvj7\" (UID: \"beb17737-f6a0-47e1-8524-e56e8d267502\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-wzvj7" Nov 02 06:36:46.789340 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:46.789318 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/beb17737-f6a0-47e1-8524-e56e8d267502-istio-envoy\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-wzvj7\" (UID: \"beb17737-f6a0-47e1-8524-e56e8d267502\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-wzvj7" Nov 02 06:36:46.826031 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:46.826003 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-9mj97"] Nov 02 06:36:46.830784 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:36:46.830763 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ee541ef_78e0_4075_9780_eb53b607ddcc.slice/crio-7f9381e29b4e774b1697710c4268a04db70e6a6bdf68145e24f5e2a3026ef571 WatchSource:0}: Error finding container 7f9381e29b4e774b1697710c4268a04db70e6a6bdf68145e24f5e2a3026ef571: Status 404 returned error can't find the container with id 7f9381e29b4e774b1697710c4268a04db70e6a6bdf68145e24f5e2a3026ef571 Nov 02 06:36:46.833220 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:46.833191 5916 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 06:36:46.833291 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:46.833254 5916 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 06:36:46.833291 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:46.833283 5916 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 06:36:46.846553 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:46.846530 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hf4j6\" (UniqueName: \"kubernetes.io/projected/beb17737-f6a0-47e1-8524-e56e8d267502-kube-api-access-hf4j6\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-wzvj7\" (UID: \"beb17737-f6a0-47e1-8524-e56e8d267502\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-wzvj7" Nov 02 06:36:46.846656 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:46.846617 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/beb17737-f6a0-47e1-8524-e56e8d267502-istio-token\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-wzvj7\" (UID: \"beb17737-f6a0-47e1-8524-e56e8d267502\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-wzvj7" Nov 02 06:36:46.894241 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:46.894217 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-tkzw9"] Nov 02 06:36:46.898395 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:46.898370 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-tkzw9" Nov 02 06:36:46.917454 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:46.917429 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-wzvj7" Nov 02 06:36:46.950924 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:46.950204 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-dcf9c3-openshift-default-dockercfg-7kbz6\"" Nov 02 06:36:46.962876 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:46.962849 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-tkzw9"] Nov 02 06:36:46.988832 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:46.988798 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e77a579-9ee6-4304-85b6-96d292d0bb1d-istio-token\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-tkzw9\" (UID: \"3e77a579-9ee6-4304-85b6-96d292d0bb1d\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-tkzw9" Nov 02 06:36:46.988966 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:46.988882 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdjqh\" (UniqueName: \"kubernetes.io/projected/3e77a579-9ee6-4304-85b6-96d292d0bb1d-kube-api-access-vdjqh\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-tkzw9\" (UID: \"3e77a579-9ee6-4304-85b6-96d292d0bb1d\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-tkzw9" Nov 02 06:36:46.989971 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:46.989941 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e77a579-9ee6-4304-85b6-96d292d0bb1d-istiod-ca-cert\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-tkzw9\" (UID: \"3e77a579-9ee6-4304-85b6-96d292d0bb1d\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-tkzw9" Nov 02 06:36:46.990176 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:46.990151 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e77a579-9ee6-4304-85b6-96d292d0bb1d-istio-envoy\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-tkzw9\" (UID: \"3e77a579-9ee6-4304-85b6-96d292d0bb1d\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-tkzw9" Nov 02 06:36:46.990282 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:46.990220 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e77a579-9ee6-4304-85b6-96d292d0bb1d-workload-socket\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-tkzw9\" (UID: \"3e77a579-9ee6-4304-85b6-96d292d0bb1d\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-tkzw9" Nov 02 06:36:46.990917 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:46.990884 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e77a579-9ee6-4304-85b6-96d292d0bb1d-credential-socket\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-tkzw9\" (UID: \"3e77a579-9ee6-4304-85b6-96d292d0bb1d\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-tkzw9" Nov 02 06:36:46.991026 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:46.991006 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e77a579-9ee6-4304-85b6-96d292d0bb1d-istio-data\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-tkzw9\" (UID: \"3e77a579-9ee6-4304-85b6-96d292d0bb1d\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-tkzw9" Nov 02 06:36:46.991098 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:46.991082 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e77a579-9ee6-4304-85b6-96d292d0bb1d-istio-podinfo\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-tkzw9\" (UID: \"3e77a579-9ee6-4304-85b6-96d292d0bb1d\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-tkzw9" Nov 02 06:36:46.991172 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:46.991158 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e77a579-9ee6-4304-85b6-96d292d0bb1d-workload-certs\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-tkzw9\" (UID: \"3e77a579-9ee6-4304-85b6-96d292d0bb1d\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-tkzw9" Nov 02 06:36:47.091980 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:47.091881 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e77a579-9ee6-4304-85b6-96d292d0bb1d-istio-data\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-tkzw9\" (UID: \"3e77a579-9ee6-4304-85b6-96d292d0bb1d\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-tkzw9" Nov 02 06:36:47.091980 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:47.091951 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e77a579-9ee6-4304-85b6-96d292d0bb1d-istio-podinfo\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-tkzw9\" (UID: \"3e77a579-9ee6-4304-85b6-96d292d0bb1d\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-tkzw9" Nov 02 06:36:47.092213 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:47.091992 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e77a579-9ee6-4304-85b6-96d292d0bb1d-workload-certs\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-tkzw9\" (UID: \"3e77a579-9ee6-4304-85b6-96d292d0bb1d\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-tkzw9" Nov 02 06:36:47.092213 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:47.092022 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e77a579-9ee6-4304-85b6-96d292d0bb1d-istio-token\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-tkzw9\" (UID: \"3e77a579-9ee6-4304-85b6-96d292d0bb1d\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-tkzw9" Nov 02 06:36:47.092213 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:47.092075 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdjqh\" (UniqueName: \"kubernetes.io/projected/3e77a579-9ee6-4304-85b6-96d292d0bb1d-kube-api-access-vdjqh\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-tkzw9\" (UID: \"3e77a579-9ee6-4304-85b6-96d292d0bb1d\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-tkzw9" Nov 02 06:36:47.092213 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:47.092102 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e77a579-9ee6-4304-85b6-96d292d0bb1d-istiod-ca-cert\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-tkzw9\" (UID: \"3e77a579-9ee6-4304-85b6-96d292d0bb1d\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-tkzw9" Nov 02 06:36:47.092213 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:47.092143 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e77a579-9ee6-4304-85b6-96d292d0bb1d-istio-envoy\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-tkzw9\" (UID: \"3e77a579-9ee6-4304-85b6-96d292d0bb1d\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-tkzw9" Nov 02 06:36:47.092213 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:47.092193 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e77a579-9ee6-4304-85b6-96d292d0bb1d-workload-socket\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-tkzw9\" (UID: \"3e77a579-9ee6-4304-85b6-96d292d0bb1d\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-tkzw9" Nov 02 06:36:47.092548 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:47.092275 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e77a579-9ee6-4304-85b6-96d292d0bb1d-credential-socket\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-tkzw9\" (UID: \"3e77a579-9ee6-4304-85b6-96d292d0bb1d\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-tkzw9" Nov 02 06:36:47.092548 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:47.092328 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e77a579-9ee6-4304-85b6-96d292d0bb1d-istio-data\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-tkzw9\" (UID: \"3e77a579-9ee6-4304-85b6-96d292d0bb1d\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-tkzw9" Nov 02 06:36:47.092548 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:47.092430 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e77a579-9ee6-4304-85b6-96d292d0bb1d-workload-certs\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-tkzw9\" (UID: \"3e77a579-9ee6-4304-85b6-96d292d0bb1d\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-tkzw9" Nov 02 06:36:47.092750 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:47.092725 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e77a579-9ee6-4304-85b6-96d292d0bb1d-credential-socket\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-tkzw9\" (UID: \"3e77a579-9ee6-4304-85b6-96d292d0bb1d\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-tkzw9" Nov 02 06:36:47.093050 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:47.093030 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e77a579-9ee6-4304-85b6-96d292d0bb1d-istiod-ca-cert\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-tkzw9\" (UID: \"3e77a579-9ee6-4304-85b6-96d292d0bb1d\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-tkzw9" Nov 02 06:36:47.093140 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:47.093029 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e77a579-9ee6-4304-85b6-96d292d0bb1d-workload-socket\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-tkzw9\" (UID: \"3e77a579-9ee6-4304-85b6-96d292d0bb1d\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-tkzw9" Nov 02 06:36:47.094636 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:47.094608 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e77a579-9ee6-4304-85b6-96d292d0bb1d-istio-podinfo\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-tkzw9\" (UID: \"3e77a579-9ee6-4304-85b6-96d292d0bb1d\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-tkzw9" Nov 02 06:36:47.095092 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:47.095071 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e77a579-9ee6-4304-85b6-96d292d0bb1d-istio-envoy\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-tkzw9\" (UID: \"3e77a579-9ee6-4304-85b6-96d292d0bb1d\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-tkzw9" Nov 02 06:36:47.147287 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:47.147257 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e77a579-9ee6-4304-85b6-96d292d0bb1d-istio-token\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-tkzw9\" (UID: \"3e77a579-9ee6-4304-85b6-96d292d0bb1d\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-tkzw9" Nov 02 06:36:47.147411 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:47.147379 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdjqh\" (UniqueName: \"kubernetes.io/projected/3e77a579-9ee6-4304-85b6-96d292d0bb1d-kube-api-access-vdjqh\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-tkzw9\" (UID: \"3e77a579-9ee6-4304-85b6-96d292d0bb1d\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-tkzw9" Nov 02 06:36:47.213016 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:47.212987 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-tkzw9" Nov 02 06:36:47.288951 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:47.284055 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-9mj97" event={"ID":"9ee541ef-78e0-4075-9780-eb53b607ddcc","Type":"ContainerStarted","Data":"f1fee98fec7a45b64284962d0f5ea124ca899e15fc4bb4e65fb4ab30aca40d76"} Nov 02 06:36:47.288951 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:47.284125 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-9mj97" event={"ID":"9ee541ef-78e0-4075-9780-eb53b607ddcc","Type":"ContainerStarted","Data":"7f9381e29b4e774b1697710c4268a04db70e6a6bdf68145e24f5e2a3026ef571"} Nov 02 06:36:47.288951 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:47.287235 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9f9a772-sgd5d" event={"ID":"253c37b4-7bd4-4310-b8be-b37c64f6b57a","Type":"ContainerDied","Data":"5afecd858ff30f56c950a2d85306620a332a5d8e548f18d4b6b527334eda2dc4"} Nov 02 06:36:47.288951 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:47.287287 5916 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5afecd858ff30f56c950a2d85306620a332a5d8e548f18d4b6b527334eda2dc4" Nov 02 06:36:47.288951 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:47.287250 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9f9a772-sgd5d" Nov 02 06:36:47.319579 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:47.319546 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-wzvj7"] Nov 02 06:36:47.324839 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:36:47.324804 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbeb17737_f6a0_47e1_8524_e56e8d267502.slice/crio-6bc35fe3f41b0ce11a082cec77ef70832cf96aebf162526c3b092ad49e2805dc WatchSource:0}: Error finding container 6bc35fe3f41b0ce11a082cec77ef70832cf96aebf162526c3b092ad49e2805dc: Status 404 returned error can't find the container with id 6bc35fe3f41b0ce11a082cec77ef70832cf96aebf162526c3b092ad49e2805dc Nov 02 06:36:47.327897 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:47.327858 5916 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 06:36:47.328023 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:47.327979 5916 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 06:36:47.328097 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:47.328024 5916 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 06:36:47.334506 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:47.332942 5916 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-9mj97" Nov 02 06:36:47.335928 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:47.335632 5916 patch_prober.go:29] interesting pod/e2e-worker-w2-581608-openshift-default-cd7c896df-9mj97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Nov 02 06:36:47.335928 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:47.335700 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-9mj97" podUID="9ee541ef-78e0-4075-9780-eb53b607ddcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Nov 02 06:36:47.337949 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:36:47.337772 5916 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod253c37b4_7bd4_4310_b8be_b37c64f6b57a.slice\": RecentStats: unable to find data in memory cache]" Nov 02 06:36:47.401999 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:47.400382 5916 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-9mj97" podStartSLOduration=2.400368452 podStartE2EDuration="2.400368452s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:36:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:36:47.398979452 +0000 UTC m=+1195.670799845" watchObservedRunningTime="2026-11-02 06:36:47.400368452 +0000 UTC m=+1195.672188845" Nov 02 06:36:47.603237 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:47.603202 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-tkzw9"] Nov 02 06:36:47.607843 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:36:47.607816 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e77a579_9ee6_4304_85b6_96d292d0bb1d.slice/crio-e4a76ecaf9cffe1e14227deea5946d6c53d8780390b471fe754eb0c255e429ce WatchSource:0}: Error finding container e4a76ecaf9cffe1e14227deea5946d6c53d8780390b471fe754eb0c255e429ce: Status 404 returned error can't find the container with id e4a76ecaf9cffe1e14227deea5946d6c53d8780390b471fe754eb0c255e429ce Nov 02 06:36:47.610599 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:47.610563 5916 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 06:36:47.610700 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:47.610650 5916 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 06:36:47.610743 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:47.610702 5916 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 06:36:48.308494 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:48.308447 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-tkzw9" event={"ID":"3e77a579-9ee6-4304-85b6-96d292d0bb1d","Type":"ContainerStarted","Data":"aacb00e236e7ca4a84a68d6dca69058ddd11f3caa6c711d9ba0a23dbb2627d8f"} Nov 02 06:36:48.308679 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:48.308515 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-tkzw9" event={"ID":"3e77a579-9ee6-4304-85b6-96d292d0bb1d","Type":"ContainerStarted","Data":"e4a76ecaf9cffe1e14227deea5946d6c53d8780390b471fe754eb0c255e429ce"} Nov 02 06:36:48.311556 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:48.311529 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-wzvj7" event={"ID":"beb17737-f6a0-47e1-8524-e56e8d267502","Type":"ContainerStarted","Data":"91af1497b1fbf0b6bfd3480afccc099dfba64d005ed943094029093396c994fa"} Nov 02 06:36:48.311691 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:48.311584 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-wzvj7" event={"ID":"beb17737-f6a0-47e1-8524-e56e8d267502","Type":"ContainerStarted","Data":"6bc35fe3f41b0ce11a082cec77ef70832cf96aebf162526c3b092ad49e2805dc"} Nov 02 06:36:48.345944 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:48.338787 5916 patch_prober.go:29] interesting pod/e2e-worker-w2-581608-openshift-default-cd7c896df-9mj97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Nov 02 06:36:48.345944 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:48.338846 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-9mj97" podUID="9ee541ef-78e0-4075-9780-eb53b607ddcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Nov 02 06:36:48.493931 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:48.483966 5916 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-tkzw9" podStartSLOduration=2.46467905 podStartE2EDuration="2.46467905s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:36:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:36:48.446540486 +0000 UTC m=+1196.718360880" watchObservedRunningTime="2026-11-02 06:36:48.46467905 +0000 UTC m=+1196.736499446" Nov 02 06:36:48.654939 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:48.652069 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8f6d2ba-bs5d7"] Nov 02 06:36:48.763939 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:48.763701 5916 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-wzvj7" podStartSLOduration=2.763677232 podStartE2EDuration="2.763677232s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:36:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:36:48.566767782 +0000 UTC m=+1196.838588187" watchObservedRunningTime="2026-11-02 06:36:48.763677232 +0000 UTC m=+1197.035497627" Nov 02 06:36:48.876275 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:48.876234 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8f6d2ba-bs5d7"] Nov 02 06:36:48.876613 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:48.876593 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8f6d2ba-bs5d7" Nov 02 06:36:48.937990 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:48.932188 5916 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-wzvj7" Nov 02 06:36:48.937990 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:48.932319 5916 patch_prober.go:29] interesting pod/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-wzvj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Nov 02 06:36:48.937990 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:48.932373 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-wzvj7" podUID="beb17737-f6a0-47e1-8524-e56e8d267502" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Nov 02 06:36:48.937990 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:48.932869 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-68l4r\"" Nov 02 06:36:48.937990 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:48.933017 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 06:36:48.972937 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:48.968689 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88ea4c51-e049-495a-b69c-306b6b665085-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c8f6d2ba-bs5d7\" (UID: \"88ea4c51-e049-495a-b69c-306b6b665085\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8f6d2ba-bs5d7" Nov 02 06:36:48.973399 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:48.973373 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88ea4c51-e049-495a-b69c-306b6b665085-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c8f6d2ba-bs5d7\" (UID: \"88ea4c51-e049-495a-b69c-306b6b665085\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8f6d2ba-bs5d7" Nov 02 06:36:48.973571 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:48.973557 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpch6\" (UniqueName: \"kubernetes.io/projected/88ea4c51-e049-495a-b69c-306b6b665085-kube-api-access-lpch6\") pod \"maas-api-revoke-sub-c8f6d2ba-bs5d7\" (UID: \"88ea4c51-e049-495a-b69c-306b6b665085\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8f6d2ba-bs5d7" Nov 02 06:36:49.076636 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:49.076601 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88ea4c51-e049-495a-b69c-306b6b665085-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c8f6d2ba-bs5d7\" (UID: \"88ea4c51-e049-495a-b69c-306b6b665085\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8f6d2ba-bs5d7" Nov 02 06:36:49.076971 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:49.076954 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88ea4c51-e049-495a-b69c-306b6b665085-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c8f6d2ba-bs5d7\" (UID: \"88ea4c51-e049-495a-b69c-306b6b665085\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8f6d2ba-bs5d7" Nov 02 06:36:49.077116 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:49.077103 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpch6\" (UniqueName: \"kubernetes.io/projected/88ea4c51-e049-495a-b69c-306b6b665085-kube-api-access-lpch6\") pod \"maas-api-revoke-sub-c8f6d2ba-bs5d7\" (UID: \"88ea4c51-e049-495a-b69c-306b6b665085\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8f6d2ba-bs5d7" Nov 02 06:36:49.085938 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:49.078361 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88ea4c51-e049-495a-b69c-306b6b665085-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c8f6d2ba-bs5d7\" (UID: \"88ea4c51-e049-495a-b69c-306b6b665085\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8f6d2ba-bs5d7" Nov 02 06:36:49.119464 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:49.119428 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88ea4c51-e049-495a-b69c-306b6b665085-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c8f6d2ba-bs5d7\" (UID: \"88ea4c51-e049-495a-b69c-306b6b665085\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8f6d2ba-bs5d7" Nov 02 06:36:49.144938 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:49.140841 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpch6\" (UniqueName: \"kubernetes.io/projected/88ea4c51-e049-495a-b69c-306b6b665085-kube-api-access-lpch6\") pod \"maas-api-revoke-sub-c8f6d2ba-bs5d7\" (UID: \"88ea4c51-e049-495a-b69c-306b6b665085\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8f6d2ba-bs5d7" Nov 02 06:36:49.223986 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:49.222932 5916 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-tkzw9" Nov 02 06:36:49.223986 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:49.223071 5916 patch_prober.go:29] interesting pod/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-tkzw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Nov 02 06:36:49.223986 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:49.223132 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-tkzw9" podUID="3e77a579-9ee6-4304-85b6-96d292d0bb1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Nov 02 06:36:49.283932 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:49.264238 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8f6d2ba-bs5d7" Nov 02 06:36:49.347284 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:49.345381 5916 patch_prober.go:29] interesting pod/e2e-worker-w2-581608-openshift-default-cd7c896df-9mj97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Nov 02 06:36:49.347284 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:49.345440 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-9mj97" podUID="9ee541ef-78e0-4075-9780-eb53b607ddcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Nov 02 06:36:49.923991 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:49.923946 5916 patch_prober.go:29] interesting pod/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-wzvj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Nov 02 06:36:49.924307 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:49.924281 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-wzvj7" podUID="beb17737-f6a0-47e1-8524-e56e8d267502" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Nov 02 06:36:50.125935 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:36:50.119358 5916 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p98cq" containerName="extractor" Nov 02 06:36:50.126261 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:50.123783 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8f6d2ba-bs5d7"] Nov 02 06:36:50.182008 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:36:50.180249 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88ea4c51_e049_495a_b69c_306b6b665085.slice/crio-3c3db728bc9bb5f0a9913f9d79df3142d35865526e54226cea26b824c4505d3d WatchSource:0}: Error finding container 3c3db728bc9bb5f0a9913f9d79df3142d35865526e54226cea26b824c4505d3d: Status 404 returned error can't find the container with id 3c3db728bc9bb5f0a9913f9d79df3142d35865526e54226cea26b824c4505d3d Nov 02 06:36:50.224184 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:50.224131 5916 patch_prober.go:29] interesting pod/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-tkzw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Nov 02 06:36:50.224365 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:50.224215 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-tkzw9" podUID="3e77a579-9ee6-4304-85b6-96d292d0bb1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Nov 02 06:36:50.357093 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:50.357048 5916 patch_prober.go:29] interesting pod/e2e-worker-w2-581608-openshift-default-cd7c896df-9mj97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Nov 02 06:36:50.365093 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:50.365051 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-9mj97" podUID="9ee541ef-78e0-4075-9780-eb53b607ddcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Nov 02 06:36:50.424407 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:50.424379 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8f6d2ba-bs5d7" event={"ID":"88ea4c51-e049-495a-b69c-306b6b665085","Type":"ContainerStarted","Data":"3c3db728bc9bb5f0a9913f9d79df3142d35865526e54226cea26b824c4505d3d"} Nov 02 06:36:50.918992 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:50.918958 5916 patch_prober.go:29] interesting pod/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-wzvj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Nov 02 06:36:50.919171 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:50.919019 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-wzvj7" podUID="beb17737-f6a0-47e1-8524-e56e8d267502" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Nov 02 06:36:51.215604 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:51.214090 5916 patch_prober.go:29] interesting pod/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-tkzw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Nov 02 06:36:51.215604 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:51.214142 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-tkzw9" podUID="3e77a579-9ee6-4304-85b6-96d292d0bb1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Nov 02 06:36:51.333468 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:51.333429 5916 patch_prober.go:29] interesting pod/e2e-worker-w2-581608-openshift-default-cd7c896df-9mj97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Nov 02 06:36:51.333743 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:51.333711 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-9mj97" podUID="9ee541ef-78e0-4075-9780-eb53b607ddcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Nov 02 06:36:51.516894 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:51.516797 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8f6d2ba-bs5d7" event={"ID":"88ea4c51-e049-495a-b69c-306b6b665085","Type":"ContainerStarted","Data":"48adf154072ce3abd5c2604be874752bcae88be693172a8d2528b6c30ff594d1"} Nov 02 06:36:51.637937 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:51.633696 5916 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8f6d2ba-bs5d7" podStartSLOduration=3.633675791 podStartE2EDuration="3.633675791s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:36:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:36:51.633294817 +0000 UTC m=+1199.905115222" watchObservedRunningTime="2026-11-02 06:36:51.633675791 +0000 UTC m=+1199.905496188" Nov 02 06:36:51.920995 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:51.920451 5916 patch_prober.go:29] interesting pod/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-wzvj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Nov 02 06:36:51.920995 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:51.920499 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-wzvj7" podUID="beb17737-f6a0-47e1-8524-e56e8d267502" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Nov 02 06:36:52.152943 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:52.091524 5916 scope.go:122] "RemoveContainer" containerID="f38cd233fe0b65e996dd98e61656367d601fcf6bb59af4497c1451744f2ba32f" Nov 02 06:36:52.241102 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:52.219224 5916 patch_prober.go:29] interesting pod/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-tkzw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Nov 02 06:36:52.241102 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:52.219281 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-tkzw9" podUID="3e77a579-9ee6-4304-85b6-96d292d0bb1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Nov 02 06:36:52.407939 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:52.390092 5916 patch_prober.go:29] interesting pod/e2e-worker-w2-581608-openshift-default-cd7c896df-9mj97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Nov 02 06:36:52.407939 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:52.390151 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-9mj97" podUID="9ee541ef-78e0-4075-9780-eb53b607ddcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Nov 02 06:36:52.919220 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:52.919053 5916 patch_prober.go:29] interesting pod/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-wzvj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Nov 02 06:36:52.919220 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:52.919112 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-wzvj7" podUID="beb17737-f6a0-47e1-8524-e56e8d267502" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Nov 02 06:36:52.921642 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:52.921372 5916 generic.go:356] "Generic (PLEG): container finished" podID="88ea4c51-e049-495a-b69c-306b6b665085" containerID="48adf154072ce3abd5c2604be874752bcae88be693172a8d2528b6c30ff594d1" exitCode=0 Nov 02 06:36:52.921642 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:52.921410 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8f6d2ba-bs5d7" event={"ID":"88ea4c51-e049-495a-b69c-306b6b665085","Type":"ContainerDied","Data":"48adf154072ce3abd5c2604be874752bcae88be693172a8d2528b6c30ff594d1"} Nov 02 06:36:52.985875 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:36:52.985838 5916 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f38cd233fe0b65e996dd98e61656367d601fcf6bb59af4497c1451744f2ba32f\": container with ID starting with f38cd233fe0b65e996dd98e61656367d601fcf6bb59af4497c1451744f2ba32f not found: ID does not exist" containerID="f38cd233fe0b65e996dd98e61656367d601fcf6bb59af4497c1451744f2ba32f" Nov 02 06:36:52.986146 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:52.986124 5916 image_gc_manager.go:223] "Failed to monitor images" err="get container status: runtime container status: rpc error: code = NotFound desc = could not find container \"f38cd233fe0b65e996dd98e61656367d601fcf6bb59af4497c1451744f2ba32f\": container with ID starting with f38cd233fe0b65e996dd98e61656367d601fcf6bb59af4497c1451744f2ba32f not found: ID does not exist" Nov 02 06:36:52.994612 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:36:52.994584 5916 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="get container status: runtime container status: rpc error: code = NotFound desc = could not find container \"f38cd233fe0b65e996dd98e61656367d601fcf6bb59af4497c1451744f2ba32f\": container with ID starting with f38cd233fe0b65e996dd98e61656367d601fcf6bb59af4497c1451744f2ba32f not found: ID does not exist" Nov 02 06:36:53.213538 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:53.213455 5916 patch_prober.go:29] interesting pod/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-tkzw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Nov 02 06:36:53.213538 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:53.213515 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-tkzw9" podUID="3e77a579-9ee6-4304-85b6-96d292d0bb1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Nov 02 06:36:53.346940 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:53.343284 5916 patch_prober.go:29] interesting pod/e2e-worker-w2-581608-openshift-default-cd7c896df-9mj97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Nov 02 06:36:53.346940 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:53.343345 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-9mj97" podUID="9ee541ef-78e0-4075-9780-eb53b607ddcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Nov 02 06:36:53.917925 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:53.917862 5916 patch_prober.go:29] interesting pod/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-wzvj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Nov 02 06:36:53.918144 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:53.917967 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-wzvj7" podUID="beb17737-f6a0-47e1-8524-e56e8d267502" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Nov 02 06:36:54.214079 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:54.214006 5916 patch_prober.go:29] interesting pod/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-tkzw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Nov 02 06:36:54.214079 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:54.214068 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-tkzw9" podUID="3e77a579-9ee6-4304-85b6-96d292d0bb1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Nov 02 06:36:54.225940 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:54.225895 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8f6d2ba-bs5d7" Nov 02 06:36:54.385075 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:54.384719 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88ea4c51-e049-495a-b69c-306b6b665085-kube-api-access-lpch6\" (UniqueName: \"kubernetes.io/projected/88ea4c51-e049-495a-b69c-306b6b665085-kube-api-access-lpch6\") pod \"88ea4c51-e049-495a-b69c-306b6b665085\" (UID: \"88ea4c51-e049-495a-b69c-306b6b665085\") " Nov 02 06:36:54.385075 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:54.384807 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/88ea4c51-e049-495a-b69c-306b6b665085-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88ea4c51-e049-495a-b69c-306b6b665085-maas-api-xks-ca\") pod \"88ea4c51-e049-495a-b69c-306b6b665085\" (UID: \"88ea4c51-e049-495a-b69c-306b6b665085\") " Nov 02 06:36:54.385075 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:54.385057 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88ea4c51-e049-495a-b69c-306b6b665085-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88ea4c51-e049-495a-b69c-306b6b665085-maas-api-openshift-ca\") pod \"88ea4c51-e049-495a-b69c-306b6b665085\" (UID: \"88ea4c51-e049-495a-b69c-306b6b665085\") " Nov 02 06:36:54.392256 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:54.392193 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88ea4c51-e049-495a-b69c-306b6b665085-maas-api-openshift-ca" pod "88ea4c51-e049-495a-b69c-306b6b665085" (UID: "88ea4c51-e049-495a-b69c-306b6b665085"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:36:54.407951 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:54.397855 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88ea4c51-e049-495a-b69c-306b6b665085-maas-api-openshift-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:36:54.407951 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:54.398830 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8bf36a9a-jgxp9"] Nov 02 06:36:54.407951 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:54.399692 5916 cpu_manager.go:414] "containerMap: removing container" podUID="88ea4c51-e049-495a-b69c-306b6b665085" containerName="revoke-keys" Nov 02 06:36:54.407951 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:54.399711 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88ea4c51-e049-495a-b69c-306b6b665085" containerName="revoke-keys" Nov 02 06:36:54.407951 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:54.399884 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="88ea4c51-e049-495a-b69c-306b6b665085" containerName="revoke-keys" Nov 02 06:36:54.407951 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:54.404196 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bf36a9a-jgxp9" Nov 02 06:36:54.426395 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:54.425871 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88ea4c51-e049-495a-b69c-306b6b665085-kube-api-access-lpch6" pod "88ea4c51-e049-495a-b69c-306b6b665085" (UID: "88ea4c51-e049-495a-b69c-306b6b665085"). InnerVolumeSpecName "kube-api-access-lpch6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:36:54.428518 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:54.428473 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88ea4c51-e049-495a-b69c-306b6b665085-maas-api-xks-ca" pod "88ea4c51-e049-495a-b69c-306b6b665085" (UID: "88ea4c51-e049-495a-b69c-306b6b665085"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:36:54.456352 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:54.456269 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8bf36a9a-jgxp9"] Nov 02 06:36:54.500395 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:54.500324 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lpch6\" (UniqueName: \"kubernetes.io/projected/88ea4c51-e049-495a-b69c-306b6b665085-kube-api-access-lpch6\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:36:54.500395 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:54.500364 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88ea4c51-e049-495a-b69c-306b6b665085-maas-api-xks-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:36:54.601815 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:54.601760 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5zhm\" (UniqueName: \"kubernetes.io/projected/3e7cefed-95d9-40d1-9c49-60394d0b4c84-kube-api-access-t5zhm\") pod \"maas-api-revoke-sub-8bf36a9a-jgxp9\" (UID: \"3e7cefed-95d9-40d1-9c49-60394d0b4c84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bf36a9a-jgxp9" Nov 02 06:36:54.602019 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:54.601856 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e7cefed-95d9-40d1-9c49-60394d0b4c84-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8bf36a9a-jgxp9\" (UID: \"3e7cefed-95d9-40d1-9c49-60394d0b4c84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bf36a9a-jgxp9" Nov 02 06:36:54.602121 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:54.602092 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e7cefed-95d9-40d1-9c49-60394d0b4c84-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8bf36a9a-jgxp9\" (UID: \"3e7cefed-95d9-40d1-9c49-60394d0b4c84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bf36a9a-jgxp9" Nov 02 06:36:54.702814 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:54.702780 5916 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-9mj97" Nov 02 06:36:54.703728 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:54.703706 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-9mj97" Nov 02 06:36:54.704244 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:54.704218 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e7cefed-95d9-40d1-9c49-60394d0b4c84-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8bf36a9a-jgxp9\" (UID: \"3e7cefed-95d9-40d1-9c49-60394d0b4c84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bf36a9a-jgxp9" Nov 02 06:36:54.704432 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:54.704407 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e7cefed-95d9-40d1-9c49-60394d0b4c84-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8bf36a9a-jgxp9\" (UID: \"3e7cefed-95d9-40d1-9c49-60394d0b4c84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bf36a9a-jgxp9" Nov 02 06:36:54.704680 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:54.704660 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-9mj97" Nov 02 06:36:54.705608 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:54.705586 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e7cefed-95d9-40d1-9c49-60394d0b4c84-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8bf36a9a-jgxp9\" (UID: \"3e7cefed-95d9-40d1-9c49-60394d0b4c84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bf36a9a-jgxp9" Nov 02 06:36:54.705751 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:54.705730 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5zhm\" (UniqueName: \"kubernetes.io/projected/3e7cefed-95d9-40d1-9c49-60394d0b4c84-kube-api-access-t5zhm\") pod \"maas-api-revoke-sub-8bf36a9a-jgxp9\" (UID: \"3e7cefed-95d9-40d1-9c49-60394d0b4c84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bf36a9a-jgxp9" Nov 02 06:36:54.707655 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:54.707577 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e7cefed-95d9-40d1-9c49-60394d0b4c84-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8bf36a9a-jgxp9\" (UID: \"3e7cefed-95d9-40d1-9c49-60394d0b4c84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bf36a9a-jgxp9" Nov 02 06:36:54.759256 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:54.759224 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5zhm\" (UniqueName: \"kubernetes.io/projected/3e7cefed-95d9-40d1-9c49-60394d0b4c84-kube-api-access-t5zhm\") pod \"maas-api-revoke-sub-8bf36a9a-jgxp9\" (UID: \"3e7cefed-95d9-40d1-9c49-60394d0b4c84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bf36a9a-jgxp9" Nov 02 06:36:54.767877 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:54.767845 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bf36a9a-jgxp9" Nov 02 06:36:54.954935 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:54.954265 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8f6d2ba-bs5d7" Nov 02 06:36:54.957956 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:54.957858 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8f6d2ba-bs5d7" event={"ID":"88ea4c51-e049-495a-b69c-306b6b665085","Type":"ContainerDied","Data":"3c3db728bc9bb5f0a9913f9d79df3142d35865526e54226cea26b824c4505d3d"} Nov 02 06:36:54.958141 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:54.957981 5916 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3c3db728bc9bb5f0a9913f9d79df3142d35865526e54226cea26b824c4505d3d" Nov 02 06:36:55.169888 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:55.169843 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8bf36a9a-jgxp9"] Nov 02 06:36:55.213416 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:55.213340 5916 patch_prober.go:29] interesting pod/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-tkzw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Nov 02 06:36:55.213568 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:55.213404 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-tkzw9" podUID="3e77a579-9ee6-4304-85b6-96d292d0bb1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Nov 02 06:36:55.380939 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:55.380849 5916 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-wzvj7" Nov 02 06:36:55.382361 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:55.382335 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-wzvj7" Nov 02 06:36:55.383108 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:55.383087 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-wzvj7" Nov 02 06:36:55.963594 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:55.963554 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bf36a9a-jgxp9" event={"ID":"3e7cefed-95d9-40d1-9c49-60394d0b4c84","Type":"ContainerStarted","Data":"01c9211a8423c4ccdb6f02ad23124fc0d3f2b6f959cf6e4034055ee436191301"} Nov 02 06:36:55.963769 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:55.963611 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bf36a9a-jgxp9" event={"ID":"3e7cefed-95d9-40d1-9c49-60394d0b4c84","Type":"ContainerStarted","Data":"01dd7d3c8969b0cafe187d6df74d2db2aac2b645089bfd7b88a6cdac07ce4f0c"} Nov 02 06:36:56.073251 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:56.073192 5916 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bf36a9a-jgxp9" podStartSLOduration=2.073176847 podStartE2EDuration="2.073176847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:36:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:36:56.069962435 +0000 UTC m=+1204.341782828" watchObservedRunningTime="2026-11-02 06:36:56.073176847 +0000 UTC m=+1204.344997241" Nov 02 06:36:56.218400 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:56.218314 5916 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-tkzw9" Nov 02 06:36:56.218670 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:56.218648 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-tkzw9" Nov 02 06:36:56.219295 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:56.219275 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-tkzw9" Nov 02 06:36:56.973320 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:56.973278 5916 generic.go:356] "Generic (PLEG): container finished" podID="3e7cefed-95d9-40d1-9c49-60394d0b4c84" containerID="01c9211a8423c4ccdb6f02ad23124fc0d3f2b6f959cf6e4034055ee436191301" exitCode=0 Nov 02 06:36:56.973320 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:56.973311 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bf36a9a-jgxp9" event={"ID":"3e7cefed-95d9-40d1-9c49-60394d0b4c84","Type":"ContainerDied","Data":"01c9211a8423c4ccdb6f02ad23124fc0d3f2b6f959cf6e4034055ee436191301"} Nov 02 06:36:58.140495 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:58.140468 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bf36a9a-jgxp9" Nov 02 06:36:58.162436 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:58.162390 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3e7cefed-95d9-40d1-9c49-60394d0b4c84-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e7cefed-95d9-40d1-9c49-60394d0b4c84-maas-api-xks-ca\") pod \"3e7cefed-95d9-40d1-9c49-60394d0b4c84\" (UID: \"3e7cefed-95d9-40d1-9c49-60394d0b4c84\") " Nov 02 06:36:58.162649 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:58.162468 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e7cefed-95d9-40d1-9c49-60394d0b4c84-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e7cefed-95d9-40d1-9c49-60394d0b4c84-maas-api-openshift-ca\") pod \"3e7cefed-95d9-40d1-9c49-60394d0b4c84\" (UID: \"3e7cefed-95d9-40d1-9c49-60394d0b4c84\") " Nov 02 06:36:58.162649 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:58.162553 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e7cefed-95d9-40d1-9c49-60394d0b4c84-kube-api-access-t5zhm\" (UniqueName: \"kubernetes.io/projected/3e7cefed-95d9-40d1-9c49-60394d0b4c84-kube-api-access-t5zhm\") pod \"3e7cefed-95d9-40d1-9c49-60394d0b4c84\" (UID: \"3e7cefed-95d9-40d1-9c49-60394d0b4c84\") " Nov 02 06:36:58.162860 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:58.162832 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e7cefed-95d9-40d1-9c49-60394d0b4c84-maas-api-openshift-ca" pod "3e7cefed-95d9-40d1-9c49-60394d0b4c84" (UID: "3e7cefed-95d9-40d1-9c49-60394d0b4c84"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:36:58.164626 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:58.164602 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e7cefed-95d9-40d1-9c49-60394d0b4c84-maas-api-xks-ca" pod "3e7cefed-95d9-40d1-9c49-60394d0b4c84" (UID: "3e7cefed-95d9-40d1-9c49-60394d0b4c84"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:36:58.164712 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:58.164676 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e7cefed-95d9-40d1-9c49-60394d0b4c84-kube-api-access-t5zhm" pod "3e7cefed-95d9-40d1-9c49-60394d0b4c84" (UID: "3e7cefed-95d9-40d1-9c49-60394d0b4c84"). InnerVolumeSpecName "kube-api-access-t5zhm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:36:58.263602 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:58.263532 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e7cefed-95d9-40d1-9c49-60394d0b4c84-maas-api-xks-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:36:58.263602 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:58.263558 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e7cefed-95d9-40d1-9c49-60394d0b4c84-maas-api-openshift-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:36:58.263602 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:58.263568 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5zhm\" (UniqueName: \"kubernetes.io/projected/3e7cefed-95d9-40d1-9c49-60394d0b4c84-kube-api-access-t5zhm\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:36:58.996165 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:58.996131 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bf36a9a-jgxp9" Nov 02 06:36:58.996348 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:58.996162 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bf36a9a-jgxp9" event={"ID":"3e7cefed-95d9-40d1-9c49-60394d0b4c84","Type":"ContainerDied","Data":"01dd7d3c8969b0cafe187d6df74d2db2aac2b645089bfd7b88a6cdac07ce4f0c"} Nov 02 06:36:58.996348 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:58.996219 5916 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="01dd7d3c8969b0cafe187d6df74d2db2aac2b645089bfd7b88a6cdac07ce4f0c" Nov 02 06:36:59.608020 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:59.607981 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f42bb79-ddrpm"] Nov 02 06:36:59.608943 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:59.608888 5916 cpu_manager.go:414] "containerMap: removing container" podUID="3e7cefed-95d9-40d1-9c49-60394d0b4c84" containerName="revoke-keys" Nov 02 06:36:59.608943 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:59.608922 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e7cefed-95d9-40d1-9c49-60394d0b4c84" containerName="revoke-keys" Nov 02 06:36:59.609091 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:59.609078 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e7cefed-95d9-40d1-9c49-60394d0b4c84" containerName="revoke-keys" Nov 02 06:36:59.613490 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:59.613469 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f42bb79-ddrpm" Nov 02 06:36:59.678496 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:59.678460 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f42bb79-ddrpm"] Nov 02 06:36:59.683984 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:59.683163 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c80d239a-c8b0-4869-80e7-6e6f491f812e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6f42bb79-ddrpm\" (UID: \"c80d239a-c8b0-4869-80e7-6e6f491f812e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f42bb79-ddrpm" Nov 02 06:36:59.683984 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:59.683254 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlccs\" (UniqueName: \"kubernetes.io/projected/c80d239a-c8b0-4869-80e7-6e6f491f812e-kube-api-access-rlccs\") pod \"maas-api-revoke-sub-6f42bb79-ddrpm\" (UID: \"c80d239a-c8b0-4869-80e7-6e6f491f812e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f42bb79-ddrpm" Nov 02 06:36:59.683984 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:59.683452 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c80d239a-c8b0-4869-80e7-6e6f491f812e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6f42bb79-ddrpm\" (UID: \"c80d239a-c8b0-4869-80e7-6e6f491f812e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f42bb79-ddrpm" Nov 02 06:36:59.703529 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:59.703499 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 06:36:59.703697 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:59.703548 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-68l4r\"" Nov 02 06:36:59.788464 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:59.788422 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c80d239a-c8b0-4869-80e7-6e6f491f812e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6f42bb79-ddrpm\" (UID: \"c80d239a-c8b0-4869-80e7-6e6f491f812e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f42bb79-ddrpm" Nov 02 06:36:59.788683 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:59.788512 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlccs\" (UniqueName: \"kubernetes.io/projected/c80d239a-c8b0-4869-80e7-6e6f491f812e-kube-api-access-rlccs\") pod \"maas-api-revoke-sub-6f42bb79-ddrpm\" (UID: \"c80d239a-c8b0-4869-80e7-6e6f491f812e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f42bb79-ddrpm" Nov 02 06:36:59.788683 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:59.788609 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c80d239a-c8b0-4869-80e7-6e6f491f812e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6f42bb79-ddrpm\" (UID: \"c80d239a-c8b0-4869-80e7-6e6f491f812e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f42bb79-ddrpm" Nov 02 06:36:59.789419 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:59.789390 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c80d239a-c8b0-4869-80e7-6e6f491f812e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6f42bb79-ddrpm\" (UID: \"c80d239a-c8b0-4869-80e7-6e6f491f812e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f42bb79-ddrpm" Nov 02 06:36:59.791088 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:59.791061 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c80d239a-c8b0-4869-80e7-6e6f491f812e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6f42bb79-ddrpm\" (UID: \"c80d239a-c8b0-4869-80e7-6e6f491f812e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f42bb79-ddrpm" Nov 02 06:36:59.852402 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:59.846684 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlccs\" (UniqueName: \"kubernetes.io/projected/c80d239a-c8b0-4869-80e7-6e6f491f812e-kube-api-access-rlccs\") pod \"maas-api-revoke-sub-6f42bb79-ddrpm\" (UID: \"c80d239a-c8b0-4869-80e7-6e6f491f812e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f42bb79-ddrpm" Nov 02 06:36:59.935474 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:36:59.935413 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f42bb79-ddrpm" Nov 02 06:37:00.348812 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:00.347832 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f42bb79-ddrpm"] Nov 02 06:37:00.356940 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:37:00.352445 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc80d239a_c8b0_4869_80e7_6e6f491f812e.slice/crio-9a867efdde6f9d1e79ce63571ff9fc6dc4d4d33b29f1327b17126612cdee9822 WatchSource:0}: Error finding container 9a867efdde6f9d1e79ce63571ff9fc6dc4d4d33b29f1327b17126612cdee9822: Status 404 returned error can't find the container with id 9a867efdde6f9d1e79ce63571ff9fc6dc4d4d33b29f1327b17126612cdee9822 Nov 02 06:37:01.019265 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:01.019177 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f42bb79-ddrpm" event={"ID":"c80d239a-c8b0-4869-80e7-6e6f491f812e","Type":"ContainerStarted","Data":"99ac39e5536f07fb0b8085bd565fb219aedb0aa6b9be09a387fa736e0a56b852"} Nov 02 06:37:01.019265 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:01.019234 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f42bb79-ddrpm" event={"ID":"c80d239a-c8b0-4869-80e7-6e6f491f812e","Type":"ContainerStarted","Data":"9a867efdde6f9d1e79ce63571ff9fc6dc4d4d33b29f1327b17126612cdee9822"} Nov 02 06:37:01.130523 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:01.130456 5916 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f42bb79-ddrpm" podStartSLOduration=2.13043816 podStartE2EDuration="2.13043816s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:36:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:37:01.124341825 +0000 UTC m=+1209.396162232" watchObservedRunningTime="2026-11-02 06:37:01.13043816 +0000 UTC m=+1209.402258556" Nov 02 06:37:01.799001 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:01.798971 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06e94cdb-5sj4f"] Nov 02 06:37:01.803272 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:01.803252 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06e94cdb-5sj4f" Nov 02 06:37:01.857991 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:01.857957 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06e94cdb-5sj4f"] Nov 02 06:37:01.918996 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:01.918962 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a491cd95-8967-4f09-82f9-d154761e79ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06e94cdb-5sj4f\" (UID: \"a491cd95-8967-4f09-82f9-d154761e79ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06e94cdb-5sj4f" Nov 02 06:37:01.919158 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:01.919093 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lq5lq\" (UniqueName: \"kubernetes.io/projected/a491cd95-8967-4f09-82f9-d154761e79ea-kube-api-access-lq5lq\") pod \"maas-api-revoke-sub-06e94cdb-5sj4f\" (UID: \"a491cd95-8967-4f09-82f9-d154761e79ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06e94cdb-5sj4f" Nov 02 06:37:01.919211 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:01.919161 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a491cd95-8967-4f09-82f9-d154761e79ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06e94cdb-5sj4f\" (UID: \"a491cd95-8967-4f09-82f9-d154761e79ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06e94cdb-5sj4f" Nov 02 06:37:02.020555 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:02.020525 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a491cd95-8967-4f09-82f9-d154761e79ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06e94cdb-5sj4f\" (UID: \"a491cd95-8967-4f09-82f9-d154761e79ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06e94cdb-5sj4f" Nov 02 06:37:02.021059 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:02.020607 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lq5lq\" (UniqueName: \"kubernetes.io/projected/a491cd95-8967-4f09-82f9-d154761e79ea-kube-api-access-lq5lq\") pod \"maas-api-revoke-sub-06e94cdb-5sj4f\" (UID: \"a491cd95-8967-4f09-82f9-d154761e79ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06e94cdb-5sj4f" Nov 02 06:37:02.021059 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:02.020667 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a491cd95-8967-4f09-82f9-d154761e79ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06e94cdb-5sj4f\" (UID: \"a491cd95-8967-4f09-82f9-d154761e79ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06e94cdb-5sj4f" Nov 02 06:37:02.021259 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:02.021234 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a491cd95-8967-4f09-82f9-d154761e79ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06e94cdb-5sj4f\" (UID: \"a491cd95-8967-4f09-82f9-d154761e79ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06e94cdb-5sj4f" Nov 02 06:37:02.023003 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:02.022977 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a491cd95-8967-4f09-82f9-d154761e79ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06e94cdb-5sj4f\" (UID: \"a491cd95-8967-4f09-82f9-d154761e79ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06e94cdb-5sj4f" Nov 02 06:37:02.030981 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:02.030955 5916 generic.go:356] "Generic (PLEG): container finished" podID="c80d239a-c8b0-4869-80e7-6e6f491f812e" containerID="99ac39e5536f07fb0b8085bd565fb219aedb0aa6b9be09a387fa736e0a56b852" exitCode=0 Nov 02 06:37:02.031109 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:02.031019 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f42bb79-ddrpm" event={"ID":"c80d239a-c8b0-4869-80e7-6e6f491f812e","Type":"ContainerDied","Data":"99ac39e5536f07fb0b8085bd565fb219aedb0aa6b9be09a387fa736e0a56b852"} Nov 02 06:37:02.074200 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:02.074130 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lq5lq\" (UniqueName: \"kubernetes.io/projected/a491cd95-8967-4f09-82f9-d154761e79ea-kube-api-access-lq5lq\") pod \"maas-api-revoke-sub-06e94cdb-5sj4f\" (UID: \"a491cd95-8967-4f09-82f9-d154761e79ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06e94cdb-5sj4f" Nov 02 06:37:02.117673 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:02.117644 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06e94cdb-5sj4f" Nov 02 06:37:02.507376 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:02.507343 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06e94cdb-5sj4f"] Nov 02 06:37:02.511686 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:37:02.511654 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda491cd95_8967_4f09_82f9_d154761e79ea.slice/crio-426d60d5d7b6233bbe56df91743a8b38a4014ad82d7c04fcc9316fdb69a73059 WatchSource:0}: Error finding container 426d60d5d7b6233bbe56df91743a8b38a4014ad82d7c04fcc9316fdb69a73059: Status 404 returned error can't find the container with id 426d60d5d7b6233bbe56df91743a8b38a4014ad82d7c04fcc9316fdb69a73059 Nov 02 06:37:03.044210 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:03.044175 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06e94cdb-5sj4f" event={"ID":"a491cd95-8967-4f09-82f9-d154761e79ea","Type":"ContainerStarted","Data":"07acc4b8b08df889f973d4cd62a74b6069a3402a875c65052ae14d6763476a4e"} Nov 02 06:37:03.044647 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:03.044238 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06e94cdb-5sj4f" event={"ID":"a491cd95-8967-4f09-82f9-d154761e79ea","Type":"ContainerStarted","Data":"426d60d5d7b6233bbe56df91743a8b38a4014ad82d7c04fcc9316fdb69a73059"} Nov 02 06:37:03.155046 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:03.154758 5916 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06e94cdb-5sj4f" podStartSLOduration=2.154738733 podStartE2EDuration="2.154738733s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:37:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:37:03.1532604 +0000 UTC m=+1211.425080795" watchObservedRunningTime="2026-11-02 06:37:03.154738733 +0000 UTC m=+1211.426559128" Nov 02 06:37:03.214274 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:03.214243 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f42bb79-ddrpm" Nov 02 06:37:03.336980 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:03.336941 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c80d239a-c8b0-4869-80e7-6e6f491f812e-kube-api-access-rlccs\" (UniqueName: \"kubernetes.io/projected/c80d239a-c8b0-4869-80e7-6e6f491f812e-kube-api-access-rlccs\") pod \"c80d239a-c8b0-4869-80e7-6e6f491f812e\" (UID: \"c80d239a-c8b0-4869-80e7-6e6f491f812e\") " Nov 02 06:37:03.337160 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:03.337035 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c80d239a-c8b0-4869-80e7-6e6f491f812e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c80d239a-c8b0-4869-80e7-6e6f491f812e-maas-api-xks-ca\") pod \"c80d239a-c8b0-4869-80e7-6e6f491f812e\" (UID: \"c80d239a-c8b0-4869-80e7-6e6f491f812e\") " Nov 02 06:37:03.337160 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:03.337089 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c80d239a-c8b0-4869-80e7-6e6f491f812e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c80d239a-c8b0-4869-80e7-6e6f491f812e-maas-api-openshift-ca\") pod \"c80d239a-c8b0-4869-80e7-6e6f491f812e\" (UID: \"c80d239a-c8b0-4869-80e7-6e6f491f812e\") " Nov 02 06:37:03.338276 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:03.338244 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c80d239a-c8b0-4869-80e7-6e6f491f812e-maas-api-openshift-ca" pod "c80d239a-c8b0-4869-80e7-6e6f491f812e" (UID: "c80d239a-c8b0-4869-80e7-6e6f491f812e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:37:03.341300 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:03.341270 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c80d239a-c8b0-4869-80e7-6e6f491f812e-maas-api-xks-ca" pod "c80d239a-c8b0-4869-80e7-6e6f491f812e" (UID: "c80d239a-c8b0-4869-80e7-6e6f491f812e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:37:03.347441 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:03.347409 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c80d239a-c8b0-4869-80e7-6e6f491f812e-kube-api-access-rlccs" pod "c80d239a-c8b0-4869-80e7-6e6f491f812e" (UID: "c80d239a-c8b0-4869-80e7-6e6f491f812e"). InnerVolumeSpecName "kube-api-access-rlccs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:37:03.438698 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:03.438659 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c80d239a-c8b0-4869-80e7-6e6f491f812e-maas-api-xks-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:37:03.438698 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:03.438697 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c80d239a-c8b0-4869-80e7-6e6f491f812e-maas-api-openshift-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:37:03.438698 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:03.438713 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rlccs\" (UniqueName: \"kubernetes.io/projected/c80d239a-c8b0-4869-80e7-6e6f491f812e-kube-api-access-rlccs\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:37:04.047387 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:04.047355 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20e416e2-6t2tp"] Nov 02 06:37:04.055885 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:04.055857 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f42bb79-ddrpm" Nov 02 06:37:04.056048 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:04.055889 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f42bb79-ddrpm" event={"ID":"c80d239a-c8b0-4869-80e7-6e6f491f812e","Type":"ContainerDied","Data":"9a867efdde6f9d1e79ce63571ff9fc6dc4d4d33b29f1327b17126612cdee9822"} Nov 02 06:37:04.056048 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:04.055960 5916 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9a867efdde6f9d1e79ce63571ff9fc6dc4d4d33b29f1327b17126612cdee9822" Nov 02 06:37:04.057947 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:04.057885 5916 generic.go:356] "Generic (PLEG): container finished" podID="a491cd95-8967-4f09-82f9-d154761e79ea" containerID="07acc4b8b08df889f973d4cd62a74b6069a3402a875c65052ae14d6763476a4e" exitCode=0 Nov 02 06:37:04.058073 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:04.057957 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06e94cdb-5sj4f" event={"ID":"a491cd95-8967-4f09-82f9-d154761e79ea","Type":"ContainerDied","Data":"07acc4b8b08df889f973d4cd62a74b6069a3402a875c65052ae14d6763476a4e"} Nov 02 06:37:04.093396 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:04.093372 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20e416e2-6t2tp"] Nov 02 06:37:05.232898 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:05.232871 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06e94cdb-5sj4f" Nov 02 06:37:05.366578 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:05.366479 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a491cd95-8967-4f09-82f9-d154761e79ea-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a491cd95-8967-4f09-82f9-d154761e79ea-maas-api-openshift-ca\") pod \"a491cd95-8967-4f09-82f9-d154761e79ea\" (UID: \"a491cd95-8967-4f09-82f9-d154761e79ea\") " Nov 02 06:37:05.366578 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:05.366544 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a491cd95-8967-4f09-82f9-d154761e79ea-kube-api-access-lq5lq\" (UniqueName: \"kubernetes.io/projected/a491cd95-8967-4f09-82f9-d154761e79ea-kube-api-access-lq5lq\") pod \"a491cd95-8967-4f09-82f9-d154761e79ea\" (UID: \"a491cd95-8967-4f09-82f9-d154761e79ea\") " Nov 02 06:37:05.366849 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:05.366641 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a491cd95-8967-4f09-82f9-d154761e79ea-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a491cd95-8967-4f09-82f9-d154761e79ea-maas-api-xks-ca\") pod \"a491cd95-8967-4f09-82f9-d154761e79ea\" (UID: \"a491cd95-8967-4f09-82f9-d154761e79ea\") " Nov 02 06:37:05.367665 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:05.367613 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a491cd95-8967-4f09-82f9-d154761e79ea-maas-api-openshift-ca" pod "a491cd95-8967-4f09-82f9-d154761e79ea" (UID: "a491cd95-8967-4f09-82f9-d154761e79ea"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:37:05.368031 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:05.367995 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a491cd95-8967-4f09-82f9-d154761e79ea-maas-api-openshift-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:37:05.369542 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:05.369513 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a491cd95-8967-4f09-82f9-d154761e79ea-maas-api-xks-ca" pod "a491cd95-8967-4f09-82f9-d154761e79ea" (UID: "a491cd95-8967-4f09-82f9-d154761e79ea"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:37:05.369767 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:05.369748 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a491cd95-8967-4f09-82f9-d154761e79ea-kube-api-access-lq5lq" pod "a491cd95-8967-4f09-82f9-d154761e79ea" (UID: "a491cd95-8967-4f09-82f9-d154761e79ea"). InnerVolumeSpecName "kube-api-access-lq5lq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:37:05.469051 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:05.469017 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a491cd95-8967-4f09-82f9-d154761e79ea-maas-api-xks-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:37:05.469051 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:05.469045 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lq5lq\" (UniqueName: \"kubernetes.io/projected/a491cd95-8967-4f09-82f9-d154761e79ea-kube-api-access-lq5lq\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:37:05.842980 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:05.842949 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e02b453-ca29-488c-a56c-3281e1d39f2f" path="/var/lib/kubelet/pods/0e02b453-ca29-488c-a56c-3281e1d39f2f/volumes" Nov 02 06:37:06.050373 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:06.050332 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38e2d5b3-9qdrh"] Nov 02 06:37:06.077474 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:06.077441 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06e94cdb-5sj4f" event={"ID":"a491cd95-8967-4f09-82f9-d154761e79ea","Type":"ContainerDied","Data":"426d60d5d7b6233bbe56df91743a8b38a4014ad82d7c04fcc9316fdb69a73059"} Nov 02 06:37:06.077648 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:06.077484 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06e94cdb-5sj4f" Nov 02 06:37:06.077648 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:06.077490 5916 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="426d60d5d7b6233bbe56df91743a8b38a4014ad82d7c04fcc9316fdb69a73059" Nov 02 06:37:06.096137 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:06.096066 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38e2d5b3-9qdrh"] Nov 02 06:37:07.650329 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:07.650277 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-262f26f2-p6vln"] Nov 02 06:37:07.651815 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:07.651788 5916 cpu_manager.go:414] "containerMap: removing container" podUID="a491cd95-8967-4f09-82f9-d154761e79ea" containerName="revoke-keys" Nov 02 06:37:07.656037 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:07.656017 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a491cd95-8967-4f09-82f9-d154761e79ea" containerName="revoke-keys" Nov 02 06:37:07.656183 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:07.656169 5916 cpu_manager.go:414] "containerMap: removing container" podUID="c80d239a-c8b0-4869-80e7-6e6f491f812e" containerName="revoke-keys" Nov 02 06:37:07.656271 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:07.656259 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c80d239a-c8b0-4869-80e7-6e6f491f812e" containerName="revoke-keys" Nov 02 06:37:07.656616 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:07.656597 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="a491cd95-8967-4f09-82f9-d154761e79ea" containerName="revoke-keys" Nov 02 06:37:07.656725 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:07.656712 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="c80d239a-c8b0-4869-80e7-6e6f491f812e" containerName="revoke-keys" Nov 02 06:37:07.661531 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:07.661510 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-262f26f2-p6vln" Nov 02 06:37:07.699930 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:07.697309 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-262f26f2-p6vln"] Nov 02 06:37:07.760965 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:07.741102 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e2b0972-08eb-4028-9e16-1905c9855114-maas-api-xks-ca\") pod \"maas-api-revoke-sub-262f26f2-p6vln\" (UID: \"8e2b0972-08eb-4028-9e16-1905c9855114\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-262f26f2-p6vln" Nov 02 06:37:07.760965 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:07.741177 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4nnf\" (UniqueName: \"kubernetes.io/projected/8e2b0972-08eb-4028-9e16-1905c9855114-kube-api-access-l4nnf\") pod \"maas-api-revoke-sub-262f26f2-p6vln\" (UID: \"8e2b0972-08eb-4028-9e16-1905c9855114\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-262f26f2-p6vln" Nov 02 06:37:07.760965 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:07.741247 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e2b0972-08eb-4028-9e16-1905c9855114-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-262f26f2-p6vln\" (UID: \"8e2b0972-08eb-4028-9e16-1905c9855114\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-262f26f2-p6vln" Nov 02 06:37:07.760965 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:07.748249 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-68l4r\"" Nov 02 06:37:07.760965 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:07.749984 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 06:37:07.860765 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:07.859403 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e2b0972-08eb-4028-9e16-1905c9855114-maas-api-xks-ca\") pod \"maas-api-revoke-sub-262f26f2-p6vln\" (UID: \"8e2b0972-08eb-4028-9e16-1905c9855114\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-262f26f2-p6vln" Nov 02 06:37:07.860765 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:07.859478 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4nnf\" (UniqueName: \"kubernetes.io/projected/8e2b0972-08eb-4028-9e16-1905c9855114-kube-api-access-l4nnf\") pod \"maas-api-revoke-sub-262f26f2-p6vln\" (UID: \"8e2b0972-08eb-4028-9e16-1905c9855114\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-262f26f2-p6vln" Nov 02 06:37:07.860765 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:07.859543 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e2b0972-08eb-4028-9e16-1905c9855114-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-262f26f2-p6vln\" (UID: \"8e2b0972-08eb-4028-9e16-1905c9855114\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-262f26f2-p6vln" Nov 02 06:37:07.860765 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:07.860461 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e2b0972-08eb-4028-9e16-1905c9855114-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-262f26f2-p6vln\" (UID: \"8e2b0972-08eb-4028-9e16-1905c9855114\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-262f26f2-p6vln" Nov 02 06:37:07.914761 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:07.914643 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e2b0972-08eb-4028-9e16-1905c9855114-maas-api-xks-ca\") pod \"maas-api-revoke-sub-262f26f2-p6vln\" (UID: \"8e2b0972-08eb-4028-9e16-1905c9855114\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-262f26f2-p6vln" Nov 02 06:37:07.934587 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:07.934556 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4nnf\" (UniqueName: \"kubernetes.io/projected/8e2b0972-08eb-4028-9e16-1905c9855114-kube-api-access-l4nnf\") pod \"maas-api-revoke-sub-262f26f2-p6vln\" (UID: \"8e2b0972-08eb-4028-9e16-1905c9855114\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-262f26f2-p6vln" Nov 02 06:37:07.993933 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:07.985783 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e308286b-3f6e-4950-9605-daa0d1d68b5d" path="/var/lib/kubelet/pods/e308286b-3f6e-4950-9605-daa0d1d68b5d/volumes" Nov 02 06:37:08.049551 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:08.049518 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8475f4f7-p54s9"] Nov 02 06:37:08.058179 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:08.058144 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8475f4f7-p54s9"] Nov 02 06:37:08.093174 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:08.091895 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-262f26f2-p6vln" Nov 02 06:37:08.790028 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:08.786151 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-262f26f2-p6vln"] Nov 02 06:37:09.123848 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:09.123760 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-262f26f2-p6vln" event={"ID":"8e2b0972-08eb-4028-9e16-1905c9855114","Type":"ContainerStarted","Data":"0f68dbab2467db4c5e7bc69b99febbddd12547e82e18c0991c2c40756c22ba1c"} Nov 02 06:37:09.845792 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:09.845753 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc833f51-60f9-4e3c-8c15-1d0db873eea5" path="/var/lib/kubelet/pods/bc833f51-60f9-4e3c-8c15-1d0db873eea5/volumes" Nov 02 06:37:10.136166 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:10.136129 5916 generic.go:356] "Generic (PLEG): container finished" podID="8e2b0972-08eb-4028-9e16-1905c9855114" containerID="1b626993ae088eed6c2d0befb2c49e83b403221684b69bc4f322f3f478cbd588" exitCode=0 Nov 02 06:37:10.136365 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:10.136217 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-262f26f2-p6vln" event={"ID":"8e2b0972-08eb-4028-9e16-1905c9855114","Type":"ContainerDied","Data":"1b626993ae088eed6c2d0befb2c49e83b403221684b69bc4f322f3f478cbd588"} Nov 02 06:37:11.320195 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:11.320168 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-262f26f2-p6vln" Nov 02 06:37:11.424299 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:11.424219 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e2b0972-08eb-4028-9e16-1905c9855114-kube-api-access-l4nnf\" (UniqueName: \"kubernetes.io/projected/8e2b0972-08eb-4028-9e16-1905c9855114-kube-api-access-l4nnf\") pod \"8e2b0972-08eb-4028-9e16-1905c9855114\" (UID: \"8e2b0972-08eb-4028-9e16-1905c9855114\") " Nov 02 06:37:11.424438 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:11.424359 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e2b0972-08eb-4028-9e16-1905c9855114-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e2b0972-08eb-4028-9e16-1905c9855114-maas-api-xks-ca\") pod \"8e2b0972-08eb-4028-9e16-1905c9855114\" (UID: \"8e2b0972-08eb-4028-9e16-1905c9855114\") " Nov 02 06:37:11.424438 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:11.424408 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e2b0972-08eb-4028-9e16-1905c9855114-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e2b0972-08eb-4028-9e16-1905c9855114-maas-api-openshift-ca\") pod \"8e2b0972-08eb-4028-9e16-1905c9855114\" (UID: \"8e2b0972-08eb-4028-9e16-1905c9855114\") " Nov 02 06:37:11.424778 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:11.424751 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e2b0972-08eb-4028-9e16-1905c9855114-maas-api-openshift-ca" pod "8e2b0972-08eb-4028-9e16-1905c9855114" (UID: "8e2b0972-08eb-4028-9e16-1905c9855114"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:37:11.426528 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:11.426501 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e2b0972-08eb-4028-9e16-1905c9855114-kube-api-access-l4nnf" pod "8e2b0972-08eb-4028-9e16-1905c9855114" (UID: "8e2b0972-08eb-4028-9e16-1905c9855114"). InnerVolumeSpecName "kube-api-access-l4nnf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:37:11.426830 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:11.426800 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e2b0972-08eb-4028-9e16-1905c9855114-maas-api-xks-ca" pod "8e2b0972-08eb-4028-9e16-1905c9855114" (UID: "8e2b0972-08eb-4028-9e16-1905c9855114"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:37:11.526348 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:11.526314 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4nnf\" (UniqueName: \"kubernetes.io/projected/8e2b0972-08eb-4028-9e16-1905c9855114-kube-api-access-l4nnf\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:37:11.526348 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:11.526349 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e2b0972-08eb-4028-9e16-1905c9855114-maas-api-xks-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:37:11.526573 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:11.526362 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e2b0972-08eb-4028-9e16-1905c9855114-maas-api-openshift-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:37:12.157617 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:12.157581 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-262f26f2-p6vln" Nov 02 06:37:12.157617 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:12.157603 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-262f26f2-p6vln" event={"ID":"8e2b0972-08eb-4028-9e16-1905c9855114","Type":"ContainerDied","Data":"0f68dbab2467db4c5e7bc69b99febbddd12547e82e18c0991c2c40756c22ba1c"} Nov 02 06:37:12.157845 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:12.157663 5916 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0f68dbab2467db4c5e7bc69b99febbddd12547e82e18c0991c2c40756c22ba1c" Nov 02 06:37:13.117568 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:13.117537 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6119a045-whwqc"] Nov 02 06:37:13.118100 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:13.118086 5916 cpu_manager.go:414] "containerMap: removing container" podUID="8e2b0972-08eb-4028-9e16-1905c9855114" containerName="revoke-keys" Nov 02 06:37:13.118160 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:13.118100 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e2b0972-08eb-4028-9e16-1905c9855114" containerName="revoke-keys" Nov 02 06:37:13.118226 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:13.118217 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e2b0972-08eb-4028-9e16-1905c9855114" containerName="revoke-keys" Nov 02 06:37:13.121790 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:13.121772 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6119a045-whwqc" Nov 02 06:37:13.170784 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:13.170744 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 06:37:13.216679 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:13.216648 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-68l4r\"" Nov 02 06:37:13.221211 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:13.221185 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6119a045-whwqc"] Nov 02 06:37:13.248846 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:13.248815 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/106a8465-7e5e-4f8d-ae57-f8b00c6e5a09-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6119a045-whwqc\" (UID: \"106a8465-7e5e-4f8d-ae57-f8b00c6e5a09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6119a045-whwqc" Nov 02 06:37:13.249482 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:13.248859 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/106a8465-7e5e-4f8d-ae57-f8b00c6e5a09-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6119a045-whwqc\" (UID: \"106a8465-7e5e-4f8d-ae57-f8b00c6e5a09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6119a045-whwqc" Nov 02 06:37:13.249482 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:13.249074 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cltfn\" (UniqueName: \"kubernetes.io/projected/106a8465-7e5e-4f8d-ae57-f8b00c6e5a09-kube-api-access-cltfn\") pod \"maas-api-revoke-sub-6119a045-whwqc\" (UID: \"106a8465-7e5e-4f8d-ae57-f8b00c6e5a09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6119a045-whwqc" Nov 02 06:37:13.352208 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:13.351598 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cltfn\" (UniqueName: \"kubernetes.io/projected/106a8465-7e5e-4f8d-ae57-f8b00c6e5a09-kube-api-access-cltfn\") pod \"maas-api-revoke-sub-6119a045-whwqc\" (UID: \"106a8465-7e5e-4f8d-ae57-f8b00c6e5a09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6119a045-whwqc" Nov 02 06:37:13.352208 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:13.351718 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/106a8465-7e5e-4f8d-ae57-f8b00c6e5a09-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6119a045-whwqc\" (UID: \"106a8465-7e5e-4f8d-ae57-f8b00c6e5a09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6119a045-whwqc" Nov 02 06:37:13.352208 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:13.351761 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/106a8465-7e5e-4f8d-ae57-f8b00c6e5a09-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6119a045-whwqc\" (UID: \"106a8465-7e5e-4f8d-ae57-f8b00c6e5a09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6119a045-whwqc" Nov 02 06:37:13.353135 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:13.353105 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/106a8465-7e5e-4f8d-ae57-f8b00c6e5a09-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6119a045-whwqc\" (UID: \"106a8465-7e5e-4f8d-ae57-f8b00c6e5a09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6119a045-whwqc" Nov 02 06:37:13.355217 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:13.355191 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/106a8465-7e5e-4f8d-ae57-f8b00c6e5a09-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6119a045-whwqc\" (UID: \"106a8465-7e5e-4f8d-ae57-f8b00c6e5a09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6119a045-whwqc" Nov 02 06:37:13.406060 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:13.406030 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cltfn\" (UniqueName: \"kubernetes.io/projected/106a8465-7e5e-4f8d-ae57-f8b00c6e5a09-kube-api-access-cltfn\") pod \"maas-api-revoke-sub-6119a045-whwqc\" (UID: \"106a8465-7e5e-4f8d-ae57-f8b00c6e5a09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6119a045-whwqc" Nov 02 06:37:13.435397 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:13.435365 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6119a045-whwqc" Nov 02 06:37:13.690414 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:13.690339 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75a270a2-sj59m"] Nov 02 06:37:13.695651 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:13.695627 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75a270a2-sj59m" Nov 02 06:37:13.747757 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:13.747720 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75a270a2-sj59m"] Nov 02 06:37:13.857859 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:13.857815 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6119a045-whwqc"] Nov 02 06:37:13.858389 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:13.858361 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d220594f-4b71-4f58-9efd-d95525b5f8a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-75a270a2-sj59m\" (UID: \"d220594f-4b71-4f58-9efd-d95525b5f8a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75a270a2-sj59m" Nov 02 06:37:13.858531 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:13.858475 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d220594f-4b71-4f58-9efd-d95525b5f8a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-75a270a2-sj59m\" (UID: \"d220594f-4b71-4f58-9efd-d95525b5f8a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75a270a2-sj59m" Nov 02 06:37:13.858657 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:13.858528 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tb4l8\" (UniqueName: \"kubernetes.io/projected/d220594f-4b71-4f58-9efd-d95525b5f8a0-kube-api-access-tb4l8\") pod \"maas-api-revoke-sub-75a270a2-sj59m\" (UID: \"d220594f-4b71-4f58-9efd-d95525b5f8a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75a270a2-sj59m" Nov 02 06:37:13.861011 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:37:13.860984 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod106a8465_7e5e_4f8d_ae57_f8b00c6e5a09.slice/crio-6b00d5679f01deb1cbe2190c1987102e68c41a464c9a0472841b860a59083d7b WatchSource:0}: Error finding container 6b00d5679f01deb1cbe2190c1987102e68c41a464c9a0472841b860a59083d7b: Status 404 returned error can't find the container with id 6b00d5679f01deb1cbe2190c1987102e68c41a464c9a0472841b860a59083d7b Nov 02 06:37:13.959540 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:13.959459 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d220594f-4b71-4f58-9efd-d95525b5f8a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-75a270a2-sj59m\" (UID: \"d220594f-4b71-4f58-9efd-d95525b5f8a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75a270a2-sj59m" Nov 02 06:37:13.959687 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:13.959572 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d220594f-4b71-4f58-9efd-d95525b5f8a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-75a270a2-sj59m\" (UID: \"d220594f-4b71-4f58-9efd-d95525b5f8a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75a270a2-sj59m" Nov 02 06:37:13.959687 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:13.959615 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tb4l8\" (UniqueName: \"kubernetes.io/projected/d220594f-4b71-4f58-9efd-d95525b5f8a0-kube-api-access-tb4l8\") pod \"maas-api-revoke-sub-75a270a2-sj59m\" (UID: \"d220594f-4b71-4f58-9efd-d95525b5f8a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75a270a2-sj59m" Nov 02 06:37:13.960290 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:13.960261 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d220594f-4b71-4f58-9efd-d95525b5f8a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-75a270a2-sj59m\" (UID: \"d220594f-4b71-4f58-9efd-d95525b5f8a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75a270a2-sj59m" Nov 02 06:37:13.961997 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:13.961975 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d220594f-4b71-4f58-9efd-d95525b5f8a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-75a270a2-sj59m\" (UID: \"d220594f-4b71-4f58-9efd-d95525b5f8a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75a270a2-sj59m" Nov 02 06:37:14.014116 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:14.014090 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tb4l8\" (UniqueName: \"kubernetes.io/projected/d220594f-4b71-4f58-9efd-d95525b5f8a0-kube-api-access-tb4l8\") pod \"maas-api-revoke-sub-75a270a2-sj59m\" (UID: \"d220594f-4b71-4f58-9efd-d95525b5f8a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75a270a2-sj59m" Nov 02 06:37:14.176474 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:14.176434 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6119a045-whwqc" event={"ID":"106a8465-7e5e-4f8d-ae57-f8b00c6e5a09","Type":"ContainerStarted","Data":"6b00d5679f01deb1cbe2190c1987102e68c41a464c9a0472841b860a59083d7b"} Nov 02 06:37:14.309745 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:14.309715 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75a270a2-sj59m" Nov 02 06:37:14.701473 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:14.701426 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75a270a2-sj59m"] Nov 02 06:37:14.704985 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:37:14.704958 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd220594f_4b71_4f58_9efd_d95525b5f8a0.slice/crio-bb69d414a0a03e85e504411be8c7380cf22d425bf8c221fddd720762f2379b99 WatchSource:0}: Error finding container bb69d414a0a03e85e504411be8c7380cf22d425bf8c221fddd720762f2379b99: Status 404 returned error can't find the container with id bb69d414a0a03e85e504411be8c7380cf22d425bf8c221fddd720762f2379b99 Nov 02 06:37:15.185290 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:15.185255 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75a270a2-sj59m" event={"ID":"d220594f-4b71-4f58-9efd-d95525b5f8a0","Type":"ContainerStarted","Data":"14eccec5a556da217ed081f8a420c720be6b5bf11ba748ee7f60b10c192f98d6"} Nov 02 06:37:15.185761 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:15.185317 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75a270a2-sj59m" event={"ID":"d220594f-4b71-4f58-9efd-d95525b5f8a0","Type":"ContainerStarted","Data":"bb69d414a0a03e85e504411be8c7380cf22d425bf8c221fddd720762f2379b99"} Nov 02 06:37:15.186757 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:15.186731 5916 generic.go:356] "Generic (PLEG): container finished" podID="106a8465-7e5e-4f8d-ae57-f8b00c6e5a09" containerID="1cec20cbab42866ab6af84b4ed85481b7f7237cb0698d5aee8e4ffa7e0935f10" exitCode=0 Nov 02 06:37:15.186882 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:15.186810 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6119a045-whwqc" event={"ID":"106a8465-7e5e-4f8d-ae57-f8b00c6e5a09","Type":"ContainerDied","Data":"1cec20cbab42866ab6af84b4ed85481b7f7237cb0698d5aee8e4ffa7e0935f10"} Nov 02 06:37:15.292242 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:15.292140 5916 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75a270a2-sj59m" podStartSLOduration=2.292128233 podStartE2EDuration="2.292128233s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:37:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:37:15.289828096 +0000 UTC m=+1223.561648491" watchObservedRunningTime="2026-11-02 06:37:15.292128233 +0000 UTC m=+1223.563948626" Nov 02 06:37:16.196738 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:16.196706 5916 generic.go:356] "Generic (PLEG): container finished" podID="d220594f-4b71-4f58-9efd-d95525b5f8a0" containerID="14eccec5a556da217ed081f8a420c720be6b5bf11ba748ee7f60b10c192f98d6" exitCode=0 Nov 02 06:37:16.197195 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:16.196789 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75a270a2-sj59m" event={"ID":"d220594f-4b71-4f58-9efd-d95525b5f8a0","Type":"ContainerDied","Data":"14eccec5a556da217ed081f8a420c720be6b5bf11ba748ee7f60b10c192f98d6"} Nov 02 06:37:16.360155 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:16.360124 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6119a045-whwqc" Nov 02 06:37:16.401531 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:16.401485 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/106a8465-7e5e-4f8d-ae57-f8b00c6e5a09-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/106a8465-7e5e-4f8d-ae57-f8b00c6e5a09-maas-api-xks-ca\") pod \"106a8465-7e5e-4f8d-ae57-f8b00c6e5a09\" (UID: \"106a8465-7e5e-4f8d-ae57-f8b00c6e5a09\") " Nov 02 06:37:16.401680 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:16.401544 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/106a8465-7e5e-4f8d-ae57-f8b00c6e5a09-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/106a8465-7e5e-4f8d-ae57-f8b00c6e5a09-maas-api-openshift-ca\") pod \"106a8465-7e5e-4f8d-ae57-f8b00c6e5a09\" (UID: \"106a8465-7e5e-4f8d-ae57-f8b00c6e5a09\") " Nov 02 06:37:16.401680 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:16.401637 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/106a8465-7e5e-4f8d-ae57-f8b00c6e5a09-kube-api-access-cltfn\" (UniqueName: \"kubernetes.io/projected/106a8465-7e5e-4f8d-ae57-f8b00c6e5a09-kube-api-access-cltfn\") pod \"106a8465-7e5e-4f8d-ae57-f8b00c6e5a09\" (UID: \"106a8465-7e5e-4f8d-ae57-f8b00c6e5a09\") " Nov 02 06:37:16.401873 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:16.401849 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/106a8465-7e5e-4f8d-ae57-f8b00c6e5a09-maas-api-openshift-ca" pod "106a8465-7e5e-4f8d-ae57-f8b00c6e5a09" (UID: "106a8465-7e5e-4f8d-ae57-f8b00c6e5a09"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:37:16.402108 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:16.402087 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/106a8465-7e5e-4f8d-ae57-f8b00c6e5a09-maas-api-openshift-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:37:16.403570 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:16.403545 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/106a8465-7e5e-4f8d-ae57-f8b00c6e5a09-maas-api-xks-ca" pod "106a8465-7e5e-4f8d-ae57-f8b00c6e5a09" (UID: "106a8465-7e5e-4f8d-ae57-f8b00c6e5a09"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:37:16.403656 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:16.403606 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/106a8465-7e5e-4f8d-ae57-f8b00c6e5a09-kube-api-access-cltfn" pod "106a8465-7e5e-4f8d-ae57-f8b00c6e5a09" (UID: "106a8465-7e5e-4f8d-ae57-f8b00c6e5a09"). InnerVolumeSpecName "kube-api-access-cltfn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:37:16.503042 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:16.502949 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/106a8465-7e5e-4f8d-ae57-f8b00c6e5a09-maas-api-xks-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:37:16.503042 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:16.502979 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cltfn\" (UniqueName: \"kubernetes.io/projected/106a8465-7e5e-4f8d-ae57-f8b00c6e5a09-kube-api-access-cltfn\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:37:17.206067 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:17.206035 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6119a045-whwqc" Nov 02 06:37:17.206067 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:17.206049 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6119a045-whwqc" event={"ID":"106a8465-7e5e-4f8d-ae57-f8b00c6e5a09","Type":"ContainerDied","Data":"6b00d5679f01deb1cbe2190c1987102e68c41a464c9a0472841b860a59083d7b"} Nov 02 06:37:17.206558 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:17.206092 5916 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6b00d5679f01deb1cbe2190c1987102e68c41a464c9a0472841b860a59083d7b" Nov 02 06:37:17.357311 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:17.357286 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75a270a2-sj59m" Nov 02 06:37:17.414260 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:17.414224 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d220594f-4b71-4f58-9efd-d95525b5f8a0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d220594f-4b71-4f58-9efd-d95525b5f8a0-maas-api-openshift-ca\") pod \"d220594f-4b71-4f58-9efd-d95525b5f8a0\" (UID: \"d220594f-4b71-4f58-9efd-d95525b5f8a0\") " Nov 02 06:37:17.414421 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:17.414307 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d220594f-4b71-4f58-9efd-d95525b5f8a0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d220594f-4b71-4f58-9efd-d95525b5f8a0-maas-api-xks-ca\") pod \"d220594f-4b71-4f58-9efd-d95525b5f8a0\" (UID: \"d220594f-4b71-4f58-9efd-d95525b5f8a0\") " Nov 02 06:37:17.414421 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:17.414342 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d220594f-4b71-4f58-9efd-d95525b5f8a0-kube-api-access-tb4l8\" (UniqueName: \"kubernetes.io/projected/d220594f-4b71-4f58-9efd-d95525b5f8a0-kube-api-access-tb4l8\") pod \"d220594f-4b71-4f58-9efd-d95525b5f8a0\" (UID: \"d220594f-4b71-4f58-9efd-d95525b5f8a0\") " Nov 02 06:37:17.414586 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:17.414562 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d220594f-4b71-4f58-9efd-d95525b5f8a0-maas-api-openshift-ca" pod "d220594f-4b71-4f58-9efd-d95525b5f8a0" (UID: "d220594f-4b71-4f58-9efd-d95525b5f8a0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:37:17.414763 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:17.414746 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d220594f-4b71-4f58-9efd-d95525b5f8a0-maas-api-openshift-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:37:17.416514 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:17.416486 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d220594f-4b71-4f58-9efd-d95525b5f8a0-maas-api-xks-ca" pod "d220594f-4b71-4f58-9efd-d95525b5f8a0" (UID: "d220594f-4b71-4f58-9efd-d95525b5f8a0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:37:17.416601 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:17.416556 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d220594f-4b71-4f58-9efd-d95525b5f8a0-kube-api-access-tb4l8" pod "d220594f-4b71-4f58-9efd-d95525b5f8a0" (UID: "d220594f-4b71-4f58-9efd-d95525b5f8a0"). InnerVolumeSpecName "kube-api-access-tb4l8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:37:17.515831 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:17.515761 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d220594f-4b71-4f58-9efd-d95525b5f8a0-maas-api-xks-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:37:17.515831 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:17.515789 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tb4l8\" (UniqueName: \"kubernetes.io/projected/d220594f-4b71-4f58-9efd-d95525b5f8a0-kube-api-access-tb4l8\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:37:17.738707 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:17.738671 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04d7ec61-hch7s"] Nov 02 06:37:17.739276 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:17.739261 5916 cpu_manager.go:414] "containerMap: removing container" podUID="d220594f-4b71-4f58-9efd-d95525b5f8a0" containerName="revoke-keys" Nov 02 06:37:17.739328 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:17.739277 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d220594f-4b71-4f58-9efd-d95525b5f8a0" containerName="revoke-keys" Nov 02 06:37:17.739328 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:17.739320 5916 cpu_manager.go:414] "containerMap: removing container" podUID="106a8465-7e5e-4f8d-ae57-f8b00c6e5a09" containerName="revoke-keys" Nov 02 06:37:17.739328 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:17.739326 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="106a8465-7e5e-4f8d-ae57-f8b00c6e5a09" containerName="revoke-keys" Nov 02 06:37:17.739456 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:17.739443 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="d220594f-4b71-4f58-9efd-d95525b5f8a0" containerName="revoke-keys" Nov 02 06:37:17.739504 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:17.739467 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="106a8465-7e5e-4f8d-ae57-f8b00c6e5a09" containerName="revoke-keys" Nov 02 06:37:17.743788 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:17.743767 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04d7ec61-hch7s" Nov 02 06:37:17.796522 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:17.796451 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04d7ec61-hch7s"] Nov 02 06:37:17.819862 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:17.819832 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k97gq\" (UniqueName: \"kubernetes.io/projected/e338821a-dac8-4f6f-80fa-dd6e4a180258-kube-api-access-k97gq\") pod \"maas-api-revoke-sub-04d7ec61-hch7s\" (UID: \"e338821a-dac8-4f6f-80fa-dd6e4a180258\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04d7ec61-hch7s" Nov 02 06:37:17.820040 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:17.819890 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e338821a-dac8-4f6f-80fa-dd6e4a180258-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-04d7ec61-hch7s\" (UID: \"e338821a-dac8-4f6f-80fa-dd6e4a180258\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04d7ec61-hch7s" Nov 02 06:37:17.820040 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:17.819983 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e338821a-dac8-4f6f-80fa-dd6e4a180258-maas-api-xks-ca\") pod \"maas-api-revoke-sub-04d7ec61-hch7s\" (UID: \"e338821a-dac8-4f6f-80fa-dd6e4a180258\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04d7ec61-hch7s" Nov 02 06:37:17.921322 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:17.921290 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e338821a-dac8-4f6f-80fa-dd6e4a180258-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-04d7ec61-hch7s\" (UID: \"e338821a-dac8-4f6f-80fa-dd6e4a180258\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04d7ec61-hch7s" Nov 02 06:37:17.921470 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:17.921339 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e338821a-dac8-4f6f-80fa-dd6e4a180258-maas-api-xks-ca\") pod \"maas-api-revoke-sub-04d7ec61-hch7s\" (UID: \"e338821a-dac8-4f6f-80fa-dd6e4a180258\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04d7ec61-hch7s" Nov 02 06:37:17.921549 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:17.921518 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k97gq\" (UniqueName: \"kubernetes.io/projected/e338821a-dac8-4f6f-80fa-dd6e4a180258-kube-api-access-k97gq\") pod \"maas-api-revoke-sub-04d7ec61-hch7s\" (UID: \"e338821a-dac8-4f6f-80fa-dd6e4a180258\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04d7ec61-hch7s" Nov 02 06:37:17.926952 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:17.922415 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e338821a-dac8-4f6f-80fa-dd6e4a180258-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-04d7ec61-hch7s\" (UID: \"e338821a-dac8-4f6f-80fa-dd6e4a180258\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04d7ec61-hch7s" Nov 02 06:37:17.932519 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:17.932496 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e338821a-dac8-4f6f-80fa-dd6e4a180258-maas-api-xks-ca\") pod \"maas-api-revoke-sub-04d7ec61-hch7s\" (UID: \"e338821a-dac8-4f6f-80fa-dd6e4a180258\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04d7ec61-hch7s" Nov 02 06:37:17.974183 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:17.974155 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k97gq\" (UniqueName: \"kubernetes.io/projected/e338821a-dac8-4f6f-80fa-dd6e4a180258-kube-api-access-k97gq\") pod \"maas-api-revoke-sub-04d7ec61-hch7s\" (UID: \"e338821a-dac8-4f6f-80fa-dd6e4a180258\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04d7ec61-hch7s" Nov 02 06:37:18.058383 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:18.058305 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04d7ec61-hch7s" Nov 02 06:37:18.214599 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:18.214571 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75a270a2-sj59m" Nov 02 06:37:18.215033 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:18.214612 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75a270a2-sj59m" event={"ID":"d220594f-4b71-4f58-9efd-d95525b5f8a0","Type":"ContainerDied","Data":"bb69d414a0a03e85e504411be8c7380cf22d425bf8c221fddd720762f2379b99"} Nov 02 06:37:18.215033 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:18.214704 5916 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bb69d414a0a03e85e504411be8c7380cf22d425bf8c221fddd720762f2379b99" Nov 02 06:37:18.432876 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:18.432851 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04d7ec61-hch7s"] Nov 02 06:37:18.435100 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:37:18.435073 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode338821a_dac8_4f6f_80fa_dd6e4a180258.slice/crio-7811252b895c29918d7d0b216d5492f3084e6735d9f7c7acd3be89045a14a817 WatchSource:0}: Error finding container 7811252b895c29918d7d0b216d5492f3084e6735d9f7c7acd3be89045a14a817: Status 404 returned error can't find the container with id 7811252b895c29918d7d0b216d5492f3084e6735d9f7c7acd3be89045a14a817 Nov 02 06:37:19.224632 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:19.224595 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04d7ec61-hch7s" event={"ID":"e338821a-dac8-4f6f-80fa-dd6e4a180258","Type":"ContainerStarted","Data":"c6fe07e6465bfbbff5b5d2d9a7ebc012cfac082697c13934b3547d024e411925"} Nov 02 06:37:19.225171 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:19.224658 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04d7ec61-hch7s" event={"ID":"e338821a-dac8-4f6f-80fa-dd6e4a180258","Type":"ContainerStarted","Data":"7811252b895c29918d7d0b216d5492f3084e6735d9f7c7acd3be89045a14a817"} Nov 02 06:37:19.334120 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:19.334071 5916 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04d7ec61-hch7s" podStartSLOduration=2.334057259 podStartE2EDuration="2.334057259s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:37:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:37:19.330190657 +0000 UTC m=+1227.602011050" watchObservedRunningTime="2026-11-02 06:37:19.334057259 +0000 UTC m=+1227.605877652" Nov 02 06:37:19.893977 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:37:19.893940 5916 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p98cq" containerName="extractor" Nov 02 06:37:20.234166 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:20.234088 5916 generic.go:356] "Generic (PLEG): container finished" podID="e338821a-dac8-4f6f-80fa-dd6e4a180258" containerID="c6fe07e6465bfbbff5b5d2d9a7ebc012cfac082697c13934b3547d024e411925" exitCode=0 Nov 02 06:37:20.234166 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:20.234130 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04d7ec61-hch7s" event={"ID":"e338821a-dac8-4f6f-80fa-dd6e4a180258","Type":"ContainerDied","Data":"c6fe07e6465bfbbff5b5d2d9a7ebc012cfac082697c13934b3547d024e411925"} Nov 02 06:37:21.395219 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:21.395192 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04d7ec61-hch7s" Nov 02 06:37:21.464968 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:21.464938 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e338821a-dac8-4f6f-80fa-dd6e4a180258-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e338821a-dac8-4f6f-80fa-dd6e4a180258-maas-api-xks-ca\") pod \"e338821a-dac8-4f6f-80fa-dd6e4a180258\" (UID: \"e338821a-dac8-4f6f-80fa-dd6e4a180258\") " Nov 02 06:37:21.465149 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:21.465082 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e338821a-dac8-4f6f-80fa-dd6e4a180258-kube-api-access-k97gq\" (UniqueName: \"kubernetes.io/projected/e338821a-dac8-4f6f-80fa-dd6e4a180258-kube-api-access-k97gq\") pod \"e338821a-dac8-4f6f-80fa-dd6e4a180258\" (UID: \"e338821a-dac8-4f6f-80fa-dd6e4a180258\") " Nov 02 06:37:21.465207 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:21.465179 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e338821a-dac8-4f6f-80fa-dd6e4a180258-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e338821a-dac8-4f6f-80fa-dd6e4a180258-maas-api-openshift-ca\") pod \"e338821a-dac8-4f6f-80fa-dd6e4a180258\" (UID: \"e338821a-dac8-4f6f-80fa-dd6e4a180258\") " Nov 02 06:37:21.465660 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:21.465631 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e338821a-dac8-4f6f-80fa-dd6e4a180258-maas-api-openshift-ca" pod "e338821a-dac8-4f6f-80fa-dd6e4a180258" (UID: "e338821a-dac8-4f6f-80fa-dd6e4a180258"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:37:21.467122 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:21.467101 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e338821a-dac8-4f6f-80fa-dd6e4a180258-maas-api-xks-ca" pod "e338821a-dac8-4f6f-80fa-dd6e4a180258" (UID: "e338821a-dac8-4f6f-80fa-dd6e4a180258"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:37:21.467237 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:21.467218 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e338821a-dac8-4f6f-80fa-dd6e4a180258-kube-api-access-k97gq" pod "e338821a-dac8-4f6f-80fa-dd6e4a180258" (UID: "e338821a-dac8-4f6f-80fa-dd6e4a180258"). InnerVolumeSpecName "kube-api-access-k97gq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:37:21.566965 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:21.566857 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e338821a-dac8-4f6f-80fa-dd6e4a180258-maas-api-xks-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:37:21.566965 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:21.566888 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k97gq\" (UniqueName: \"kubernetes.io/projected/e338821a-dac8-4f6f-80fa-dd6e4a180258-kube-api-access-k97gq\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:37:21.566965 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:21.566898 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e338821a-dac8-4f6f-80fa-dd6e4a180258-maas-api-openshift-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:37:22.047515 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:22.047481 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8684e79-gjw67"] Nov 02 06:37:22.094482 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:22.094448 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8684e79-gjw67"] Nov 02 06:37:22.254946 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:22.254920 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04d7ec61-hch7s" Nov 02 06:37:22.254946 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:22.254899 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04d7ec61-hch7s" event={"ID":"e338821a-dac8-4f6f-80fa-dd6e4a180258","Type":"ContainerDied","Data":"7811252b895c29918d7d0b216d5492f3084e6735d9f7c7acd3be89045a14a817"} Nov 02 06:37:22.255164 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:22.254979 5916 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7811252b895c29918d7d0b216d5492f3084e6735d9f7c7acd3be89045a14a817" Nov 02 06:37:22.475220 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:22.475189 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f5fa4b3-9jtjn"] Nov 02 06:37:22.475765 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:22.475751 5916 cpu_manager.go:414] "containerMap: removing container" podUID="e338821a-dac8-4f6f-80fa-dd6e4a180258" containerName="revoke-keys" Nov 02 06:37:22.475807 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:22.475765 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e338821a-dac8-4f6f-80fa-dd6e4a180258" containerName="revoke-keys" Nov 02 06:37:22.475870 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:22.475862 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="e338821a-dac8-4f6f-80fa-dd6e4a180258" containerName="revoke-keys" Nov 02 06:37:22.479697 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:22.479677 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f5fa4b3-9jtjn" Nov 02 06:37:22.528600 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:22.528525 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 06:37:22.528808 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:22.528781 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-68l4r\"" Nov 02 06:37:22.536190 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:22.535391 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f5fa4b3-9jtjn"] Nov 02 06:37:22.579178 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:22.579145 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f458ad6-b62f-4f8a-ad43-95f51f2bee4c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9f5fa4b3-9jtjn\" (UID: \"9f458ad6-b62f-4f8a-ad43-95f51f2bee4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f5fa4b3-9jtjn" Nov 02 06:37:22.579354 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:22.579205 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdrtf\" (UniqueName: \"kubernetes.io/projected/9f458ad6-b62f-4f8a-ad43-95f51f2bee4c-kube-api-access-vdrtf\") pod \"maas-api-revoke-sub-9f5fa4b3-9jtjn\" (UID: \"9f458ad6-b62f-4f8a-ad43-95f51f2bee4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f5fa4b3-9jtjn" Nov 02 06:37:22.579405 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:22.579353 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f458ad6-b62f-4f8a-ad43-95f51f2bee4c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9f5fa4b3-9jtjn\" (UID: \"9f458ad6-b62f-4f8a-ad43-95f51f2bee4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f5fa4b3-9jtjn" Nov 02 06:37:22.680501 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:22.680465 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f458ad6-b62f-4f8a-ad43-95f51f2bee4c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9f5fa4b3-9jtjn\" (UID: \"9f458ad6-b62f-4f8a-ad43-95f51f2bee4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f5fa4b3-9jtjn" Nov 02 06:37:22.680708 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:22.680625 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f458ad6-b62f-4f8a-ad43-95f51f2bee4c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9f5fa4b3-9jtjn\" (UID: \"9f458ad6-b62f-4f8a-ad43-95f51f2bee4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f5fa4b3-9jtjn" Nov 02 06:37:22.680708 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:22.680672 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdrtf\" (UniqueName: \"kubernetes.io/projected/9f458ad6-b62f-4f8a-ad43-95f51f2bee4c-kube-api-access-vdrtf\") pod \"maas-api-revoke-sub-9f5fa4b3-9jtjn\" (UID: \"9f458ad6-b62f-4f8a-ad43-95f51f2bee4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f5fa4b3-9jtjn" Nov 02 06:37:22.681361 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:22.681337 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f458ad6-b62f-4f8a-ad43-95f51f2bee4c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9f5fa4b3-9jtjn\" (UID: \"9f458ad6-b62f-4f8a-ad43-95f51f2bee4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f5fa4b3-9jtjn" Nov 02 06:37:22.682873 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:22.682853 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f458ad6-b62f-4f8a-ad43-95f51f2bee4c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9f5fa4b3-9jtjn\" (UID: \"9f458ad6-b62f-4f8a-ad43-95f51f2bee4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f5fa4b3-9jtjn" Nov 02 06:37:22.734251 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:22.734215 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdrtf\" (UniqueName: \"kubernetes.io/projected/9f458ad6-b62f-4f8a-ad43-95f51f2bee4c-kube-api-access-vdrtf\") pod \"maas-api-revoke-sub-9f5fa4b3-9jtjn\" (UID: \"9f458ad6-b62f-4f8a-ad43-95f51f2bee4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f5fa4b3-9jtjn" Nov 02 06:37:22.793095 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:22.793019 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f5fa4b3-9jtjn" Nov 02 06:37:23.051166 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:23.051081 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b66e4e9-hpz5x"] Nov 02 06:37:23.054988 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:23.054966 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b66e4e9-hpz5x"] Nov 02 06:37:23.180845 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:37:23.180816 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f458ad6_b62f_4f8a_ad43_95f51f2bee4c.slice/crio-ad5559c47429c9d1ad91be349eced08b00caa1f078ac1346c7316863a4c4f6ac WatchSource:0}: Error finding container ad5559c47429c9d1ad91be349eced08b00caa1f078ac1346c7316863a4c4f6ac: Status 404 returned error can't find the container with id ad5559c47429c9d1ad91be349eced08b00caa1f078ac1346c7316863a4c4f6ac Nov 02 06:37:23.180966 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:23.180933 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f5fa4b3-9jtjn"] Nov 02 06:37:23.263622 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:23.263591 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f5fa4b3-9jtjn" event={"ID":"9f458ad6-b62f-4f8a-ad43-95f51f2bee4c","Type":"ContainerStarted","Data":"ad5559c47429c9d1ad91be349eced08b00caa1f078ac1346c7316863a4c4f6ac"} Nov 02 06:37:23.843195 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:23.843158 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f950069-ac4b-4c86-ab48-eb0496c385f5" path="/var/lib/kubelet/pods/2f950069-ac4b-4c86-ab48-eb0496c385f5/volumes" Nov 02 06:37:23.843612 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:23.843597 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88aedb5c-54d5-4c77-a228-2064e91e9228" path="/var/lib/kubelet/pods/88aedb5c-54d5-4c77-a228-2064e91e9228/volumes" Nov 02 06:37:24.273345 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:24.273311 5916 generic.go:356] "Generic (PLEG): container finished" podID="9f458ad6-b62f-4f8a-ad43-95f51f2bee4c" containerID="4c99d2608b531894bed14e9738827afcfaae33740a6e2061412ca7108625f1f3" exitCode=0 Nov 02 06:37:24.273500 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:24.273394 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f5fa4b3-9jtjn" event={"ID":"9f458ad6-b62f-4f8a-ad43-95f51f2bee4c","Type":"ContainerDied","Data":"4c99d2608b531894bed14e9738827afcfaae33740a6e2061412ca7108625f1f3"} Nov 02 06:37:25.442699 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:25.442674 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f5fa4b3-9jtjn" Nov 02 06:37:25.498808 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:25.498773 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d714765b-npkpv"] Nov 02 06:37:25.499627 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:25.499606 5916 cpu_manager.go:414] "containerMap: removing container" podUID="9f458ad6-b62f-4f8a-ad43-95f51f2bee4c" containerName="revoke-keys" Nov 02 06:37:25.499627 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:25.499627 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f458ad6-b62f-4f8a-ad43-95f51f2bee4c" containerName="revoke-keys" Nov 02 06:37:25.499876 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:25.499862 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f458ad6-b62f-4f8a-ad43-95f51f2bee4c" containerName="revoke-keys" Nov 02 06:37:25.503833 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:25.503814 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d714765b-npkpv" Nov 02 06:37:25.521122 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:25.521101 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f458ad6-b62f-4f8a-ad43-95f51f2bee4c-kube-api-access-vdrtf\" (UniqueName: \"kubernetes.io/projected/9f458ad6-b62f-4f8a-ad43-95f51f2bee4c-kube-api-access-vdrtf\") pod \"9f458ad6-b62f-4f8a-ad43-95f51f2bee4c\" (UID: \"9f458ad6-b62f-4f8a-ad43-95f51f2bee4c\") " Nov 02 06:37:25.521213 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:25.521145 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f458ad6-b62f-4f8a-ad43-95f51f2bee4c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f458ad6-b62f-4f8a-ad43-95f51f2bee4c-maas-api-xks-ca\") pod \"9f458ad6-b62f-4f8a-ad43-95f51f2bee4c\" (UID: \"9f458ad6-b62f-4f8a-ad43-95f51f2bee4c\") " Nov 02 06:37:25.521213 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:25.521196 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f458ad6-b62f-4f8a-ad43-95f51f2bee4c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f458ad6-b62f-4f8a-ad43-95f51f2bee4c-maas-api-openshift-ca\") pod \"9f458ad6-b62f-4f8a-ad43-95f51f2bee4c\" (UID: \"9f458ad6-b62f-4f8a-ad43-95f51f2bee4c\") " Nov 02 06:37:25.521630 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:25.521608 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f458ad6-b62f-4f8a-ad43-95f51f2bee4c-maas-api-openshift-ca" pod "9f458ad6-b62f-4f8a-ad43-95f51f2bee4c" (UID: "9f458ad6-b62f-4f8a-ad43-95f51f2bee4c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:37:25.523148 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:25.523125 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f458ad6-b62f-4f8a-ad43-95f51f2bee4c-maas-api-xks-ca" pod "9f458ad6-b62f-4f8a-ad43-95f51f2bee4c" (UID: "9f458ad6-b62f-4f8a-ad43-95f51f2bee4c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:37:25.523241 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:25.523178 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f458ad6-b62f-4f8a-ad43-95f51f2bee4c-kube-api-access-vdrtf" pod "9f458ad6-b62f-4f8a-ad43-95f51f2bee4c" (UID: "9f458ad6-b62f-4f8a-ad43-95f51f2bee4c"). InnerVolumeSpecName "kube-api-access-vdrtf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:37:25.558770 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:25.558744 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d714765b-npkpv"] Nov 02 06:37:25.622500 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:25.622460 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8tvp\" (UniqueName: \"kubernetes.io/projected/88d99242-0923-4329-bfc5-9d3f2f94c8c7-kube-api-access-b8tvp\") pod \"maas-api-revoke-sub-d714765b-npkpv\" (UID: \"88d99242-0923-4329-bfc5-9d3f2f94c8c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d714765b-npkpv" Nov 02 06:37:25.622780 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:25.622754 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88d99242-0923-4329-bfc5-9d3f2f94c8c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d714765b-npkpv\" (UID: \"88d99242-0923-4329-bfc5-9d3f2f94c8c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d714765b-npkpv" Nov 02 06:37:25.622952 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:25.622801 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88d99242-0923-4329-bfc5-9d3f2f94c8c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d714765b-npkpv\" (UID: \"88d99242-0923-4329-bfc5-9d3f2f94c8c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d714765b-npkpv" Nov 02 06:37:25.623105 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:25.623083 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdrtf\" (UniqueName: \"kubernetes.io/projected/9f458ad6-b62f-4f8a-ad43-95f51f2bee4c-kube-api-access-vdrtf\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:37:25.623166 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:25.623113 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f458ad6-b62f-4f8a-ad43-95f51f2bee4c-maas-api-xks-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:37:25.623166 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:25.623128 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f458ad6-b62f-4f8a-ad43-95f51f2bee4c-maas-api-openshift-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:37:25.724427 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:25.724387 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8tvp\" (UniqueName: \"kubernetes.io/projected/88d99242-0923-4329-bfc5-9d3f2f94c8c7-kube-api-access-b8tvp\") pod \"maas-api-revoke-sub-d714765b-npkpv\" (UID: \"88d99242-0923-4329-bfc5-9d3f2f94c8c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d714765b-npkpv" Nov 02 06:37:25.724427 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:25.724435 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88d99242-0923-4329-bfc5-9d3f2f94c8c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d714765b-npkpv\" (UID: \"88d99242-0923-4329-bfc5-9d3f2f94c8c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d714765b-npkpv" Nov 02 06:37:25.724695 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:25.724461 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88d99242-0923-4329-bfc5-9d3f2f94c8c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d714765b-npkpv\" (UID: \"88d99242-0923-4329-bfc5-9d3f2f94c8c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d714765b-npkpv" Nov 02 06:37:25.725195 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:25.725167 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88d99242-0923-4329-bfc5-9d3f2f94c8c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d714765b-npkpv\" (UID: \"88d99242-0923-4329-bfc5-9d3f2f94c8c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d714765b-npkpv" Nov 02 06:37:25.726859 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:25.726835 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88d99242-0923-4329-bfc5-9d3f2f94c8c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d714765b-npkpv\" (UID: \"88d99242-0923-4329-bfc5-9d3f2f94c8c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d714765b-npkpv" Nov 02 06:37:25.782000 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:25.781970 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8tvp\" (UniqueName: \"kubernetes.io/projected/88d99242-0923-4329-bfc5-9d3f2f94c8c7-kube-api-access-b8tvp\") pod \"maas-api-revoke-sub-d714765b-npkpv\" (UID: \"88d99242-0923-4329-bfc5-9d3f2f94c8c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d714765b-npkpv" Nov 02 06:37:25.821947 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:25.821858 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d714765b-npkpv" Nov 02 06:37:26.052340 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:26.051995 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-184131a0-djz9z"] Nov 02 06:37:26.056929 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:26.056871 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-184131a0-djz9z"] Nov 02 06:37:26.211496 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:26.211466 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d714765b-npkpv"] Nov 02 06:37:26.211960 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:37:26.211935 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88d99242_0923_4329_bfc5_9d3f2f94c8c7.slice/crio-ef08dbed7608e894bd13e6925be59d1a2e2e4e4651f6dd044a21faa1651db3c4 WatchSource:0}: Error finding container ef08dbed7608e894bd13e6925be59d1a2e2e4e4651f6dd044a21faa1651db3c4: Status 404 returned error can't find the container with id ef08dbed7608e894bd13e6925be59d1a2e2e4e4651f6dd044a21faa1651db3c4 Nov 02 06:37:26.292454 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:26.292415 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d714765b-npkpv" event={"ID":"88d99242-0923-4329-bfc5-9d3f2f94c8c7","Type":"ContainerStarted","Data":"ef08dbed7608e894bd13e6925be59d1a2e2e4e4651f6dd044a21faa1651db3c4"} Nov 02 06:37:26.293829 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:26.293807 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f5fa4b3-9jtjn" event={"ID":"9f458ad6-b62f-4f8a-ad43-95f51f2bee4c","Type":"ContainerDied","Data":"ad5559c47429c9d1ad91be349eced08b00caa1f078ac1346c7316863a4c4f6ac"} Nov 02 06:37:26.293958 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:26.293834 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f5fa4b3-9jtjn" Nov 02 06:37:26.293958 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:26.293846 5916 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ad5559c47429c9d1ad91be349eced08b00caa1f078ac1346c7316863a4c4f6ac" Nov 02 06:37:27.303103 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:27.303066 5916 generic.go:356] "Generic (PLEG): container finished" podID="88d99242-0923-4329-bfc5-9d3f2f94c8c7" containerID="26361546f5c3528be1aba8913e67eafa96d258bcdff02495f89c16169f23628e" exitCode=0 Nov 02 06:37:27.303478 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:27.303134 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d714765b-npkpv" event={"ID":"88d99242-0923-4329-bfc5-9d3f2f94c8c7","Type":"ContainerDied","Data":"26361546f5c3528be1aba8913e67eafa96d258bcdff02495f89c16169f23628e"} Nov 02 06:37:27.842980 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:27.842947 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ae95479-786b-4854-987c-d5cf22401220" path="/var/lib/kubelet/pods/1ae95479-786b-4854-987c-d5cf22401220/volumes" Nov 02 06:37:28.473449 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:28.473426 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d714765b-npkpv" Nov 02 06:37:28.558709 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:28.558667 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/88d99242-0923-4329-bfc5-9d3f2f94c8c7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88d99242-0923-4329-bfc5-9d3f2f94c8c7-maas-api-xks-ca\") pod \"88d99242-0923-4329-bfc5-9d3f2f94c8c7\" (UID: \"88d99242-0923-4329-bfc5-9d3f2f94c8c7\") " Nov 02 06:37:28.558957 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:28.558900 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88d99242-0923-4329-bfc5-9d3f2f94c8c7-kube-api-access-b8tvp\" (UniqueName: \"kubernetes.io/projected/88d99242-0923-4329-bfc5-9d3f2f94c8c7-kube-api-access-b8tvp\") pod \"88d99242-0923-4329-bfc5-9d3f2f94c8c7\" (UID: \"88d99242-0923-4329-bfc5-9d3f2f94c8c7\") " Nov 02 06:37:28.559109 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:28.558994 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88d99242-0923-4329-bfc5-9d3f2f94c8c7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88d99242-0923-4329-bfc5-9d3f2f94c8c7-maas-api-openshift-ca\") pod \"88d99242-0923-4329-bfc5-9d3f2f94c8c7\" (UID: \"88d99242-0923-4329-bfc5-9d3f2f94c8c7\") " Nov 02 06:37:28.559762 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:28.559665 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88d99242-0923-4329-bfc5-9d3f2f94c8c7-maas-api-openshift-ca" pod "88d99242-0923-4329-bfc5-9d3f2f94c8c7" (UID: "88d99242-0923-4329-bfc5-9d3f2f94c8c7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:37:28.561126 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:28.561077 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88d99242-0923-4329-bfc5-9d3f2f94c8c7-maas-api-xks-ca" pod "88d99242-0923-4329-bfc5-9d3f2f94c8c7" (UID: "88d99242-0923-4329-bfc5-9d3f2f94c8c7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:37:28.562305 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:28.562280 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88d99242-0923-4329-bfc5-9d3f2f94c8c7-kube-api-access-b8tvp" pod "88d99242-0923-4329-bfc5-9d3f2f94c8c7" (UID: "88d99242-0923-4329-bfc5-9d3f2f94c8c7"). InnerVolumeSpecName "kube-api-access-b8tvp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:37:28.660173 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:28.660138 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88d99242-0923-4329-bfc5-9d3f2f94c8c7-maas-api-xks-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:37:28.660173 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:28.660168 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8tvp\" (UniqueName: \"kubernetes.io/projected/88d99242-0923-4329-bfc5-9d3f2f94c8c7-kube-api-access-b8tvp\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:37:28.660173 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:28.660179 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88d99242-0923-4329-bfc5-9d3f2f94c8c7-maas-api-openshift-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:37:29.322442 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:29.322400 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d714765b-npkpv" event={"ID":"88d99242-0923-4329-bfc5-9d3f2f94c8c7","Type":"ContainerDied","Data":"ef08dbed7608e894bd13e6925be59d1a2e2e4e4651f6dd044a21faa1651db3c4"} Nov 02 06:37:29.322635 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:29.322466 5916 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ef08dbed7608e894bd13e6925be59d1a2e2e4e4651f6dd044a21faa1651db3c4" Nov 02 06:37:29.322635 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:29.322414 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d714765b-npkpv" Nov 02 06:37:31.919893 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:31.919858 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-br7cd"] Nov 02 06:37:31.920406 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:31.920392 5916 cpu_manager.go:414] "containerMap: removing container" podUID="88d99242-0923-4329-bfc5-9d3f2f94c8c7" containerName="revoke-keys" Nov 02 06:37:31.920465 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:31.920407 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88d99242-0923-4329-bfc5-9d3f2f94c8c7" containerName="revoke-keys" Nov 02 06:37:31.920533 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:31.920523 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="88d99242-0923-4329-bfc5-9d3f2f94c8c7" containerName="revoke-keys" Nov 02 06:37:31.923783 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:31.923765 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-br7cd" Nov 02 06:37:32.001013 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:32.000980 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f9273e5-b930-47f0-b710-234974ad9025-istio-envoy\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-br7cd\" (UID: \"2f9273e5-b930-47f0-b710-234974ad9025\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-br7cd" Nov 02 06:37:32.001013 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:32.001016 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f9273e5-b930-47f0-b710-234974ad9025-istio-token\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-br7cd\" (UID: \"2f9273e5-b930-47f0-b710-234974ad9025\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-br7cd" Nov 02 06:37:32.001216 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:32.001194 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f9273e5-b930-47f0-b710-234974ad9025-workload-socket\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-br7cd\" (UID: \"2f9273e5-b930-47f0-b710-234974ad9025\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-br7cd" Nov 02 06:37:32.001363 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:32.001340 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f9273e5-b930-47f0-b710-234974ad9025-workload-certs\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-br7cd\" (UID: \"2f9273e5-b930-47f0-b710-234974ad9025\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-br7cd" Nov 02 06:37:32.001458 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:32.001440 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f9273e5-b930-47f0-b710-234974ad9025-istiod-ca-cert\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-br7cd\" (UID: \"2f9273e5-b930-47f0-b710-234974ad9025\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-br7cd" Nov 02 06:37:32.001518 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:32.001474 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcxvj\" (UniqueName: \"kubernetes.io/projected/2f9273e5-b930-47f0-b710-234974ad9025-kube-api-access-hcxvj\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-br7cd\" (UID: \"2f9273e5-b930-47f0-b710-234974ad9025\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-br7cd" Nov 02 06:37:32.001608 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:32.001585 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f9273e5-b930-47f0-b710-234974ad9025-istio-data\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-br7cd\" (UID: \"2f9273e5-b930-47f0-b710-234974ad9025\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-br7cd" Nov 02 06:37:32.001678 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:32.001663 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f9273e5-b930-47f0-b710-234974ad9025-credential-socket\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-br7cd\" (UID: \"2f9273e5-b930-47f0-b710-234974ad9025\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-br7cd" Nov 02 06:37:32.001717 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:32.001697 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f9273e5-b930-47f0-b710-234974ad9025-istio-podinfo\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-br7cd\" (UID: \"2f9273e5-b930-47f0-b710-234974ad9025\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-br7cd" Nov 02 06:37:32.019974 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:32.019947 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5636d7f-vvq8s"] Nov 02 06:37:32.024623 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:32.024602 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5636d7f-vvq8s" Nov 02 06:37:32.035477 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:32.035448 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-br7cd"] Nov 02 06:37:32.085104 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:32.085071 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 06:37:32.085488 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:32.085466 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-68l4r\"" Nov 02 06:37:32.091116 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:32.091089 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5636d7f-vvq8s"] Nov 02 06:37:32.102870 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:32.102836 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f9273e5-b930-47f0-b710-234974ad9025-workload-certs\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-br7cd\" (UID: \"2f9273e5-b930-47f0-b710-234974ad9025\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-br7cd" Nov 02 06:37:32.103050 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:32.102924 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f9273e5-b930-47f0-b710-234974ad9025-istiod-ca-cert\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-br7cd\" (UID: \"2f9273e5-b930-47f0-b710-234974ad9025\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-br7cd" Nov 02 06:37:32.103050 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:32.102952 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcxvj\" (UniqueName: \"kubernetes.io/projected/2f9273e5-b930-47f0-b710-234974ad9025-kube-api-access-hcxvj\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-br7cd\" (UID: \"2f9273e5-b930-47f0-b710-234974ad9025\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-br7cd" Nov 02 06:37:32.103050 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:32.102989 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82b7b98c-5ebb-4517-97a1-e8f58413243c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b5636d7f-vvq8s\" (UID: \"82b7b98c-5ebb-4517-97a1-e8f58413243c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5636d7f-vvq8s" Nov 02 06:37:32.103050 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:32.103046 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f9273e5-b930-47f0-b710-234974ad9025-istio-data\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-br7cd\" (UID: \"2f9273e5-b930-47f0-b710-234974ad9025\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-br7cd" Nov 02 06:37:32.103293 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:32.103101 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqsk5\" (UniqueName: \"kubernetes.io/projected/82b7b98c-5ebb-4517-97a1-e8f58413243c-kube-api-access-pqsk5\") pod \"maas-api-revoke-sub-b5636d7f-vvq8s\" (UID: \"82b7b98c-5ebb-4517-97a1-e8f58413243c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5636d7f-vvq8s" Nov 02 06:37:32.103293 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:32.103131 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f9273e5-b930-47f0-b710-234974ad9025-credential-socket\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-br7cd\" (UID: \"2f9273e5-b930-47f0-b710-234974ad9025\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-br7cd" Nov 02 06:37:32.103293 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:32.103160 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f9273e5-b930-47f0-b710-234974ad9025-istio-podinfo\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-br7cd\" (UID: \"2f9273e5-b930-47f0-b710-234974ad9025\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-br7cd" Nov 02 06:37:32.103293 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:32.103253 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f9273e5-b930-47f0-b710-234974ad9025-istio-envoy\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-br7cd\" (UID: \"2f9273e5-b930-47f0-b710-234974ad9025\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-br7cd" Nov 02 06:37:32.103293 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:32.103277 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f9273e5-b930-47f0-b710-234974ad9025-istio-token\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-br7cd\" (UID: \"2f9273e5-b930-47f0-b710-234974ad9025\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-br7cd" Nov 02 06:37:32.103533 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:32.103320 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f9273e5-b930-47f0-b710-234974ad9025-workload-certs\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-br7cd\" (UID: \"2f9273e5-b930-47f0-b710-234974ad9025\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-br7cd" Nov 02 06:37:32.103533 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:32.103332 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f9273e5-b930-47f0-b710-234974ad9025-workload-socket\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-br7cd\" (UID: \"2f9273e5-b930-47f0-b710-234974ad9025\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-br7cd" Nov 02 06:37:32.103533 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:32.103422 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82b7b98c-5ebb-4517-97a1-e8f58413243c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b5636d7f-vvq8s\" (UID: \"82b7b98c-5ebb-4517-97a1-e8f58413243c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5636d7f-vvq8s" Nov 02 06:37:32.103678 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:32.103605 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f9273e5-b930-47f0-b710-234974ad9025-workload-socket\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-br7cd\" (UID: \"2f9273e5-b930-47f0-b710-234974ad9025\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-br7cd" Nov 02 06:37:32.103875 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:32.103852 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f9273e5-b930-47f0-b710-234974ad9025-credential-socket\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-br7cd\" (UID: \"2f9273e5-b930-47f0-b710-234974ad9025\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-br7cd" Nov 02 06:37:32.104212 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:32.104189 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f9273e5-b930-47f0-b710-234974ad9025-istiod-ca-cert\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-br7cd\" (UID: \"2f9273e5-b930-47f0-b710-234974ad9025\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-br7cd" Nov 02 06:37:32.104754 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:32.104729 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f9273e5-b930-47f0-b710-234974ad9025-istio-data\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-br7cd\" (UID: \"2f9273e5-b930-47f0-b710-234974ad9025\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-br7cd" Nov 02 06:37:32.107165 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:32.107138 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f9273e5-b930-47f0-b710-234974ad9025-istio-envoy\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-br7cd\" (UID: \"2f9273e5-b930-47f0-b710-234974ad9025\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-br7cd" Nov 02 06:37:32.107256 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:32.107225 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f9273e5-b930-47f0-b710-234974ad9025-istio-podinfo\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-br7cd\" (UID: \"2f9273e5-b930-47f0-b710-234974ad9025\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-br7cd" Nov 02 06:37:32.162790 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:32.162755 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f9273e5-b930-47f0-b710-234974ad9025-istio-token\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-br7cd\" (UID: \"2f9273e5-b930-47f0-b710-234974ad9025\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-br7cd" Nov 02 06:37:32.164014 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:32.163996 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcxvj\" (UniqueName: \"kubernetes.io/projected/2f9273e5-b930-47f0-b710-234974ad9025-kube-api-access-hcxvj\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-br7cd\" (UID: \"2f9273e5-b930-47f0-b710-234974ad9025\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-br7cd" Nov 02 06:37:32.204570 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:32.204488 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82b7b98c-5ebb-4517-97a1-e8f58413243c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b5636d7f-vvq8s\" (UID: \"82b7b98c-5ebb-4517-97a1-e8f58413243c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5636d7f-vvq8s" Nov 02 06:37:32.204708 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:32.204568 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqsk5\" (UniqueName: \"kubernetes.io/projected/82b7b98c-5ebb-4517-97a1-e8f58413243c-kube-api-access-pqsk5\") pod \"maas-api-revoke-sub-b5636d7f-vvq8s\" (UID: \"82b7b98c-5ebb-4517-97a1-e8f58413243c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5636d7f-vvq8s" Nov 02 06:37:32.204759 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:32.204711 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82b7b98c-5ebb-4517-97a1-e8f58413243c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b5636d7f-vvq8s\" (UID: \"82b7b98c-5ebb-4517-97a1-e8f58413243c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5636d7f-vvq8s" Nov 02 06:37:32.205339 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:32.205316 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82b7b98c-5ebb-4517-97a1-e8f58413243c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b5636d7f-vvq8s\" (UID: \"82b7b98c-5ebb-4517-97a1-e8f58413243c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5636d7f-vvq8s" Nov 02 06:37:32.206838 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:32.206816 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82b7b98c-5ebb-4517-97a1-e8f58413243c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b5636d7f-vvq8s\" (UID: \"82b7b98c-5ebb-4517-97a1-e8f58413243c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5636d7f-vvq8s" Nov 02 06:37:32.238417 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:32.238386 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-br7cd" Nov 02 06:37:32.263157 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:32.263126 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqsk5\" (UniqueName: \"kubernetes.io/projected/82b7b98c-5ebb-4517-97a1-e8f58413243c-kube-api-access-pqsk5\") pod \"maas-api-revoke-sub-b5636d7f-vvq8s\" (UID: \"82b7b98c-5ebb-4517-97a1-e8f58413243c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5636d7f-vvq8s" Nov 02 06:37:32.355740 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:32.355471 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5636d7f-vvq8s" Nov 02 06:37:32.356396 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:32.356368 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-rvnb6"] Nov 02 06:37:32.364550 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:32.363070 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-rvnb6" Nov 02 06:37:32.421380 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:32.421335 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-rvnb6"] Nov 02 06:37:32.511219 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:32.511186 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d4ba27f-3588-4c0b-9446-700fcff8bb14-workload-socket\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-rvnb6\" (UID: \"0d4ba27f-3588-4c0b-9446-700fcff8bb14\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-rvnb6" Nov 02 06:37:32.511384 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:32.511230 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d4ba27f-3588-4c0b-9446-700fcff8bb14-workload-certs\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-rvnb6\" (UID: \"0d4ba27f-3588-4c0b-9446-700fcff8bb14\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-rvnb6" Nov 02 06:37:32.511384 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:32.511359 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d4ba27f-3588-4c0b-9446-700fcff8bb14-istio-podinfo\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-rvnb6\" (UID: \"0d4ba27f-3588-4c0b-9446-700fcff8bb14\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-rvnb6" Nov 02 06:37:32.511457 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:32.511438 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d4ba27f-3588-4c0b-9446-700fcff8bb14-credential-socket\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-rvnb6\" (UID: \"0d4ba27f-3588-4c0b-9446-700fcff8bb14\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-rvnb6" Nov 02 06:37:32.511538 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:32.511520 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d4ba27f-3588-4c0b-9446-700fcff8bb14-istiod-ca-cert\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-rvnb6\" (UID: \"0d4ba27f-3588-4c0b-9446-700fcff8bb14\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-rvnb6" Nov 02 06:37:32.511571 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:32.511554 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zm9gf\" (UniqueName: \"kubernetes.io/projected/0d4ba27f-3588-4c0b-9446-700fcff8bb14-kube-api-access-zm9gf\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-rvnb6\" (UID: \"0d4ba27f-3588-4c0b-9446-700fcff8bb14\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-rvnb6" Nov 02 06:37:32.511633 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:32.511618 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d4ba27f-3588-4c0b-9446-700fcff8bb14-istio-token\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-rvnb6\" (UID: \"0d4ba27f-3588-4c0b-9446-700fcff8bb14\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-rvnb6" Nov 02 06:37:32.511723 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:32.511705 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d4ba27f-3588-4c0b-9446-700fcff8bb14-istio-data\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-rvnb6\" (UID: \"0d4ba27f-3588-4c0b-9446-700fcff8bb14\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-rvnb6" Nov 02 06:37:32.511787 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:32.511770 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d4ba27f-3588-4c0b-9446-700fcff8bb14-istio-envoy\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-rvnb6\" (UID: \"0d4ba27f-3588-4c0b-9446-700fcff8bb14\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-rvnb6" Nov 02 06:37:32.612960 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:32.612923 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d4ba27f-3588-4c0b-9446-700fcff8bb14-istio-token\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-rvnb6\" (UID: \"0d4ba27f-3588-4c0b-9446-700fcff8bb14\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-rvnb6" Nov 02 06:37:32.613157 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:32.612983 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d4ba27f-3588-4c0b-9446-700fcff8bb14-istio-data\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-rvnb6\" (UID: \"0d4ba27f-3588-4c0b-9446-700fcff8bb14\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-rvnb6" Nov 02 06:37:32.613157 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:32.613023 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d4ba27f-3588-4c0b-9446-700fcff8bb14-istio-envoy\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-rvnb6\" (UID: \"0d4ba27f-3588-4c0b-9446-700fcff8bb14\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-rvnb6" Nov 02 06:37:32.613157 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:32.613080 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d4ba27f-3588-4c0b-9446-700fcff8bb14-workload-socket\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-rvnb6\" (UID: \"0d4ba27f-3588-4c0b-9446-700fcff8bb14\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-rvnb6" Nov 02 06:37:32.613157 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:32.613105 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d4ba27f-3588-4c0b-9446-700fcff8bb14-workload-certs\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-rvnb6\" (UID: \"0d4ba27f-3588-4c0b-9446-700fcff8bb14\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-rvnb6" Nov 02 06:37:32.613157 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:32.613157 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d4ba27f-3588-4c0b-9446-700fcff8bb14-istio-podinfo\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-rvnb6\" (UID: \"0d4ba27f-3588-4c0b-9446-700fcff8bb14\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-rvnb6" Nov 02 06:37:32.613424 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:32.613225 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d4ba27f-3588-4c0b-9446-700fcff8bb14-credential-socket\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-rvnb6\" (UID: \"0d4ba27f-3588-4c0b-9446-700fcff8bb14\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-rvnb6" Nov 02 06:37:32.613424 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:32.613295 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d4ba27f-3588-4c0b-9446-700fcff8bb14-istiod-ca-cert\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-rvnb6\" (UID: \"0d4ba27f-3588-4c0b-9446-700fcff8bb14\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-rvnb6" Nov 02 06:37:32.613424 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:32.613327 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zm9gf\" (UniqueName: \"kubernetes.io/projected/0d4ba27f-3588-4c0b-9446-700fcff8bb14-kube-api-access-zm9gf\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-rvnb6\" (UID: \"0d4ba27f-3588-4c0b-9446-700fcff8bb14\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-rvnb6" Nov 02 06:37:32.613579 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:32.613553 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d4ba27f-3588-4c0b-9446-700fcff8bb14-istio-data\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-rvnb6\" (UID: \"0d4ba27f-3588-4c0b-9446-700fcff8bb14\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-rvnb6" Nov 02 06:37:32.613648 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:32.613617 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d4ba27f-3588-4c0b-9446-700fcff8bb14-workload-socket\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-rvnb6\" (UID: \"0d4ba27f-3588-4c0b-9446-700fcff8bb14\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-rvnb6" Nov 02 06:37:32.613706 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:32.613661 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d4ba27f-3588-4c0b-9446-700fcff8bb14-workload-certs\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-rvnb6\" (UID: \"0d4ba27f-3588-4c0b-9446-700fcff8bb14\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-rvnb6" Nov 02 06:37:32.613845 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:32.613818 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d4ba27f-3588-4c0b-9446-700fcff8bb14-credential-socket\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-rvnb6\" (UID: \"0d4ba27f-3588-4c0b-9446-700fcff8bb14\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-rvnb6" Nov 02 06:37:32.614100 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:32.614063 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d4ba27f-3588-4c0b-9446-700fcff8bb14-istiod-ca-cert\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-rvnb6\" (UID: \"0d4ba27f-3588-4c0b-9446-700fcff8bb14\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-rvnb6" Nov 02 06:37:32.615517 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:32.615496 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d4ba27f-3588-4c0b-9446-700fcff8bb14-istio-envoy\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-rvnb6\" (UID: \"0d4ba27f-3588-4c0b-9446-700fcff8bb14\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-rvnb6" Nov 02 06:37:32.616124 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:32.616107 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d4ba27f-3588-4c0b-9446-700fcff8bb14-istio-podinfo\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-rvnb6\" (UID: \"0d4ba27f-3588-4c0b-9446-700fcff8bb14\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-rvnb6" Nov 02 06:37:32.670202 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:32.670163 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d4ba27f-3588-4c0b-9446-700fcff8bb14-istio-token\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-rvnb6\" (UID: \"0d4ba27f-3588-4c0b-9446-700fcff8bb14\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-rvnb6" Nov 02 06:37:32.670402 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:32.670331 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zm9gf\" (UniqueName: \"kubernetes.io/projected/0d4ba27f-3588-4c0b-9446-700fcff8bb14-kube-api-access-zm9gf\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-rvnb6\" (UID: \"0d4ba27f-3588-4c0b-9446-700fcff8bb14\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-rvnb6" Nov 02 06:37:32.678767 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:37:32.678741 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f9273e5_b930_47f0_b710_234974ad9025.slice/crio-52835324304d8b359c2ffc8a55510eac0121d083d7ba878086125635846c2ee6 WatchSource:0}: Error finding container 52835324304d8b359c2ffc8a55510eac0121d083d7ba878086125635846c2ee6: Status 404 returned error can't find the container with id 52835324304d8b359c2ffc8a55510eac0121d083d7ba878086125635846c2ee6 Nov 02 06:37:32.678977 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:32.678944 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-br7cd"] Nov 02 06:37:32.681152 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:32.681122 5916 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 06:37:32.681236 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:32.681216 5916 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 06:37:32.681309 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:32.681262 5916 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 06:37:32.685753 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:32.685735 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-rvnb6" Nov 02 06:37:32.776071 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:32.773815 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5636d7f-vvq8s"] Nov 02 06:37:32.816937 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:32.816880 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-x2qpj"] Nov 02 06:37:32.822982 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:32.822959 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-x2qpj" Nov 02 06:37:32.882943 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:32.882142 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-x2qpj"] Nov 02 06:37:32.917405 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:32.917372 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33373aea-62e6-4fa4-870d-707c3e7bc858-istiod-ca-cert\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-x2qpj\" (UID: \"33373aea-62e6-4fa4-870d-707c3e7bc858\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-x2qpj" Nov 02 06:37:32.917596 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:32.917498 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33373aea-62e6-4fa4-870d-707c3e7bc858-credential-socket\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-x2qpj\" (UID: \"33373aea-62e6-4fa4-870d-707c3e7bc858\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-x2qpj" Nov 02 06:37:32.917596 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:32.917549 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33373aea-62e6-4fa4-870d-707c3e7bc858-istio-token\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-x2qpj\" (UID: \"33373aea-62e6-4fa4-870d-707c3e7bc858\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-x2qpj" Nov 02 06:37:32.917596 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:32.917576 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bn7pw\" (UniqueName: \"kubernetes.io/projected/33373aea-62e6-4fa4-870d-707c3e7bc858-kube-api-access-bn7pw\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-x2qpj\" (UID: \"33373aea-62e6-4fa4-870d-707c3e7bc858\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-x2qpj" Nov 02 06:37:32.917731 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:32.917695 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33373aea-62e6-4fa4-870d-707c3e7bc858-workload-socket\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-x2qpj\" (UID: \"33373aea-62e6-4fa4-870d-707c3e7bc858\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-x2qpj" Nov 02 06:37:32.918003 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:32.917985 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33373aea-62e6-4fa4-870d-707c3e7bc858-istio-podinfo\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-x2qpj\" (UID: \"33373aea-62e6-4fa4-870d-707c3e7bc858\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-x2qpj" Nov 02 06:37:32.918196 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:32.918181 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33373aea-62e6-4fa4-870d-707c3e7bc858-workload-certs\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-x2qpj\" (UID: \"33373aea-62e6-4fa4-870d-707c3e7bc858\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-x2qpj" Nov 02 06:37:32.918462 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:32.918444 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33373aea-62e6-4fa4-870d-707c3e7bc858-istio-envoy\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-x2qpj\" (UID: \"33373aea-62e6-4fa4-870d-707c3e7bc858\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-x2qpj" Nov 02 06:37:32.918621 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:32.918605 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33373aea-62e6-4fa4-870d-707c3e7bc858-istio-data\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-x2qpj\" (UID: \"33373aea-62e6-4fa4-870d-707c3e7bc858\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-x2qpj" Nov 02 06:37:33.019495 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:33.019413 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33373aea-62e6-4fa4-870d-707c3e7bc858-credential-socket\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-x2qpj\" (UID: \"33373aea-62e6-4fa4-870d-707c3e7bc858\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-x2qpj" Nov 02 06:37:33.019495 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:33.019462 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33373aea-62e6-4fa4-870d-707c3e7bc858-istio-token\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-x2qpj\" (UID: \"33373aea-62e6-4fa4-870d-707c3e7bc858\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-x2qpj" Nov 02 06:37:33.019495 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:33.019492 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bn7pw\" (UniqueName: \"kubernetes.io/projected/33373aea-62e6-4fa4-870d-707c3e7bc858-kube-api-access-bn7pw\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-x2qpj\" (UID: \"33373aea-62e6-4fa4-870d-707c3e7bc858\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-x2qpj" Nov 02 06:37:33.020079 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:33.019531 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33373aea-62e6-4fa4-870d-707c3e7bc858-workload-socket\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-x2qpj\" (UID: \"33373aea-62e6-4fa4-870d-707c3e7bc858\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-x2qpj" Nov 02 06:37:33.020079 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:33.019594 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33373aea-62e6-4fa4-870d-707c3e7bc858-istio-podinfo\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-x2qpj\" (UID: \"33373aea-62e6-4fa4-870d-707c3e7bc858\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-x2qpj" Nov 02 06:37:33.020079 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:33.019632 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33373aea-62e6-4fa4-870d-707c3e7bc858-workload-certs\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-x2qpj\" (UID: \"33373aea-62e6-4fa4-870d-707c3e7bc858\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-x2qpj" Nov 02 06:37:33.020079 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:33.019764 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33373aea-62e6-4fa4-870d-707c3e7bc858-istio-envoy\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-x2qpj\" (UID: \"33373aea-62e6-4fa4-870d-707c3e7bc858\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-x2qpj" Nov 02 06:37:33.020079 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:33.019819 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33373aea-62e6-4fa4-870d-707c3e7bc858-istio-data\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-x2qpj\" (UID: \"33373aea-62e6-4fa4-870d-707c3e7bc858\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-x2qpj" Nov 02 06:37:33.020079 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:33.019857 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33373aea-62e6-4fa4-870d-707c3e7bc858-istiod-ca-cert\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-x2qpj\" (UID: \"33373aea-62e6-4fa4-870d-707c3e7bc858\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-x2qpj" Nov 02 06:37:33.020079 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:33.020032 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33373aea-62e6-4fa4-870d-707c3e7bc858-credential-socket\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-x2qpj\" (UID: \"33373aea-62e6-4fa4-870d-707c3e7bc858\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-x2qpj" Nov 02 06:37:33.020450 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:33.020138 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33373aea-62e6-4fa4-870d-707c3e7bc858-workload-socket\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-x2qpj\" (UID: \"33373aea-62e6-4fa4-870d-707c3e7bc858\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-x2qpj" Nov 02 06:37:33.020665 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:33.020643 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33373aea-62e6-4fa4-870d-707c3e7bc858-istiod-ca-cert\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-x2qpj\" (UID: \"33373aea-62e6-4fa4-870d-707c3e7bc858\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-x2qpj" Nov 02 06:37:33.020789 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:33.020773 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33373aea-62e6-4fa4-870d-707c3e7bc858-istio-data\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-x2qpj\" (UID: \"33373aea-62e6-4fa4-870d-707c3e7bc858\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-x2qpj" Nov 02 06:37:33.020850 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:33.020820 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33373aea-62e6-4fa4-870d-707c3e7bc858-workload-certs\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-x2qpj\" (UID: \"33373aea-62e6-4fa4-870d-707c3e7bc858\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-x2qpj" Nov 02 06:37:33.023312 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:33.023290 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33373aea-62e6-4fa4-870d-707c3e7bc858-istio-envoy\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-x2qpj\" (UID: \"33373aea-62e6-4fa4-870d-707c3e7bc858\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-x2qpj" Nov 02 06:37:33.023611 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:33.023590 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33373aea-62e6-4fa4-870d-707c3e7bc858-istio-podinfo\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-x2qpj\" (UID: \"33373aea-62e6-4fa4-870d-707c3e7bc858\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-x2qpj" Nov 02 06:37:33.051926 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:33.051018 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e2c401b-drbp6"] Nov 02 06:37:33.055766 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:33.055737 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e2c401b-drbp6"] Nov 02 06:37:33.078291 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:33.078196 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-rvnb6"] Nov 02 06:37:33.078363 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:37:33.078309 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d4ba27f_3588_4c0b_9446_700fcff8bb14.slice/crio-a530747df764043f66602972d8730601929abb60f7205b4d0880216fd5cac3c8 WatchSource:0}: Error finding container a530747df764043f66602972d8730601929abb60f7205b4d0880216fd5cac3c8: Status 404 returned error can't find the container with id a530747df764043f66602972d8730601929abb60f7205b4d0880216fd5cac3c8 Nov 02 06:37:33.080096 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:33.080074 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33373aea-62e6-4fa4-870d-707c3e7bc858-istio-token\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-x2qpj\" (UID: \"33373aea-62e6-4fa4-870d-707c3e7bc858\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-x2qpj" Nov 02 06:37:33.080271 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:33.080251 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bn7pw\" (UniqueName: \"kubernetes.io/projected/33373aea-62e6-4fa4-870d-707c3e7bc858-kube-api-access-bn7pw\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-x2qpj\" (UID: \"33373aea-62e6-4fa4-870d-707c3e7bc858\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-x2qpj" Nov 02 06:37:33.080489 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:33.080461 5916 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 06:37:33.080553 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:33.080518 5916 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 06:37:33.080553 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:33.080547 5916 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 06:37:33.153003 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:33.152980 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-x2qpj" Nov 02 06:37:33.381748 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:33.381570 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5636d7f-vvq8s" event={"ID":"82b7b98c-5ebb-4517-97a1-e8f58413243c","Type":"ContainerStarted","Data":"b77e455e57f5140e7beb47f436ffafa57a91a986d08b5a46ce04a85ed38ac1eb"} Nov 02 06:37:33.381748 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:33.381670 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5636d7f-vvq8s" event={"ID":"82b7b98c-5ebb-4517-97a1-e8f58413243c","Type":"ContainerStarted","Data":"430e737698b284de8e48821c120d8280813225c70fbff320d595f77ad2d34db5"} Nov 02 06:37:33.383645 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:33.383614 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-rvnb6" event={"ID":"0d4ba27f-3588-4c0b-9446-700fcff8bb14","Type":"ContainerStarted","Data":"570465c34d626fe369b828e443e8a02d668daa6d71144a85af9c6112d43a4493"} Nov 02 06:37:33.383769 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:33.383667 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-rvnb6" event={"ID":"0d4ba27f-3588-4c0b-9446-700fcff8bb14","Type":"ContainerStarted","Data":"a530747df764043f66602972d8730601929abb60f7205b4d0880216fd5cac3c8"} Nov 02 06:37:33.385317 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:33.385294 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-br7cd" event={"ID":"2f9273e5-b930-47f0-b710-234974ad9025","Type":"ContainerStarted","Data":"d1262d9a4b500b35cab1feee8b0155366102b0d2bd27772791f5ee3824a45811"} Nov 02 06:37:33.385435 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:33.385335 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-br7cd" event={"ID":"2f9273e5-b930-47f0-b710-234974ad9025","Type":"ContainerStarted","Data":"52835324304d8b359c2ffc8a55510eac0121d083d7ba878086125635846c2ee6"} Nov 02 06:37:33.499332 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:33.499281 5916 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5636d7f-vvq8s" podStartSLOduration=2.499267153 podStartE2EDuration="2.499267153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:37:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:37:33.49496595 +0000 UTC m=+1241.766786345" watchObservedRunningTime="2026-11-02 06:37:33.499267153 +0000 UTC m=+1241.771087610" Nov 02 06:37:33.565455 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:33.565412 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-x2qpj"] Nov 02 06:37:33.579617 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:37:33.579582 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33373aea_62e6_4fa4_870d_707c3e7bc858.slice/crio-8b0b5b283c8dfec362e0b0c3be2f6486cd97f92bf8b6b2acd4849d891613b462 WatchSource:0}: Error finding container 8b0b5b283c8dfec362e0b0c3be2f6486cd97f92bf8b6b2acd4849d891613b462: Status 404 returned error can't find the container with id 8b0b5b283c8dfec362e0b0c3be2f6486cd97f92bf8b6b2acd4849d891613b462 Nov 02 06:37:33.582401 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:33.582360 5916 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 06:37:33.582511 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:33.582448 5916 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 06:37:33.582511 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:33.582487 5916 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 06:37:33.618797 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:33.618746 5916 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-rvnb6" podStartSLOduration=1.618730193 podStartE2EDuration="1.618730193s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:37:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:37:33.613561601 +0000 UTC m=+1241.885381996" watchObservedRunningTime="2026-11-02 06:37:33.618730193 +0000 UTC m=+1241.890550587" Nov 02 06:37:33.686551 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:33.686501 5916 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-rvnb6" Nov 02 06:37:33.768039 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:33.764183 5916 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-br7cd" podStartSLOduration=2.764159547 podStartE2EDuration="2.764159547s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:37:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:37:33.731029923 +0000 UTC m=+1242.002850321" watchObservedRunningTime="2026-11-02 06:37:33.764159547 +0000 UTC m=+1242.035979942" Nov 02 06:37:33.867935 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:33.867281 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12e9b5d0-00a7-4b96-9a9a-d403c5bae27c" path="/var/lib/kubelet/pods/12e9b5d0-00a7-4b96-9a9a-d403c5bae27c/volumes" Nov 02 06:37:34.246939 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:34.245955 5916 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-br7cd" Nov 02 06:37:34.268950 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:34.264041 5916 patch_prober.go:29] interesting pod/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-br7cd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Nov 02 06:37:34.268950 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:34.264111 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-br7cd" podUID="2f9273e5-b930-47f0-b710-234974ad9025" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Nov 02 06:37:34.410336 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:34.410294 5916 generic.go:356] "Generic (PLEG): container finished" podID="82b7b98c-5ebb-4517-97a1-e8f58413243c" containerID="b77e455e57f5140e7beb47f436ffafa57a91a986d08b5a46ce04a85ed38ac1eb" exitCode=0 Nov 02 06:37:34.410608 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:34.410579 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5636d7f-vvq8s" event={"ID":"82b7b98c-5ebb-4517-97a1-e8f58413243c","Type":"ContainerDied","Data":"b77e455e57f5140e7beb47f436ffafa57a91a986d08b5a46ce04a85ed38ac1eb"} Nov 02 06:37:34.423983 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:34.422202 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-x2qpj" event={"ID":"33373aea-62e6-4fa4-870d-707c3e7bc858","Type":"ContainerStarted","Data":"970c1dad82dd817e41dd56fb156bf90e0007567dcd40ff9c2b89cac5ad991715"} Nov 02 06:37:34.427227 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:34.427197 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-x2qpj" event={"ID":"33373aea-62e6-4fa4-870d-707c3e7bc858","Type":"ContainerStarted","Data":"8b0b5b283c8dfec362e0b0c3be2f6486cd97f92bf8b6b2acd4849d891613b462"} Nov 02 06:37:34.689931 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:34.689778 5916 patch_prober.go:29] interesting pod/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-rvnb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Nov 02 06:37:34.689931 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:34.689840 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-rvnb6" podUID="0d4ba27f-3588-4c0b-9446-700fcff8bb14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Nov 02 06:37:35.162929 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:35.162095 5916 patch_prober.go:29] interesting pod/e2e-worker-w6-6ff85e-openshift-default-698585659c-x2qpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Nov 02 06:37:35.162929 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:35.162159 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-x2qpj" podUID="33373aea-62e6-4fa4-870d-707c3e7bc858" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Nov 02 06:37:35.162929 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:35.162417 5916 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-x2qpj" Nov 02 06:37:35.246002 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:35.245966 5916 patch_prober.go:29] interesting pod/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-br7cd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Nov 02 06:37:35.246273 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:35.246247 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-br7cd" podUID="2f9273e5-b930-47f0-b710-234974ad9025" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Nov 02 06:37:35.688348 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:35.688300 5916 patch_prober.go:29] interesting pod/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-rvnb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Nov 02 06:37:35.688977 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:35.688941 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-rvnb6" podUID="0d4ba27f-3588-4c0b-9446-700fcff8bb14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Nov 02 06:37:36.164958 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:36.162032 5916 patch_prober.go:29] interesting pod/e2e-worker-w6-6ff85e-openshift-default-698585659c-x2qpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Nov 02 06:37:36.164958 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:36.162091 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-x2qpj" podUID="33373aea-62e6-4fa4-870d-707c3e7bc858" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Nov 02 06:37:36.278950 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:36.276106 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5636d7f-vvq8s" Nov 02 06:37:36.300240 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:36.294607 5916 patch_prober.go:29] interesting pod/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-br7cd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Nov 02 06:37:36.300240 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:36.294683 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-br7cd" podUID="2f9273e5-b930-47f0-b710-234974ad9025" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Nov 02 06:37:36.330929 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:36.327147 5916 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-x2qpj" podStartSLOduration=4.327126239 podStartE2EDuration="4.327126239s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:37:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:37:34.671298041 +0000 UTC m=+1242.943118435" watchObservedRunningTime="2026-11-02 06:37:36.327126239 +0000 UTC m=+1244.598946635" Nov 02 06:37:36.379931 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:36.379449 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a40f9931-lfvwh"] Nov 02 06:37:36.393925 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:36.380390 5916 cpu_manager.go:414] "containerMap: removing container" podUID="82b7b98c-5ebb-4517-97a1-e8f58413243c" containerName="revoke-keys" Nov 02 06:37:36.393925 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:36.380416 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82b7b98c-5ebb-4517-97a1-e8f58413243c" containerName="revoke-keys" Nov 02 06:37:36.393925 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:36.380623 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="82b7b98c-5ebb-4517-97a1-e8f58413243c" containerName="revoke-keys" Nov 02 06:37:36.455937 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:36.430262 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a40f9931-lfvwh" Nov 02 06:37:36.455937 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:36.438843 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a40f9931-lfvwh"] Nov 02 06:37:36.455937 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:36.450341 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82b7b98c-5ebb-4517-97a1-e8f58413243c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82b7b98c-5ebb-4517-97a1-e8f58413243c-maas-api-openshift-ca\") pod \"82b7b98c-5ebb-4517-97a1-e8f58413243c\" (UID: \"82b7b98c-5ebb-4517-97a1-e8f58413243c\") " Nov 02 06:37:36.455937 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:36.450439 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82b7b98c-5ebb-4517-97a1-e8f58413243c-kube-api-access-pqsk5\" (UniqueName: \"kubernetes.io/projected/82b7b98c-5ebb-4517-97a1-e8f58413243c-kube-api-access-pqsk5\") pod \"82b7b98c-5ebb-4517-97a1-e8f58413243c\" (UID: \"82b7b98c-5ebb-4517-97a1-e8f58413243c\") " Nov 02 06:37:36.475324 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:36.458234 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/82b7b98c-5ebb-4517-97a1-e8f58413243c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82b7b98c-5ebb-4517-97a1-e8f58413243c-maas-api-xks-ca\") pod \"82b7b98c-5ebb-4517-97a1-e8f58413243c\" (UID: \"82b7b98c-5ebb-4517-97a1-e8f58413243c\") " Nov 02 06:37:36.616180 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:36.616146 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d25304c-6148-45bc-8c2b-d0e8b894a02f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a40f9931-lfvwh\" (UID: \"7d25304c-6148-45bc-8c2b-d0e8b894a02f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a40f9931-lfvwh" Nov 02 06:37:36.616367 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:36.616350 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbx82\" (UniqueName: \"kubernetes.io/projected/7d25304c-6148-45bc-8c2b-d0e8b894a02f-kube-api-access-pbx82\") pod \"maas-api-revoke-sub-a40f9931-lfvwh\" (UID: \"7d25304c-6148-45bc-8c2b-d0e8b894a02f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a40f9931-lfvwh" Nov 02 06:37:36.616495 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:36.616480 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d25304c-6148-45bc-8c2b-d0e8b894a02f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a40f9931-lfvwh\" (UID: \"7d25304c-6148-45bc-8c2b-d0e8b894a02f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a40f9931-lfvwh" Nov 02 06:37:36.704928 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:36.686956 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5636d7f-vvq8s" Nov 02 06:37:36.704928 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:36.687140 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5636d7f-vvq8s" event={"ID":"82b7b98c-5ebb-4517-97a1-e8f58413243c","Type":"ContainerDied","Data":"430e737698b284de8e48821c120d8280813225c70fbff320d595f77ad2d34db5"} Nov 02 06:37:36.704928 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:36.687195 5916 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="430e737698b284de8e48821c120d8280813225c70fbff320d595f77ad2d34db5" Nov 02 06:37:36.704928 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:36.688958 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82b7b98c-5ebb-4517-97a1-e8f58413243c-maas-api-openshift-ca" pod "82b7b98c-5ebb-4517-97a1-e8f58413243c" (UID: "82b7b98c-5ebb-4517-97a1-e8f58413243c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:37:36.704928 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:36.693085 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82b7b98c-5ebb-4517-97a1-e8f58413243c-kube-api-access-pqsk5" pod "82b7b98c-5ebb-4517-97a1-e8f58413243c" (UID: "82b7b98c-5ebb-4517-97a1-e8f58413243c"). InnerVolumeSpecName "kube-api-access-pqsk5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:37:36.704928 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:36.701132 5916 patch_prober.go:29] interesting pod/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-rvnb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Nov 02 06:37:36.704928 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:36.701186 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-rvnb6" podUID="0d4ba27f-3588-4c0b-9446-700fcff8bb14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Nov 02 06:37:36.723863 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:36.723829 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d25304c-6148-45bc-8c2b-d0e8b894a02f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a40f9931-lfvwh\" (UID: \"7d25304c-6148-45bc-8c2b-d0e8b894a02f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a40f9931-lfvwh" Nov 02 06:37:36.724606 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:36.724582 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbx82\" (UniqueName: \"kubernetes.io/projected/7d25304c-6148-45bc-8c2b-d0e8b894a02f-kube-api-access-pbx82\") pod \"maas-api-revoke-sub-a40f9931-lfvwh\" (UID: \"7d25304c-6148-45bc-8c2b-d0e8b894a02f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a40f9931-lfvwh" Nov 02 06:37:36.724769 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:36.724754 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d25304c-6148-45bc-8c2b-d0e8b894a02f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a40f9931-lfvwh\" (UID: \"7d25304c-6148-45bc-8c2b-d0e8b894a02f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a40f9931-lfvwh" Nov 02 06:37:36.734154 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:36.734128 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82b7b98c-5ebb-4517-97a1-e8f58413243c-maas-api-openshift-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:37:36.734316 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:36.734301 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqsk5\" (UniqueName: \"kubernetes.io/projected/82b7b98c-5ebb-4517-97a1-e8f58413243c-kube-api-access-pqsk5\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:37:36.735215 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:36.735194 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d25304c-6148-45bc-8c2b-d0e8b894a02f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a40f9931-lfvwh\" (UID: \"7d25304c-6148-45bc-8c2b-d0e8b894a02f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a40f9931-lfvwh" Nov 02 06:37:36.809080 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:36.803131 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82b7b98c-5ebb-4517-97a1-e8f58413243c-maas-api-xks-ca" pod "82b7b98c-5ebb-4517-97a1-e8f58413243c" (UID: "82b7b98c-5ebb-4517-97a1-e8f58413243c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:37:36.864997 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:36.856441 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d25304c-6148-45bc-8c2b-d0e8b894a02f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a40f9931-lfvwh\" (UID: \"7d25304c-6148-45bc-8c2b-d0e8b894a02f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a40f9931-lfvwh" Nov 02 06:37:36.910966 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:36.872363 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82b7b98c-5ebb-4517-97a1-e8f58413243c-maas-api-xks-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:37:37.149983 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:37.147367 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-581608/distinct-w2-581608-kserve-6bb6bb7b65-v7kfj" podUID="dcf2f562-3c50-4de7-9c84-cb659d3bc25b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:37:37.163055 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:37.154569 5916 patch_prober.go:29] interesting pod/e2e-worker-w6-6ff85e-openshift-default-698585659c-x2qpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Nov 02 06:37:37.163055 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:37.154652 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-x2qpj" podUID="33373aea-62e6-4fa4-870d-707c3e7bc858" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Nov 02 06:37:37.163454 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:37.163417 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbx82\" (UniqueName: \"kubernetes.io/projected/7d25304c-6148-45bc-8c2b-d0e8b894a02f-kube-api-access-pbx82\") pod \"maas-api-revoke-sub-a40f9931-lfvwh\" (UID: \"7d25304c-6148-45bc-8c2b-d0e8b894a02f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a40f9931-lfvwh" Nov 02 06:37:37.244930 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:37.242363 5916 patch_prober.go:29] interesting pod/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-br7cd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Nov 02 06:37:37.244930 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:37.242422 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-br7cd" podUID="2f9273e5-b930-47f0-b710-234974ad9025" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Nov 02 06:37:37.270536 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:37.270492 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-680b7b/simulator-w4-680b7b-kserve-6c4f574c56-p5xtc" podUID="be8a1039-5934-4885-93fe-070575f936ec" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:37:37.282150 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:37.282109 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a40f9931-lfvwh" Nov 02 06:37:37.442355 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:37.432029 5916 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-680b7b/simulator-w4-680b7b-kserve-6c4f574c56-p5xtc" podUID="be8a1039-5934-4885-93fe-070575f936ec" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 06:37:37.551927 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:37.550990 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bdcd80/distinct-w0-bdcd80-kserve-5d9df79d48-bn5d4" podUID="1a7c7a03-e0cf-44ca-8eee-6c3e16112b3a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded" Nov 02 06:37:37.609868 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:37.604344 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bdcd80/simulator-w0-bdcd80-kserve-5749c847cb-9ktcr" podUID="2043ff0a-8a71-4b71-9750-beb566a12bad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 06:37:37.661378 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:37.647049 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fb1398/distinct-2-w3-fb1398-kserve-597b7dc99b-dp4jf" podUID="523a31c4-4aea-4e9f-9ec4-7c28faefbf7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 06:37:37.708970 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:37.708391 5916 patch_prober.go:29] interesting pod/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-rvnb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Nov 02 06:37:37.708970 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:37.708464 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-rvnb6" podUID="0d4ba27f-3588-4c0b-9446-700fcff8bb14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Nov 02 06:37:37.876569 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:37.876518 5916 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-cdfce9/simulator-w1-cdfce9-kserve-7ffd68f78-5vwl2" podUID="b8b219aa-6a59-4d31-96f6-8d50ab910aca" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:37:37.876931 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:37.876883 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cdfce9/premium-w1-cdfce9-kserve-598cb9d57f-mrzrz" podUID="f9c5c1f7-cdca-46d6-8573-c85af1cd1946" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:37:37.930974 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:37.925980 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dcf9c3/distinct-2-w5-dcf9c3-kserve-64747dbd8f-wlwng" podUID="e37d42ab-7b62-4f07-8a29-cdedeb0fe89a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Nov 02 06:37:37.989633 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:37.989236 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-581608/distinct-2-w2-581608-kserve-57dd49c4b-qnt9k" podUID="837780ad-8ea3-4ed2-922f-917cffc7bb93" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:37:38.203929 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:38.135396 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-581608/simulator-w2-581608-kserve-68f68d8559-xkcwp" podUID="529e25e6-4479-49d6-85c7-85dad6c2193e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 06:37:38.203929 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:38.177148 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-581608/distinct-w2-581608-kserve-6bb6bb7b65-v7kfj" podUID="dcf2f562-3c50-4de7-9c84-cb659d3bc25b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:37:38.238272 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:38.205065 5916 patch_prober.go:29] interesting pod/e2e-worker-w6-6ff85e-openshift-default-698585659c-x2qpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Nov 02 06:37:38.238272 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:38.205118 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-x2qpj" podUID="33373aea-62e6-4fa4-870d-707c3e7bc858" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Nov 02 06:37:38.266984 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:38.265713 5916 patch_prober.go:29] interesting pod/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-br7cd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Nov 02 06:37:38.266984 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:38.265773 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-br7cd" podUID="2f9273e5-b930-47f0-b710-234974ad9025" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Nov 02 06:37:38.280004 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:38.278995 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-680b7b/simulator-w4-680b7b-kserve-6c4f574c56-p5xtc" podUID="be8a1039-5934-4885-93fe-070575f936ec" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Nov 02 06:37:38.314277 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:38.298926 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-581608/distinct-w2-581608-kserve-6bb6bb7b65-v7kfj" Nov 02 06:37:38.314277 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:38.298976 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2e0b70a-b2m7s"] Nov 02 06:37:38.314277 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:38.299022 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-680b7b/simulator-w4-680b7b-kserve-6c4f574c56-p5xtc" Nov 02 06:37:38.411003 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:38.346206 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cdfce9/simulator-w1-cdfce9-kserve-7ffd68f78-5vwl2" podUID="b8b219aa-6a59-4d31-96f6-8d50ab910aca" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:37:38.457051 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:38.456998 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6ff85e/premium-w6-6ff85e-kserve-59848ff6d4-szbxx" podUID="eb386976-f898-4f1d-b27c-77bccc735a3b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:37:38.590551 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:38.572400 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bdcd80/distinct-w0-bdcd80-kserve-5d9df79d48-bn5d4" podUID="1a7c7a03-e0cf-44ca-8eee-6c3e16112b3a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:37:38.590551 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:38.573265 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-bdcd80/distinct-w0-bdcd80-kserve-5d9df79d48-bn5d4" Nov 02 06:37:38.627931 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:38.613014 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bdcd80/simulator-w0-bdcd80-kserve-5749c847cb-9ktcr" podUID="2043ff0a-8a71-4b71-9750-beb566a12bad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:37:38.627931 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:38.615652 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-bdcd80/simulator-w0-bdcd80-kserve-5749c847cb-9ktcr" Nov 02 06:37:38.655937 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:38.652500 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fb1398/distinct-2-w3-fb1398-kserve-597b7dc99b-dp4jf" podUID="523a31c4-4aea-4e9f-9ec4-7c28faefbf7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:37:38.655937 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:37:38.653139 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d25304c_6148_45bc_8c2b_d0e8b894a02f.slice/crio-8b2cc0afaea886a84ab64a6ae6c15b233c9770d14d0ba06d244bab48274c1c3c WatchSource:0}: Error finding container 8b2cc0afaea886a84ab64a6ae6c15b233c9770d14d0ba06d244bab48274c1c3c: Status 404 returned error can't find the container with id 8b2cc0afaea886a84ab64a6ae6c15b233c9770d14d0ba06d244bab48274c1c3c Nov 02 06:37:38.697976 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:38.697940 5916 patch_prober.go:29] interesting pod/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-rvnb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Nov 02 06:37:38.724935 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:38.700069 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-rvnb6" podUID="0d4ba27f-3588-4c0b-9446-700fcff8bb14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Nov 02 06:37:38.784331 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:38.699211 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6ff85e/simulator-w6-6ff85e-kserve-9b9fd978f-q76nl" podUID="30114285-d730-4abb-a618-6833d5746feb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Nov 02 06:37:38.809931 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:38.703379 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-fb1398/distinct-2-w3-fb1398-kserve-597b7dc99b-dp4jf" Nov 02 06:37:38.810753 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:38.810729 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2e0b70a-b2m7s"] Nov 02 06:37:38.955955 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:38.902975 5916 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-bdcd80/simulator-w0-bdcd80-kserve-5749c847cb-9ktcr" podUID="2043ff0a-8a71-4b71-9750-beb566a12bad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:37:38.955955 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:38.939573 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dcf9c3/distinct-2-w5-dcf9c3-kserve-64747dbd8f-wlwng" podUID="e37d42ab-7b62-4f07-8a29-cdedeb0fe89a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:37:38.955955 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:38.947011 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-581608/distinct-w2-581608-kserve-6bb6bb7b65-v7kfj" Nov 02 06:37:38.955955 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:38.950289 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-dcf9c3/distinct-2-w5-dcf9c3-kserve-64747dbd8f-wlwng" Nov 02 06:37:38.971939 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:38.963976 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a40f9931-lfvwh" event={"ID":"7d25304c-6148-45bc-8c2b-d0e8b894a02f","Type":"ContainerStarted","Data":"8b2cc0afaea886a84ab64a6ae6c15b233c9770d14d0ba06d244bab48274c1c3c"} Nov 02 06:37:38.991947 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:38.990278 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-581608/distinct-2-w2-581608-kserve-57dd49c4b-qnt9k" podUID="837780ad-8ea3-4ed2-922f-917cffc7bb93" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:37:38.991947 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:38.990899 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-581608/distinct-2-w2-581608-kserve-57dd49c4b-qnt9k" Nov 02 06:37:39.020845 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:38.901814 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cdfce9/premium-w1-cdfce9-kserve-598cb9d57f-mrzrz" podUID="f9c5c1f7-cdca-46d6-8573-c85af1cd1946" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:37:39.020845 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:39.011512 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-cdfce9/premium-w1-cdfce9-kserve-598cb9d57f-mrzrz" Nov 02 06:37:39.048994 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:39.042458 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-fb1398/distinct-2-w3-fb1398-kserve-597b7dc99b-dp4jf" Nov 02 06:37:39.106801 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:39.104717 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-bdcd80/simulator-w0-bdcd80-kserve-5749c847cb-9ktcr" Nov 02 06:37:39.136642 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:39.136598 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-581608/simulator-w2-581608-kserve-68f68d8559-xkcwp" podUID="529e25e6-4479-49d6-85c7-85dad6c2193e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:37:39.137287 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:39.136976 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-581608/simulator-w2-581608-kserve-68f68d8559-xkcwp" Nov 02 06:37:39.138046 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:39.138031 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-dcf9c3/distinct-2-w5-dcf9c3-kserve-64747dbd8f-wlwng" Nov 02 06:37:39.159337 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:39.159309 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-581608/simulator-w2-581608-kserve-68f68d8559-xkcwp" Nov 02 06:37:39.201032 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:39.200998 5916 patch_prober.go:29] interesting pod/e2e-worker-w6-6ff85e-openshift-default-698585659c-x2qpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Nov 02 06:37:39.201573 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:39.201543 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-x2qpj" podUID="33373aea-62e6-4fa4-870d-707c3e7bc858" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Nov 02 06:37:39.201777 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:39.201254 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a40f9931-lfvwh"] Nov 02 06:37:39.303975 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:39.256325 5916 patch_prober.go:29] interesting pod/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-br7cd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Nov 02 06:37:39.303975 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:39.256380 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-br7cd" podUID="2f9273e5-b930-47f0-b710-234974ad9025" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Nov 02 06:37:39.303975 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:39.282024 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-680b7b/simulator-w4-680b7b-kserve-6c4f574c56-p5xtc" podUID="be8a1039-5934-4885-93fe-070575f936ec" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:37:39.420005 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:39.416085 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-680b7b/simulator-w4-680b7b-kserve-6c4f574c56-p5xtc" Nov 02 06:37:39.499213 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:39.499120 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6ff85e/premium-w6-6ff85e-kserve-59848ff6d4-szbxx" podUID="eb386976-f898-4f1d-b27c-77bccc735a3b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:37:39.499652 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:39.499504 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-6ff85e/premium-w6-6ff85e-kserve-59848ff6d4-szbxx" Nov 02 06:37:39.508262 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:39.508164 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-cdfce9/premium-w1-cdfce9-kserve-598cb9d57f-mrzrz" Nov 02 06:37:39.603079 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:39.603032 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bdcd80/distinct-w0-bdcd80-kserve-5d9df79d48-bn5d4" podUID="1a7c7a03-e0cf-44ca-8eee-6c3e16112b3a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:37:39.624292 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:39.623880 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-581608/distinct-2-w2-581608-kserve-57dd49c4b-qnt9k" Nov 02 06:37:39.624292 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:39.624114 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-6ff85e/premium-w6-6ff85e-kserve-59848ff6d4-szbxx" Nov 02 06:37:39.713058 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:39.712066 5916 patch_prober.go:29] interesting pod/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-rvnb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Nov 02 06:37:39.713058 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:39.712128 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-rvnb6" podUID="0d4ba27f-3588-4c0b-9446-700fcff8bb14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Nov 02 06:37:39.743040 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:39.733394 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-bdcd80/distinct-w0-bdcd80-kserve-5d9df79d48-bn5d4" Nov 02 06:37:39.998042 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:39.998009 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92434cc8-1efe-4e63-8a29-b9093d774e51" path="/var/lib/kubelet/pods/92434cc8-1efe-4e63-8a29-b9093d774e51/volumes" Nov 02 06:37:40.157929 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:40.157546 5916 patch_prober.go:29] interesting pod/e2e-worker-w6-6ff85e-openshift-default-698585659c-x2qpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Nov 02 06:37:40.157929 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:40.157609 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-x2qpj" podUID="33373aea-62e6-4fa4-870d-707c3e7bc858" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Nov 02 06:37:40.239945 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:40.238890 5916 patch_prober.go:29] interesting pod/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-br7cd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Nov 02 06:37:40.239945 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:40.238980 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-br7cd" podUID="2f9273e5-b930-47f0-b710-234974ad9025" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Nov 02 06:37:40.772159 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:40.718372 5916 patch_prober.go:29] interesting pod/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-rvnb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Nov 02 06:37:40.772159 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:40.718430 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-rvnb6" podUID="0d4ba27f-3588-4c0b-9446-700fcff8bb14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Nov 02 06:37:41.182943 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:41.176426 5916 patch_prober.go:29] interesting pod/e2e-worker-w6-6ff85e-openshift-default-698585659c-x2qpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Nov 02 06:37:41.182943 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:41.176511 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-x2qpj" podUID="33373aea-62e6-4fa4-870d-707c3e7bc858" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Nov 02 06:37:41.337942 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:41.316275 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a40f9931-lfvwh" event={"ID":"7d25304c-6148-45bc-8c2b-d0e8b894a02f","Type":"ContainerStarted","Data":"134f5f98e3419367bf5bc641c64ad06b61742cd313094b4a3c379882fd222bfd"} Nov 02 06:37:42.192930 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:42.191111 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dcf9c3/distinct-2-w5-dcf9c3-kserve-64747dbd8f-wlwng" podUID="e37d42ab-7b62-4f07-8a29-cdedeb0fe89a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 06:37:42.264929 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:42.264426 5916 patch_prober.go:29] interesting pod/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-br7cd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 06:37:42.264929 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:42.264493 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-br7cd" podUID="2f9273e5-b930-47f0-b710-234974ad9025" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 06:37:42.316142 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:42.316094 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cdfce9/simulator-w1-cdfce9-kserve-7ffd68f78-5vwl2" podUID="b8b219aa-6a59-4d31-96f6-8d50ab910aca" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:37:42.383939 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:42.382457 5916 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-bdcd80/distinct-w0-bdcd80-kserve-5d9df79d48-bn5d4" podUID="1a7c7a03-e0cf-44ca-8eee-6c3e16112b3a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:37:42.430858 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:42.430822 5916 generic.go:356] "Generic (PLEG): container finished" podID="7d25304c-6148-45bc-8c2b-d0e8b894a02f" containerID="134f5f98e3419367bf5bc641c64ad06b61742cd313094b4a3c379882fd222bfd" exitCode=0 Nov 02 06:37:42.436701 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:42.435444 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a40f9931-lfvwh" event={"ID":"7d25304c-6148-45bc-8c2b-d0e8b894a02f","Type":"ContainerDied","Data":"134f5f98e3419367bf5bc641c64ad06b61742cd313094b4a3c379882fd222bfd"} Nov 02 06:37:42.475109 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:42.475007 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-680b7b/simulator-w4-680b7b-kserve-6c4f574c56-p5xtc" podUID="be8a1039-5934-4885-93fe-070575f936ec" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 06:37:42.510410 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:42.510085 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cdfce9/premium-w1-cdfce9-kserve-598cb9d57f-mrzrz" podUID="f9c5c1f7-cdca-46d6-8573-c85af1cd1946" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:37:42.645453 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:42.645409 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-581608/distinct-2-w2-581608-kserve-57dd49c4b-qnt9k" podUID="837780ad-8ea3-4ed2-922f-917cffc7bb93" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:37:42.741999 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:42.646137 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6ff85e/premium-w6-6ff85e-kserve-59848ff6d4-szbxx" podUID="eb386976-f898-4f1d-b27c-77bccc735a3b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Nov 02 06:37:42.741999 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:42.691187 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6ff85e/simulator-w6-6ff85e-kserve-9b9fd978f-q76nl" podUID="30114285-d730-4abb-a618-6833d5746feb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:37:42.741999 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:42.715019 5916 patch_prober.go:29] interesting pod/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-rvnb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 06:37:42.741999 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:42.741303 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-rvnb6" podUID="0d4ba27f-3588-4c0b-9446-700fcff8bb14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 06:37:42.741999 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:42.732180 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bdcd80/distinct-w0-bdcd80-kserve-5d9df79d48-bn5d4" podUID="1a7c7a03-e0cf-44ca-8eee-6c3e16112b3a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:37:42.753028 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:42.751002 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-581608/distinct-w2-581608-kserve-6bb6bb7b65-v7kfj" podUID="dcf2f562-3c50-4de7-9c84-cb659d3bc25b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:37:42.984974 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:42.981560 5916 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-6ff85e/premium-w6-6ff85e-kserve-59848ff6d4-szbxx" podUID="eb386976-f898-4f1d-b27c-77bccc735a3b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:37:43.139166 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:43.083070 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fb1398/distinct-2-w3-fb1398-kserve-597b7dc99b-dp4jf" podUID="523a31c4-4aea-4e9f-9ec4-7c28faefbf7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:37:43.192939 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:43.179792 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bdcd80/simulator-w0-bdcd80-kserve-5749c847cb-9ktcr" podUID="2043ff0a-8a71-4b71-9750-beb566a12bad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:37:43.215442 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:43.203954 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-581608/simulator-w2-581608-kserve-68f68d8559-xkcwp" podUID="529e25e6-4479-49d6-85c7-85dad6c2193e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Nov 02 06:37:43.231990 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:43.226065 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dcf9c3/distinct-2-w5-dcf9c3-kserve-64747dbd8f-wlwng" podUID="e37d42ab-7b62-4f07-8a29-cdedeb0fe89a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Nov 02 06:37:43.231990 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:43.226508 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-dcf9c3/distinct-2-w5-dcf9c3-kserve-64747dbd8f-wlwng" Nov 02 06:37:43.269926 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:43.269240 5916 patch_prober.go:29] interesting pod/e2e-worker-w6-6ff85e-openshift-default-698585659c-x2qpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 06:37:43.269926 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:43.269306 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-x2qpj" podUID="33373aea-62e6-4fa4-870d-707c3e7bc858" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 06:37:43.366930 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:43.335022 5916 patch_prober.go:29] interesting pod/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-br7cd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 06:37:43.366930 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:43.335110 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-br7cd" podUID="2f9273e5-b930-47f0-b710-234974ad9025" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 06:37:43.366930 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:43.336519 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cdfce9/simulator-w1-cdfce9-kserve-7ffd68f78-5vwl2" podUID="b8b219aa-6a59-4d31-96f6-8d50ab910aca" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:37:43.366930 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:43.337001 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-cdfce9/simulator-w1-cdfce9-kserve-7ffd68f78-5vwl2" Nov 02 06:37:43.527505 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:43.511345 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-680b7b/simulator-w4-680b7b-kserve-6c4f574c56-p5xtc" podUID="be8a1039-5934-4885-93fe-070575f936ec" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:37:43.527505 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:43.511852 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-680b7b/simulator-w4-680b7b-kserve-6c4f574c56-p5xtc" Nov 02 06:37:43.583280 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:43.542593 5916 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-cdfce9/premium-w1-cdfce9-kserve-598cb9d57f-mrzrz" podUID="f9c5c1f7-cdca-46d6-8573-c85af1cd1946" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:37:43.583280 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:43.542726 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cdfce9/premium-w1-cdfce9-kserve-598cb9d57f-mrzrz" podUID="f9c5c1f7-cdca-46d6-8573-c85af1cd1946" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Nov 02 06:37:43.583280 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:43.549061 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-cdfce9/premium-w1-cdfce9-kserve-598cb9d57f-mrzrz" Nov 02 06:37:44.054033 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:43.956837 5916 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-6ff85e/simulator-w6-6ff85e-kserve-9b9fd978f-q76nl" podUID="30114285-d730-4abb-a618-6833d5746feb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": context deadline exceeded" Nov 02 06:37:44.388990 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:44.086718 5916 patch_prober.go:29] interesting pod/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-rvnb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 06:37:44.413099 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:44.413049 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-rvnb6" podUID="0d4ba27f-3588-4c0b-9446-700fcff8bb14" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 06:37:44.413667 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:44.187203 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bdcd80/simulator-w0-bdcd80-kserve-5749c847cb-9ktcr" podUID="2043ff0a-8a71-4b71-9750-beb566a12bad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:37:44.414132 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:44.389245 5916 patch_prober.go:29] interesting pod/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-br7cd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 06:37:44.414295 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:44.414270 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-br7cd" podUID="2f9273e5-b930-47f0-b710-234974ad9025" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 06:37:44.414870 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:44.187593 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fb1398/distinct-2-w3-fb1398-kserve-597b7dc99b-dp4jf" podUID="523a31c4-4aea-4e9f-9ec4-7c28faefbf7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:37:44.415496 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:44.228046 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-581608/simulator-w2-581608-kserve-68f68d8559-xkcwp" podUID="529e25e6-4479-49d6-85c7-85dad6c2193e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:37:44.416077 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:44.229742 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-581608/distinct-w2-581608-kserve-6bb6bb7b65-v7kfj" podUID="dcf2f562-3c50-4de7-9c84-cb659d3bc25b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded" Nov 02 06:37:44.416663 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:44.229778 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dcf9c3/distinct-2-w5-dcf9c3-kserve-64747dbd8f-wlwng" podUID="e37d42ab-7b62-4f07-8a29-cdedeb0fe89a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:37:44.417238 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:44.376622 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cdfce9/simulator-w1-cdfce9-kserve-7ffd68f78-5vwl2" podUID="b8b219aa-6a59-4d31-96f6-8d50ab910aca" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Nov 02 06:37:44.556927 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:44.556313 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-680b7b/simulator-w4-680b7b-kserve-6c4f574c56-p5xtc" podUID="be8a1039-5934-4885-93fe-070575f936ec" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Nov 02 06:37:44.556927 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:44.556751 5916 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-dcf9c3/distinct-2-w5-dcf9c3-kserve-64747dbd8f-wlwng" podUID="e37d42ab-7b62-4f07-8a29-cdedeb0fe89a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/health\": context deadline exceeded" Nov 02 06:37:44.556927 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:44.435659 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6ff85e/simulator-w6-6ff85e-kserve-9b9fd978f-q76nl" podUID="30114285-d730-4abb-a618-6833d5746feb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Nov 02 06:37:44.797015 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:44.435253 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-581608/distinct-2-w2-581608-kserve-57dd49c4b-qnt9k" podUID="837780ad-8ea3-4ed2-922f-917cffc7bb93" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": context deadline exceeded" Nov 02 06:37:44.824943 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:44.435446 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6ff85e/premium-w6-6ff85e-kserve-59848ff6d4-szbxx" podUID="eb386976-f898-4f1d-b27c-77bccc735a3b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Nov 02 06:37:44.824943 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:44.438638 5916 patch_prober.go:29] interesting pod/e2e-worker-w6-6ff85e-openshift-default-698585659c-x2qpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 06:37:44.824943 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:44.821503 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-x2qpj" podUID="33373aea-62e6-4fa4-870d-707c3e7bc858" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 06:37:44.845177 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:44.845142 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a40f9931-lfvwh" Nov 02 06:37:44.931923 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:44.931240 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cdfce9/premium-w1-cdfce9-kserve-598cb9d57f-mrzrz" podUID="f9c5c1f7-cdca-46d6-8573-c85af1cd1946" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:37:45.127826 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:45.127731 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d25304c-6148-45bc-8c2b-d0e8b894a02f-kube-api-access-pbx82\" (UniqueName: \"kubernetes.io/projected/7d25304c-6148-45bc-8c2b-d0e8b894a02f-kube-api-access-pbx82\") pod \"7d25304c-6148-45bc-8c2b-d0e8b894a02f\" (UID: \"7d25304c-6148-45bc-8c2b-d0e8b894a02f\") " Nov 02 06:37:45.221640 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:45.187135 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d25304c-6148-45bc-8c2b-d0e8b894a02f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d25304c-6148-45bc-8c2b-d0e8b894a02f-maas-api-openshift-ca\") pod \"7d25304c-6148-45bc-8c2b-d0e8b894a02f\" (UID: \"7d25304c-6148-45bc-8c2b-d0e8b894a02f\") " Nov 02 06:37:45.221640 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:45.187259 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d25304c-6148-45bc-8c2b-d0e8b894a02f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d25304c-6148-45bc-8c2b-d0e8b894a02f-maas-api-xks-ca\") pod \"7d25304c-6148-45bc-8c2b-d0e8b894a02f\" (UID: \"7d25304c-6148-45bc-8c2b-d0e8b894a02f\") " Nov 02 06:37:45.221640 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:45.213453 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d25304c-6148-45bc-8c2b-d0e8b894a02f-maas-api-xks-ca" pod "7d25304c-6148-45bc-8c2b-d0e8b894a02f" (UID: "7d25304c-6148-45bc-8c2b-d0e8b894a02f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:37:45.221640 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:45.213814 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d25304c-6148-45bc-8c2b-d0e8b894a02f-maas-api-openshift-ca" pod "7d25304c-6148-45bc-8c2b-d0e8b894a02f" (UID: "7d25304c-6148-45bc-8c2b-d0e8b894a02f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:37:45.241141 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:37:45.241103 5916 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.32s" Nov 02 06:37:45.241639 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:45.241615 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-bdcd80/simulator-w0-bdcd80-kserve-5749c847cb-9ktcr" Nov 02 06:37:45.241868 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:45.241856 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-fb1398/distinct-2-w3-fb1398-kserve-597b7dc99b-dp4jf" Nov 02 06:37:45.242088 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:45.242074 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-581608/simulator-w2-581608-kserve-68f68d8559-xkcwp" Nov 02 06:37:45.242298 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:45.242284 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-581608/distinct-w2-581608-kserve-6bb6bb7b65-v7kfj" Nov 02 06:37:45.242509 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:45.242495 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-6ff85e/simulator-w6-6ff85e-kserve-9b9fd978f-q76nl" Nov 02 06:37:45.242730 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:45.242716 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-581608/distinct-2-w2-581608-kserve-57dd49c4b-qnt9k" Nov 02 06:37:45.242942 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:45.242929 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-6ff85e/premium-w6-6ff85e-kserve-59848ff6d4-szbxx" Nov 02 06:37:45.271446 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:45.271385 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bdcd80/distinct-w0-bdcd80-kserve-5d9df79d48-bn5d4" podUID="1a7c7a03-e0cf-44ca-8eee-6c3e16112b3a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:37:45.302770 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:45.243116 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-6ff85e/simulator-w6-6ff85e-kserve-9b9fd978f-q76nl" Nov 02 06:37:45.309288 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:45.309243 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d25304c-6148-45bc-8c2b-d0e8b894a02f-maas-api-openshift-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:37:45.309540 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:45.309524 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d25304c-6148-45bc-8c2b-d0e8b894a02f-maas-api-xks-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:37:45.320707 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:45.320667 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a40f9931-lfvwh" event={"ID":"7d25304c-6148-45bc-8c2b-d0e8b894a02f","Type":"ContainerDied","Data":"8b2cc0afaea886a84ab64a6ae6c15b233c9770d14d0ba06d244bab48274c1c3c"} Nov 02 06:37:45.320990 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:45.320972 5916 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8b2cc0afaea886a84ab64a6ae6c15b233c9770d14d0ba06d244bab48274c1c3c" Nov 02 06:37:45.321186 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:45.321173 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a40f9931-lfvwh" Nov 02 06:37:45.392672 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:45.392613 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d25304c-6148-45bc-8c2b-d0e8b894a02f-kube-api-access-pbx82" pod "7d25304c-6148-45bc-8c2b-d0e8b894a02f" (UID: "7d25304c-6148-45bc-8c2b-d0e8b894a02f"). InnerVolumeSpecName "kube-api-access-pbx82". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:37:45.422089 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:45.422028 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dcf9c3/distinct-2-w5-dcf9c3-kserve-64747dbd8f-wlwng" podUID="e37d42ab-7b62-4f07-8a29-cdedeb0fe89a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:37:45.425049 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:45.425008 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-581608/simulator-w2-581608-kserve-68f68d8559-xkcwp" podUID="529e25e6-4479-49d6-85c7-85dad6c2193e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:37:45.452953 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:45.425533 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-581608/distinct-w2-581608-kserve-6bb6bb7b65-v7kfj" podUID="dcf2f562-3c50-4de7-9c84-cb659d3bc25b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded" Nov 02 06:37:45.452953 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:45.430006 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fb1398/distinct-2-w3-fb1398-kserve-597b7dc99b-dp4jf" podUID="523a31c4-4aea-4e9f-9ec4-7c28faefbf7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:37:45.452953 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:45.430368 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bdcd80/simulator-w0-bdcd80-kserve-5749c847cb-9ktcr" podUID="2043ff0a-8a71-4b71-9750-beb566a12bad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Nov 02 06:37:45.454888 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:45.454850 5916 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-581608/distinct-2-w2-581608-kserve-57dd49c4b-qnt9k" podUID="837780ad-8ea3-4ed2-922f-917cffc7bb93" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:37:45.474035 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:45.471097 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbx82\" (UniqueName: \"kubernetes.io/projected/7d25304c-6148-45bc-8c2b-d0e8b894a02f-kube-api-access-pbx82\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:37:45.474035 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:45.473303 5916 patch_prober.go:29] interesting pod/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-br7cd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 06:37:45.474035 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:45.473372 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-br7cd" podUID="2f9273e5-b930-47f0-b710-234974ad9025" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 06:37:45.555500 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:45.555463 5916 patch_prober.go:29] interesting pod/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-rvnb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 06:37:45.555804 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:45.555773 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-rvnb6" podUID="0d4ba27f-3588-4c0b-9446-700fcff8bb14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 06:37:45.570173 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:45.570133 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-680b7b/simulator-w4-680b7b-kserve-6c4f574c56-p5xtc" podUID="be8a1039-5934-4885-93fe-070575f936ec" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:37:45.921627 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:45.921583 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6ff85e/premium-w6-6ff85e-kserve-59848ff6d4-szbxx" podUID="eb386976-f898-4f1d-b27c-77bccc735a3b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Nov 02 06:37:45.931072 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:45.931024 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-581608/distinct-2-w2-581608-kserve-57dd49c4b-qnt9k" podUID="837780ad-8ea3-4ed2-922f-917cffc7bb93" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:37:45.933192 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:45.933157 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cdfce9/simulator-w1-cdfce9-kserve-7ffd68f78-5vwl2" podUID="b8b219aa-6a59-4d31-96f6-8d50ab910aca" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Nov 02 06:37:45.939798 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:45.938046 5916 patch_prober.go:29] interesting pod/e2e-worker-w6-6ff85e-openshift-default-698585659c-x2qpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 06:37:45.946050 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:45.946017 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-x2qpj" podUID="33373aea-62e6-4fa4-870d-707c3e7bc858" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 06:37:46.043526 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:46.020706 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cdfce9/premium-w1-cdfce9-kserve-598cb9d57f-mrzrz" podUID="f9c5c1f7-cdca-46d6-8573-c85af1cd1946" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Nov 02 06:37:46.046485 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:46.044076 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6ff85e/simulator-w6-6ff85e-kserve-9b9fd978f-q76nl" podUID="30114285-d730-4abb-a618-6833d5746feb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:37:46.073930 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:46.073537 5916 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-fb1398/distinct-2-w3-fb1398-kserve-597b7dc99b-dp4jf" podUID="523a31c4-4aea-4e9f-9ec4-7c28faefbf7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/health\": context deadline exceeded" Nov 02 06:37:46.299575 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:46.299477 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-581608/distinct-2-w2-581608-kserve-57dd49c4b-qnt9k" Nov 02 06:37:46.299851 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:46.299825 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-581608/simulator-w2-581608-kserve-68f68d8559-xkcwp" Nov 02 06:37:46.300004 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:46.299992 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-bdcd80/simulator-w0-bdcd80-kserve-5749c847cb-9ktcr" Nov 02 06:37:46.300131 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:46.300108 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-581608/distinct-w2-581608-kserve-6bb6bb7b65-v7kfj" Nov 02 06:37:46.300269 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:46.300259 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-6ff85e/premium-w6-6ff85e-kserve-59848ff6d4-szbxx" Nov 02 06:37:46.314789 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:46.314734 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-dcf9c3/distinct-2-w5-dcf9c3-kserve-64747dbd8f-wlwng" Nov 02 06:37:46.356270 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:46.356242 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-cdfce9/simulator-w1-cdfce9-kserve-7ffd68f78-5vwl2" Nov 02 06:37:46.357034 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:46.357012 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-680b7b/simulator-w4-680b7b-kserve-6c4f574c56-p5xtc" Nov 02 06:37:46.360592 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:46.360573 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-fb1398/distinct-2-w3-fb1398-kserve-597b7dc99b-dp4jf" Nov 02 06:37:46.378064 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:46.378040 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-cdfce9/premium-w1-cdfce9-kserve-598cb9d57f-mrzrz" Nov 02 06:37:46.519144 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:46.519070 5916 patch_prober.go:29] interesting pod/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-br7cd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 06:37:46.519771 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:46.519739 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-br7cd" podUID="2f9273e5-b930-47f0-b710-234974ad9025" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 06:37:46.543733 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:46.540222 5916 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-br7cd" Nov 02 06:37:46.543733 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:46.541384 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-br7cd" Nov 02 06:37:46.544336 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:46.544304 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-br7cd" Nov 02 06:37:46.613075 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:46.612956 5916 patch_prober.go:29] interesting pod/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-rvnb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 06:37:46.613375 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:46.613324 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-rvnb6" podUID="0d4ba27f-3588-4c0b-9446-700fcff8bb14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 06:37:46.992048 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:46.992010 5916 patch_prober.go:29] interesting pod/e2e-worker-w6-6ff85e-openshift-default-698585659c-x2qpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 06:37:46.992363 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:46.992332 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-x2qpj" podUID="33373aea-62e6-4fa4-870d-707c3e7bc858" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 06:37:47.209956 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:47.207518 5916 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-rvnb6" Nov 02 06:37:47.209956 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:47.208629 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-rvnb6" Nov 02 06:37:47.225024 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:47.222434 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-rvnb6" Nov 02 06:37:47.408134 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:47.408100 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-fhb78"] Nov 02 06:37:47.409306 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:47.409281 5916 cpu_manager.go:414] "containerMap: removing container" podUID="7d25304c-6148-45bc-8c2b-d0e8b894a02f" containerName="revoke-keys" Nov 02 06:37:47.409468 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:47.409454 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d25304c-6148-45bc-8c2b-d0e8b894a02f" containerName="revoke-keys" Nov 02 06:37:47.409768 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:47.409751 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d25304c-6148-45bc-8c2b-d0e8b894a02f" containerName="revoke-keys" Nov 02 06:37:47.414760 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:47.414742 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-fhb78" Nov 02 06:37:47.501737 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:47.501613 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-fhb78"] Nov 02 06:37:47.506951 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:47.506882 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/933acc24-86ce-4896-8daa-6d36d6ba7cae-istio-podinfo\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-fhb78\" (UID: \"933acc24-86ce-4896-8daa-6d36d6ba7cae\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-fhb78" Nov 02 06:37:47.507191 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:47.507169 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xx472\" (UniqueName: \"kubernetes.io/projected/933acc24-86ce-4896-8daa-6d36d6ba7cae-kube-api-access-xx472\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-fhb78\" (UID: \"933acc24-86ce-4896-8daa-6d36d6ba7cae\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-fhb78" Nov 02 06:37:47.507369 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:47.507350 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/933acc24-86ce-4896-8daa-6d36d6ba7cae-credential-socket\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-fhb78\" (UID: \"933acc24-86ce-4896-8daa-6d36d6ba7cae\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-fhb78" Nov 02 06:37:47.507521 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:47.507484 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/933acc24-86ce-4896-8daa-6d36d6ba7cae-istio-token\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-fhb78\" (UID: \"933acc24-86ce-4896-8daa-6d36d6ba7cae\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-fhb78" Nov 02 06:37:47.507732 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:47.507713 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/933acc24-86ce-4896-8daa-6d36d6ba7cae-workload-certs\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-fhb78\" (UID: \"933acc24-86ce-4896-8daa-6d36d6ba7cae\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-fhb78" Nov 02 06:37:47.507938 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:47.507889 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/933acc24-86ce-4896-8daa-6d36d6ba7cae-istio-envoy\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-fhb78\" (UID: \"933acc24-86ce-4896-8daa-6d36d6ba7cae\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-fhb78" Nov 02 06:37:47.508127 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:47.508110 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/933acc24-86ce-4896-8daa-6d36d6ba7cae-istio-data\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-fhb78\" (UID: \"933acc24-86ce-4896-8daa-6d36d6ba7cae\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-fhb78" Nov 02 06:37:47.508306 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:47.508289 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/933acc24-86ce-4896-8daa-6d36d6ba7cae-istiod-ca-cert\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-fhb78\" (UID: \"933acc24-86ce-4896-8daa-6d36d6ba7cae\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-fhb78" Nov 02 06:37:47.508553 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:47.508534 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/933acc24-86ce-4896-8daa-6d36d6ba7cae-workload-socket\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-fhb78\" (UID: \"933acc24-86ce-4896-8daa-6d36d6ba7cae\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-fhb78" Nov 02 06:37:47.610195 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:47.609711 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/933acc24-86ce-4896-8daa-6d36d6ba7cae-credential-socket\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-fhb78\" (UID: \"933acc24-86ce-4896-8daa-6d36d6ba7cae\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-fhb78" Nov 02 06:37:47.610195 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:47.609778 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/933acc24-86ce-4896-8daa-6d36d6ba7cae-istio-token\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-fhb78\" (UID: \"933acc24-86ce-4896-8daa-6d36d6ba7cae\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-fhb78" Nov 02 06:37:47.610195 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:47.609872 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/933acc24-86ce-4896-8daa-6d36d6ba7cae-workload-certs\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-fhb78\" (UID: \"933acc24-86ce-4896-8daa-6d36d6ba7cae\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-fhb78" Nov 02 06:37:47.610195 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:47.609938 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/933acc24-86ce-4896-8daa-6d36d6ba7cae-istio-envoy\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-fhb78\" (UID: \"933acc24-86ce-4896-8daa-6d36d6ba7cae\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-fhb78" Nov 02 06:37:47.610195 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:47.610013 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/933acc24-86ce-4896-8daa-6d36d6ba7cae-istio-data\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-fhb78\" (UID: \"933acc24-86ce-4896-8daa-6d36d6ba7cae\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-fhb78" Nov 02 06:37:47.610195 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:47.610078 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/933acc24-86ce-4896-8daa-6d36d6ba7cae-istiod-ca-cert\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-fhb78\" (UID: \"933acc24-86ce-4896-8daa-6d36d6ba7cae\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-fhb78" Nov 02 06:37:47.610195 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:47.610149 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/933acc24-86ce-4896-8daa-6d36d6ba7cae-workload-socket\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-fhb78\" (UID: \"933acc24-86ce-4896-8daa-6d36d6ba7cae\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-fhb78" Nov 02 06:37:47.611101 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:47.610235 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/933acc24-86ce-4896-8daa-6d36d6ba7cae-istio-podinfo\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-fhb78\" (UID: \"933acc24-86ce-4896-8daa-6d36d6ba7cae\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-fhb78" Nov 02 06:37:47.611101 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:47.610279 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xx472\" (UniqueName: \"kubernetes.io/projected/933acc24-86ce-4896-8daa-6d36d6ba7cae-kube-api-access-xx472\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-fhb78\" (UID: \"933acc24-86ce-4896-8daa-6d36d6ba7cae\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-fhb78" Nov 02 06:37:47.611101 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:47.611084 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/933acc24-86ce-4896-8daa-6d36d6ba7cae-credential-socket\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-fhb78\" (UID: \"933acc24-86ce-4896-8daa-6d36d6ba7cae\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-fhb78" Nov 02 06:37:47.611621 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:47.611598 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/933acc24-86ce-4896-8daa-6d36d6ba7cae-workload-certs\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-fhb78\" (UID: \"933acc24-86ce-4896-8daa-6d36d6ba7cae\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-fhb78" Nov 02 06:37:47.612659 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:47.612627 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/933acc24-86ce-4896-8daa-6d36d6ba7cae-istiod-ca-cert\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-fhb78\" (UID: \"933acc24-86ce-4896-8daa-6d36d6ba7cae\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-fhb78" Nov 02 06:37:47.613013 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:47.612953 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/933acc24-86ce-4896-8daa-6d36d6ba7cae-istio-data\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-fhb78\" (UID: \"933acc24-86ce-4896-8daa-6d36d6ba7cae\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-fhb78" Nov 02 06:37:47.613372 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:47.613258 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/933acc24-86ce-4896-8daa-6d36d6ba7cae-workload-socket\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-fhb78\" (UID: \"933acc24-86ce-4896-8daa-6d36d6ba7cae\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-fhb78" Nov 02 06:37:47.617758 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:47.617737 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/933acc24-86ce-4896-8daa-6d36d6ba7cae-istio-podinfo\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-fhb78\" (UID: \"933acc24-86ce-4896-8daa-6d36d6ba7cae\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-fhb78" Nov 02 06:37:47.620541 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:47.620514 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/933acc24-86ce-4896-8daa-6d36d6ba7cae-istio-envoy\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-fhb78\" (UID: \"933acc24-86ce-4896-8daa-6d36d6ba7cae\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-fhb78" Nov 02 06:37:47.666975 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:47.666268 5916 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-x2qpj" Nov 02 06:37:47.668164 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:47.667654 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-x2qpj" Nov 02 06:37:47.671809 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:47.671776 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-x2qpj" Nov 02 06:37:47.735232 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:47.722768 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xx472\" (UniqueName: \"kubernetes.io/projected/933acc24-86ce-4896-8daa-6d36d6ba7cae-kube-api-access-xx472\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-fhb78\" (UID: \"933acc24-86ce-4896-8daa-6d36d6ba7cae\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-fhb78" Nov 02 06:37:47.736016 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:47.735980 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/933acc24-86ce-4896-8daa-6d36d6ba7cae-istio-token\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-fhb78\" (UID: \"933acc24-86ce-4896-8daa-6d36d6ba7cae\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-fhb78" Nov 02 06:37:47.759702 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:47.759131 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-fhb78" Nov 02 06:37:48.335930 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:48.321060 5916 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 06:37:48.335930 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:48.321152 5916 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 06:37:48.335930 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:48.321201 5916 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 06:37:48.344938 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:48.338155 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-fhb78"] Nov 02 06:37:48.510528 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:48.510485 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-fhb78" event={"ID":"933acc24-86ce-4896-8daa-6d36d6ba7cae","Type":"ContainerStarted","Data":"c98800f64e8d6065fd240491b5f9eddcf816d58f2147026d07c909dc621a5cf8"} Nov 02 06:37:49.069735 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:49.069704 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b4c2887-fq7c4"] Nov 02 06:37:49.074663 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:49.074635 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b4c2887-fq7c4"] Nov 02 06:37:49.538117 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:49.538057 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-fhb78" event={"ID":"933acc24-86ce-4896-8daa-6d36d6ba7cae","Type":"ContainerStarted","Data":"82d9adc772f5aad1da662164df34432d566c2c35f74746d053c8600827ca565b"} Nov 02 06:37:49.668034 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:49.662794 5916 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-fhb78" podStartSLOduration=2.662775243 podStartE2EDuration="2.662775243s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:37:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:37:49.661739997 +0000 UTC m=+1257.933560391" watchObservedRunningTime="2026-11-02 06:37:49.662775243 +0000 UTC m=+1257.934595634" Nov 02 06:37:49.768324 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:49.767157 5916 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-fhb78" Nov 02 06:37:49.769956 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:49.768633 5916 patch_prober.go:29] interesting pod/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-fhb78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body= Nov 02 06:37:49.769956 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:49.768689 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-fhb78" podUID="933acc24-86ce-4896-8daa-6d36d6ba7cae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" Nov 02 06:37:49.908932 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:49.907829 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2bb59ace-463d-45b3-8201-7b13efd8b097" path="/var/lib/kubelet/pods/2bb59ace-463d-45b3-8201-7b13efd8b097/volumes" Nov 02 06:37:50.014131 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:37:50.014085 5916 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p98cq" containerName="extractor" Nov 02 06:37:50.770200 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:50.770158 5916 patch_prober.go:29] interesting pod/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-fhb78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body= Nov 02 06:37:50.781940 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:50.770858 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-fhb78" podUID="933acc24-86ce-4896-8daa-6d36d6ba7cae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" Nov 02 06:37:51.767968 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:51.764326 5916 patch_prober.go:29] interesting pod/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-fhb78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body= Nov 02 06:37:51.767968 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:51.764387 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-fhb78" podUID="933acc24-86ce-4896-8daa-6d36d6ba7cae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" Nov 02 06:37:52.416505 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:52.416469 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8e54376-fmfs7"] Nov 02 06:37:52.421982 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:52.421961 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8e54376-fmfs7" Nov 02 06:37:52.500749 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:52.490041 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8e54376-fmfs7"] Nov 02 06:37:52.510251 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:52.510216 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/539c3a80-fe0d-43b4-8d33-6d53597edd64-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f8e54376-fmfs7\" (UID: \"539c3a80-fe0d-43b4-8d33-6d53597edd64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8e54376-fmfs7" Nov 02 06:37:52.510412 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:52.510306 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krbj9\" (UniqueName: \"kubernetes.io/projected/539c3a80-fe0d-43b4-8d33-6d53597edd64-kube-api-access-krbj9\") pod \"maas-api-revoke-sub-f8e54376-fmfs7\" (UID: \"539c3a80-fe0d-43b4-8d33-6d53597edd64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8e54376-fmfs7" Nov 02 06:37:52.510412 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:52.510384 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/539c3a80-fe0d-43b4-8d33-6d53597edd64-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f8e54376-fmfs7\" (UID: \"539c3a80-fe0d-43b4-8d33-6d53597edd64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8e54376-fmfs7" Nov 02 06:37:52.548582 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:52.548549 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 06:37:52.548935 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:52.548918 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-68l4r\"" Nov 02 06:37:52.616604 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:52.616564 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/539c3a80-fe0d-43b4-8d33-6d53597edd64-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f8e54376-fmfs7\" (UID: \"539c3a80-fe0d-43b4-8d33-6d53597edd64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8e54376-fmfs7" Nov 02 06:37:52.616794 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:52.616622 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krbj9\" (UniqueName: \"kubernetes.io/projected/539c3a80-fe0d-43b4-8d33-6d53597edd64-kube-api-access-krbj9\") pod \"maas-api-revoke-sub-f8e54376-fmfs7\" (UID: \"539c3a80-fe0d-43b4-8d33-6d53597edd64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8e54376-fmfs7" Nov 02 06:37:52.616794 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:52.616710 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/539c3a80-fe0d-43b4-8d33-6d53597edd64-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f8e54376-fmfs7\" (UID: \"539c3a80-fe0d-43b4-8d33-6d53597edd64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8e54376-fmfs7" Nov 02 06:37:52.618734 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:52.618709 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/539c3a80-fe0d-43b4-8d33-6d53597edd64-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f8e54376-fmfs7\" (UID: \"539c3a80-fe0d-43b4-8d33-6d53597edd64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8e54376-fmfs7" Nov 02 06:37:52.619364 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:52.619342 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/539c3a80-fe0d-43b4-8d33-6d53597edd64-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f8e54376-fmfs7\" (UID: \"539c3a80-fe0d-43b4-8d33-6d53597edd64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8e54376-fmfs7" Nov 02 06:37:52.681180 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:52.681095 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-krbj9\" (UniqueName: \"kubernetes.io/projected/539c3a80-fe0d-43b4-8d33-6d53597edd64-kube-api-access-krbj9\") pod \"maas-api-revoke-sub-f8e54376-fmfs7\" (UID: \"539c3a80-fe0d-43b4-8d33-6d53597edd64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8e54376-fmfs7" Nov 02 06:37:52.762930 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:52.754119 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8e54376-fmfs7" Nov 02 06:37:52.766929 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:52.763284 5916 patch_prober.go:29] interesting pod/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-fhb78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body= Nov 02 06:37:52.766929 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:52.763341 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-fhb78" podUID="933acc24-86ce-4896-8daa-6d36d6ba7cae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" Nov 02 06:37:53.014696 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:52.990983 5916 scope.go:122] "RemoveContainer" containerID="68a7ddae2c1c7b7426e8cdbbf57b8aad43e39dbc5c977f4ff7295f0aeffe4f8d" Nov 02 06:37:53.014696 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:53.011845 5916 scope.go:122] "RemoveContainer" containerID="e07273572efe74b597fc0076dd6f3b83339bf01f3887eba92bd530782e648656" Nov 02 06:37:53.121880 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:53.121849 5916 scope.go:122] "RemoveContainer" containerID="120ff91eac2d344fea415a86f04056aa7cc4582284642bf5ffe6cd8c3ffcc407" Nov 02 06:37:53.123526 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:53.122881 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a55c94e5-tw9c2"] Nov 02 06:37:53.183216 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:53.180451 5916 scope.go:122] "RemoveContainer" containerID="ad38a11dd46e5d45d23464943dba024e94ab0eebd95a454871394b55ff7652cb" Nov 02 06:37:53.184708 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:53.184670 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a55c94e5-tw9c2"] Nov 02 06:37:53.200764 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:53.200740 5916 scope.go:122] "RemoveContainer" containerID="c04c432224fb63e1ab763aff16faa05df87cc3ad667aa4c5265d0a0fadc1e75c" Nov 02 06:37:53.474011 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:53.473522 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8e54376-fmfs7"] Nov 02 06:37:53.787934 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:53.785509 5916 patch_prober.go:29] interesting pod/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-fhb78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body= Nov 02 06:37:53.787934 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:53.785601 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-fhb78" podUID="933acc24-86ce-4896-8daa-6d36d6ba7cae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" Nov 02 06:37:53.821329 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:53.812019 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8e54376-fmfs7" event={"ID":"539c3a80-fe0d-43b4-8d33-6d53597edd64","Type":"ContainerStarted","Data":"83ef36fbe59614f41e071895d43adad56064501acc4e6b4bc5a6082c2894471c"} Nov 02 06:37:53.914214 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:53.914179 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bcf9063a-1c58-445e-91bf-cccca7ee34b2" path="/var/lib/kubelet/pods/bcf9063a-1c58-445e-91bf-cccca7ee34b2/volumes" Nov 02 06:37:54.761929 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:54.761855 5916 patch_prober.go:29] interesting pod/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-fhb78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body= Nov 02 06:37:54.762437 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:54.762200 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-fhb78" podUID="933acc24-86ce-4896-8daa-6d36d6ba7cae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" Nov 02 06:37:54.822853 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:54.822809 5916 generic.go:356] "Generic (PLEG): container finished" podID="539c3a80-fe0d-43b4-8d33-6d53597edd64" containerID="cf59f9897a09cc43f9bda887f20860ec32774e23ec3bfa1fd0e72405ad794d57" exitCode=0 Nov 02 06:37:54.823136 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:54.822890 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8e54376-fmfs7" event={"ID":"539c3a80-fe0d-43b4-8d33-6d53597edd64","Type":"ContainerDied","Data":"cf59f9897a09cc43f9bda887f20860ec32774e23ec3bfa1fd0e72405ad794d57"} Nov 02 06:37:56.082830 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:56.082800 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8e54376-fmfs7" Nov 02 06:37:56.188545 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:56.188509 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/539c3a80-fe0d-43b4-8d33-6d53597edd64-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/539c3a80-fe0d-43b4-8d33-6d53597edd64-maas-api-openshift-ca\") pod \"539c3a80-fe0d-43b4-8d33-6d53597edd64\" (UID: \"539c3a80-fe0d-43b4-8d33-6d53597edd64\") " Nov 02 06:37:56.188545 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:56.188549 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/539c3a80-fe0d-43b4-8d33-6d53597edd64-kube-api-access-krbj9\" (UniqueName: \"kubernetes.io/projected/539c3a80-fe0d-43b4-8d33-6d53597edd64-kube-api-access-krbj9\") pod \"539c3a80-fe0d-43b4-8d33-6d53597edd64\" (UID: \"539c3a80-fe0d-43b4-8d33-6d53597edd64\") " Nov 02 06:37:56.189227 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:56.188607 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/539c3a80-fe0d-43b4-8d33-6d53597edd64-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/539c3a80-fe0d-43b4-8d33-6d53597edd64-maas-api-xks-ca\") pod \"539c3a80-fe0d-43b4-8d33-6d53597edd64\" (UID: \"539c3a80-fe0d-43b4-8d33-6d53597edd64\") " Nov 02 06:37:56.189227 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:56.188969 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/539c3a80-fe0d-43b4-8d33-6d53597edd64-maas-api-openshift-ca" pod "539c3a80-fe0d-43b4-8d33-6d53597edd64" (UID: "539c3a80-fe0d-43b4-8d33-6d53597edd64"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:37:56.191059 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:56.191031 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/539c3a80-fe0d-43b4-8d33-6d53597edd64-kube-api-access-krbj9" pod "539c3a80-fe0d-43b4-8d33-6d53597edd64" (UID: "539c3a80-fe0d-43b4-8d33-6d53597edd64"). InnerVolumeSpecName "kube-api-access-krbj9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:37:56.191504 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:56.191482 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/539c3a80-fe0d-43b4-8d33-6d53597edd64-maas-api-xks-ca" pod "539c3a80-fe0d-43b4-8d33-6d53597edd64" (UID: "539c3a80-fe0d-43b4-8d33-6d53597edd64"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:37:56.224238 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:56.224205 5916 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-fhb78" Nov 02 06:37:56.225621 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:56.225594 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-fhb78" Nov 02 06:37:56.226932 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:56.226887 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-fhb78" Nov 02 06:37:56.290630 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:56.290588 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/539c3a80-fe0d-43b4-8d33-6d53597edd64-maas-api-openshift-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:37:56.290630 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:56.290630 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krbj9\" (UniqueName: \"kubernetes.io/projected/539c3a80-fe0d-43b4-8d33-6d53597edd64-kube-api-access-krbj9\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:37:56.290880 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:56.290646 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/539c3a80-fe0d-43b4-8d33-6d53597edd64-maas-api-xks-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:37:56.846300 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:56.846271 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8e54376-fmfs7" Nov 02 06:37:56.846300 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:56.846293 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8e54376-fmfs7" event={"ID":"539c3a80-fe0d-43b4-8d33-6d53597edd64","Type":"ContainerDied","Data":"83ef36fbe59614f41e071895d43adad56064501acc4e6b4bc5a6082c2894471c"} Nov 02 06:37:56.846524 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:37:56.846346 5916 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="83ef36fbe59614f41e071895d43adad56064501acc4e6b4bc5a6082c2894471c" Nov 02 06:38:01.432990 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:01.432881 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-w4bhs"] Nov 02 06:38:01.433802 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:01.433782 5916 cpu_manager.go:414] "containerMap: removing container" podUID="539c3a80-fe0d-43b4-8d33-6d53597edd64" containerName="revoke-keys" Nov 02 06:38:01.433886 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:01.433803 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="539c3a80-fe0d-43b4-8d33-6d53597edd64" containerName="revoke-keys" Nov 02 06:38:01.433960 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:01.433948 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="539c3a80-fe0d-43b4-8d33-6d53597edd64" containerName="revoke-keys" Nov 02 06:38:01.439779 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:01.439755 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-w4bhs" Nov 02 06:38:01.490238 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:01.490206 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-flk8r"] Nov 02 06:38:01.495857 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:01.495831 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-flk8r" Nov 02 06:38:01.537515 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:01.537481 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c10335d-e121-4ccd-949d-964cf21650ca-istio-token\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-flk8r\" (UID: \"4c10335d-e121-4ccd-949d-964cf21650ca\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-flk8r" Nov 02 06:38:01.537515 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:01.537521 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb1d132e-7237-4c50-b70b-8f97862d20c3-workload-certs\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-w4bhs\" (UID: \"eb1d132e-7237-4c50-b70b-8f97862d20c3\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-w4bhs" Nov 02 06:38:01.537719 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:01.537539 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wrdx\" (UniqueName: \"kubernetes.io/projected/4c10335d-e121-4ccd-949d-964cf21650ca-kube-api-access-5wrdx\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-flk8r\" (UID: \"4c10335d-e121-4ccd-949d-964cf21650ca\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-flk8r" Nov 02 06:38:01.537719 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:01.537603 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c10335d-e121-4ccd-949d-964cf21650ca-istio-envoy\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-flk8r\" (UID: \"4c10335d-e121-4ccd-949d-964cf21650ca\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-flk8r" Nov 02 06:38:01.537719 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:01.537678 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb1d132e-7237-4c50-b70b-8f97862d20c3-istio-podinfo\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-w4bhs\" (UID: \"eb1d132e-7237-4c50-b70b-8f97862d20c3\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-w4bhs" Nov 02 06:38:01.537834 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:01.537746 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb1d132e-7237-4c50-b70b-8f97862d20c3-istio-data\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-w4bhs\" (UID: \"eb1d132e-7237-4c50-b70b-8f97862d20c3\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-w4bhs" Nov 02 06:38:01.537834 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:01.537785 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c10335d-e121-4ccd-949d-964cf21650ca-workload-certs\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-flk8r\" (UID: \"4c10335d-e121-4ccd-949d-964cf21650ca\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-flk8r" Nov 02 06:38:01.537834 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:01.537818 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c10335d-e121-4ccd-949d-964cf21650ca-istio-data\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-flk8r\" (UID: \"4c10335d-e121-4ccd-949d-964cf21650ca\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-flk8r" Nov 02 06:38:01.538001 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:01.537850 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb1d132e-7237-4c50-b70b-8f97862d20c3-istio-envoy\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-w4bhs\" (UID: \"eb1d132e-7237-4c50-b70b-8f97862d20c3\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-w4bhs" Nov 02 06:38:01.538001 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:01.537868 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb1d132e-7237-4c50-b70b-8f97862d20c3-workload-socket\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-w4bhs\" (UID: \"eb1d132e-7237-4c50-b70b-8f97862d20c3\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-w4bhs" Nov 02 06:38:01.538001 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:01.537942 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c10335d-e121-4ccd-949d-964cf21650ca-workload-socket\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-flk8r\" (UID: \"4c10335d-e121-4ccd-949d-964cf21650ca\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-flk8r" Nov 02 06:38:01.538159 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:01.538089 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb1d132e-7237-4c50-b70b-8f97862d20c3-istio-token\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-w4bhs\" (UID: \"eb1d132e-7237-4c50-b70b-8f97862d20c3\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-w4bhs" Nov 02 06:38:01.538159 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:01.538129 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c10335d-e121-4ccd-949d-964cf21650ca-istio-podinfo\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-flk8r\" (UID: \"4c10335d-e121-4ccd-949d-964cf21650ca\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-flk8r" Nov 02 06:38:01.538255 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:01.538166 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb1d132e-7237-4c50-b70b-8f97862d20c3-credential-socket\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-w4bhs\" (UID: \"eb1d132e-7237-4c50-b70b-8f97862d20c3\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-w4bhs" Nov 02 06:38:01.538312 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:01.538296 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c10335d-e121-4ccd-949d-964cf21650ca-istiod-ca-cert\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-flk8r\" (UID: \"4c10335d-e121-4ccd-949d-964cf21650ca\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-flk8r" Nov 02 06:38:01.538350 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:01.538330 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb1d132e-7237-4c50-b70b-8f97862d20c3-istiod-ca-cert\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-w4bhs\" (UID: \"eb1d132e-7237-4c50-b70b-8f97862d20c3\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-w4bhs" Nov 02 06:38:01.538404 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:01.538351 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqplr\" (UniqueName: \"kubernetes.io/projected/eb1d132e-7237-4c50-b70b-8f97862d20c3-kube-api-access-pqplr\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-w4bhs\" (UID: \"eb1d132e-7237-4c50-b70b-8f97862d20c3\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-w4bhs" Nov 02 06:38:01.538445 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:01.538434 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c10335d-e121-4ccd-949d-964cf21650ca-credential-socket\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-flk8r\" (UID: \"4c10335d-e121-4ccd-949d-964cf21650ca\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-flk8r" Nov 02 06:38:01.540043 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:01.540020 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-w4bhs"] Nov 02 06:38:01.590815 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:01.590783 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-25spl"] Nov 02 06:38:01.595740 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:01.595714 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-25spl" Nov 02 06:38:01.597887 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:01.597853 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-flk8r"] Nov 02 06:38:01.640226 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:01.640184 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c10335d-e121-4ccd-949d-964cf21650ca-workload-certs\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-flk8r\" (UID: \"4c10335d-e121-4ccd-949d-964cf21650ca\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-flk8r" Nov 02 06:38:01.640391 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:01.640245 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3818427d-4da9-4347-8a90-83e34f7ed19d-workload-socket\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-25spl\" (UID: \"3818427d-4da9-4347-8a90-83e34f7ed19d\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-25spl" Nov 02 06:38:01.640391 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:01.640291 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c10335d-e121-4ccd-949d-964cf21650ca-istio-data\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-flk8r\" (UID: \"4c10335d-e121-4ccd-949d-964cf21650ca\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-flk8r" Nov 02 06:38:01.640391 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:01.640315 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3818427d-4da9-4347-8a90-83e34f7ed19d-istiod-ca-cert\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-25spl\" (UID: \"3818427d-4da9-4347-8a90-83e34f7ed19d\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-25spl" Nov 02 06:38:01.640391 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:01.640366 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb1d132e-7237-4c50-b70b-8f97862d20c3-istio-envoy\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-w4bhs\" (UID: \"eb1d132e-7237-4c50-b70b-8f97862d20c3\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-w4bhs" Nov 02 06:38:01.640610 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:01.640389 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwjbv\" (UniqueName: \"kubernetes.io/projected/3818427d-4da9-4347-8a90-83e34f7ed19d-kube-api-access-cwjbv\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-25spl\" (UID: \"3818427d-4da9-4347-8a90-83e34f7ed19d\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-25spl" Nov 02 06:38:01.640610 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:01.640422 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb1d132e-7237-4c50-b70b-8f97862d20c3-workload-socket\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-w4bhs\" (UID: \"eb1d132e-7237-4c50-b70b-8f97862d20c3\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-w4bhs" Nov 02 06:38:01.640610 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:01.640518 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c10335d-e121-4ccd-949d-964cf21650ca-workload-socket\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-flk8r\" (UID: \"4c10335d-e121-4ccd-949d-964cf21650ca\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-flk8r" Nov 02 06:38:01.640610 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:01.640573 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb1d132e-7237-4c50-b70b-8f97862d20c3-istio-token\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-w4bhs\" (UID: \"eb1d132e-7237-4c50-b70b-8f97862d20c3\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-w4bhs" Nov 02 06:38:01.640610 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:01.640605 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c10335d-e121-4ccd-949d-964cf21650ca-istio-podinfo\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-flk8r\" (UID: \"4c10335d-e121-4ccd-949d-964cf21650ca\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-flk8r" Nov 02 06:38:01.640853 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:01.640637 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb1d132e-7237-4c50-b70b-8f97862d20c3-credential-socket\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-w4bhs\" (UID: \"eb1d132e-7237-4c50-b70b-8f97862d20c3\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-w4bhs" Nov 02 06:38:01.640853 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:01.640676 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3818427d-4da9-4347-8a90-83e34f7ed19d-workload-certs\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-25spl\" (UID: \"3818427d-4da9-4347-8a90-83e34f7ed19d\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-25spl" Nov 02 06:38:01.640853 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:01.640742 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3818427d-4da9-4347-8a90-83e34f7ed19d-istio-envoy\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-25spl\" (UID: \"3818427d-4da9-4347-8a90-83e34f7ed19d\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-25spl" Nov 02 06:38:01.640853 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:01.640817 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c10335d-e121-4ccd-949d-964cf21650ca-istiod-ca-cert\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-flk8r\" (UID: \"4c10335d-e121-4ccd-949d-964cf21650ca\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-flk8r" Nov 02 06:38:01.641078 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:01.640862 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb1d132e-7237-4c50-b70b-8f97862d20c3-istiod-ca-cert\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-w4bhs\" (UID: \"eb1d132e-7237-4c50-b70b-8f97862d20c3\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-w4bhs" Nov 02 06:38:01.641078 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:01.640891 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3818427d-4da9-4347-8a90-83e34f7ed19d-istio-token\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-25spl\" (UID: \"3818427d-4da9-4347-8a90-83e34f7ed19d\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-25spl" Nov 02 06:38:01.641078 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:01.640975 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqplr\" (UniqueName: \"kubernetes.io/projected/eb1d132e-7237-4c50-b70b-8f97862d20c3-kube-api-access-pqplr\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-w4bhs\" (UID: \"eb1d132e-7237-4c50-b70b-8f97862d20c3\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-w4bhs" Nov 02 06:38:01.641078 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:01.641027 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c10335d-e121-4ccd-949d-964cf21650ca-credential-socket\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-flk8r\" (UID: \"4c10335d-e121-4ccd-949d-964cf21650ca\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-flk8r" Nov 02 06:38:01.641078 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:01.641052 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3818427d-4da9-4347-8a90-83e34f7ed19d-credential-socket\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-25spl\" (UID: \"3818427d-4da9-4347-8a90-83e34f7ed19d\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-25spl" Nov 02 06:38:01.641326 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:01.641077 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3818427d-4da9-4347-8a90-83e34f7ed19d-istio-data\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-25spl\" (UID: \"3818427d-4da9-4347-8a90-83e34f7ed19d\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-25spl" Nov 02 06:38:01.641326 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:01.641102 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3818427d-4da9-4347-8a90-83e34f7ed19d-istio-podinfo\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-25spl\" (UID: \"3818427d-4da9-4347-8a90-83e34f7ed19d\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-25spl" Nov 02 06:38:01.641326 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:01.641167 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c10335d-e121-4ccd-949d-964cf21650ca-istio-token\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-flk8r\" (UID: \"4c10335d-e121-4ccd-949d-964cf21650ca\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-flk8r" Nov 02 06:38:01.641326 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:01.641199 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb1d132e-7237-4c50-b70b-8f97862d20c3-workload-certs\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-w4bhs\" (UID: \"eb1d132e-7237-4c50-b70b-8f97862d20c3\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-w4bhs" Nov 02 06:38:01.641326 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:01.641233 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wrdx\" (UniqueName: \"kubernetes.io/projected/4c10335d-e121-4ccd-949d-964cf21650ca-kube-api-access-5wrdx\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-flk8r\" (UID: \"4c10335d-e121-4ccd-949d-964cf21650ca\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-flk8r" Nov 02 06:38:01.641326 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:01.641265 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c10335d-e121-4ccd-949d-964cf21650ca-istio-envoy\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-flk8r\" (UID: \"4c10335d-e121-4ccd-949d-964cf21650ca\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-flk8r" Nov 02 06:38:01.641326 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:01.641296 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb1d132e-7237-4c50-b70b-8f97862d20c3-istio-podinfo\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-w4bhs\" (UID: \"eb1d132e-7237-4c50-b70b-8f97862d20c3\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-w4bhs" Nov 02 06:38:01.641657 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:01.641361 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb1d132e-7237-4c50-b70b-8f97862d20c3-istio-data\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-w4bhs\" (UID: \"eb1d132e-7237-4c50-b70b-8f97862d20c3\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-w4bhs" Nov 02 06:38:01.642634 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:01.641843 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb1d132e-7237-4c50-b70b-8f97862d20c3-istio-data\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-w4bhs\" (UID: \"eb1d132e-7237-4c50-b70b-8f97862d20c3\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-w4bhs" Nov 02 06:38:01.642634 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:01.642126 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb1d132e-7237-4c50-b70b-8f97862d20c3-workload-socket\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-w4bhs\" (UID: \"eb1d132e-7237-4c50-b70b-8f97862d20c3\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-w4bhs" Nov 02 06:38:01.645246 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:01.643300 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb1d132e-7237-4c50-b70b-8f97862d20c3-workload-certs\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-w4bhs\" (UID: \"eb1d132e-7237-4c50-b70b-8f97862d20c3\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-w4bhs" Nov 02 06:38:01.645246 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:01.643544 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb1d132e-7237-4c50-b70b-8f97862d20c3-credential-socket\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-w4bhs\" (UID: \"eb1d132e-7237-4c50-b70b-8f97862d20c3\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-w4bhs" Nov 02 06:38:01.645246 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:01.644398 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c10335d-e121-4ccd-949d-964cf21650ca-workload-certs\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-flk8r\" (UID: \"4c10335d-e121-4ccd-949d-964cf21650ca\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-flk8r" Nov 02 06:38:01.645246 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:01.644437 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb1d132e-7237-4c50-b70b-8f97862d20c3-istiod-ca-cert\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-w4bhs\" (UID: \"eb1d132e-7237-4c50-b70b-8f97862d20c3\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-w4bhs" Nov 02 06:38:01.645246 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:01.644669 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c10335d-e121-4ccd-949d-964cf21650ca-istio-data\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-flk8r\" (UID: \"4c10335d-e121-4ccd-949d-964cf21650ca\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-flk8r" Nov 02 06:38:01.645246 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:01.644882 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c10335d-e121-4ccd-949d-964cf21650ca-workload-socket\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-flk8r\" (UID: \"4c10335d-e121-4ccd-949d-964cf21650ca\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-flk8r" Nov 02 06:38:01.645246 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:01.645154 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c10335d-e121-4ccd-949d-964cf21650ca-credential-socket\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-flk8r\" (UID: \"4c10335d-e121-4ccd-949d-964cf21650ca\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-flk8r" Nov 02 06:38:01.647778 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:01.647600 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c10335d-e121-4ccd-949d-964cf21650ca-istiod-ca-cert\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-flk8r\" (UID: \"4c10335d-e121-4ccd-949d-964cf21650ca\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-flk8r" Nov 02 06:38:01.648900 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:01.648800 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb1d132e-7237-4c50-b70b-8f97862d20c3-istio-podinfo\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-w4bhs\" (UID: \"eb1d132e-7237-4c50-b70b-8f97862d20c3\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-w4bhs" Nov 02 06:38:01.657778 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:01.657750 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c10335d-e121-4ccd-949d-964cf21650ca-istio-envoy\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-flk8r\" (UID: \"4c10335d-e121-4ccd-949d-964cf21650ca\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-flk8r" Nov 02 06:38:01.659026 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:01.659002 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb1d132e-7237-4c50-b70b-8f97862d20c3-istio-envoy\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-w4bhs\" (UID: \"eb1d132e-7237-4c50-b70b-8f97862d20c3\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-w4bhs" Nov 02 06:38:01.660235 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:01.660212 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-25spl"] Nov 02 06:38:01.660941 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:01.660886 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c10335d-e121-4ccd-949d-964cf21650ca-istio-podinfo\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-flk8r\" (UID: \"4c10335d-e121-4ccd-949d-964cf21650ca\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-flk8r" Nov 02 06:38:01.711214 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:01.711048 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wrdx\" (UniqueName: \"kubernetes.io/projected/4c10335d-e121-4ccd-949d-964cf21650ca-kube-api-access-5wrdx\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-flk8r\" (UID: \"4c10335d-e121-4ccd-949d-964cf21650ca\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-flk8r" Nov 02 06:38:01.711214 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:01.711104 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c10335d-e121-4ccd-949d-964cf21650ca-istio-token\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-flk8r\" (UID: \"4c10335d-e121-4ccd-949d-964cf21650ca\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-flk8r" Nov 02 06:38:01.711214 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:01.711126 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqplr\" (UniqueName: \"kubernetes.io/projected/eb1d132e-7237-4c50-b70b-8f97862d20c3-kube-api-access-pqplr\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-w4bhs\" (UID: \"eb1d132e-7237-4c50-b70b-8f97862d20c3\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-w4bhs" Nov 02 06:38:01.712716 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:01.712697 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb1d132e-7237-4c50-b70b-8f97862d20c3-istio-token\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-w4bhs\" (UID: \"eb1d132e-7237-4c50-b70b-8f97862d20c3\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-w4bhs" Nov 02 06:38:01.742627 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:01.742589 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3818427d-4da9-4347-8a90-83e34f7ed19d-credential-socket\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-25spl\" (UID: \"3818427d-4da9-4347-8a90-83e34f7ed19d\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-25spl" Nov 02 06:38:01.742760 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:01.742638 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3818427d-4da9-4347-8a90-83e34f7ed19d-istio-data\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-25spl\" (UID: \"3818427d-4da9-4347-8a90-83e34f7ed19d\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-25spl" Nov 02 06:38:01.742760 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:01.742662 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3818427d-4da9-4347-8a90-83e34f7ed19d-istio-podinfo\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-25spl\" (UID: \"3818427d-4da9-4347-8a90-83e34f7ed19d\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-25spl" Nov 02 06:38:01.742888 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:01.742799 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3818427d-4da9-4347-8a90-83e34f7ed19d-workload-socket\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-25spl\" (UID: \"3818427d-4da9-4347-8a90-83e34f7ed19d\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-25spl" Nov 02 06:38:01.742888 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:01.742847 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3818427d-4da9-4347-8a90-83e34f7ed19d-istiod-ca-cert\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-25spl\" (UID: \"3818427d-4da9-4347-8a90-83e34f7ed19d\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-25spl" Nov 02 06:38:01.743033 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:01.742899 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwjbv\" (UniqueName: \"kubernetes.io/projected/3818427d-4da9-4347-8a90-83e34f7ed19d-kube-api-access-cwjbv\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-25spl\" (UID: \"3818427d-4da9-4347-8a90-83e34f7ed19d\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-25spl" Nov 02 06:38:01.743033 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:01.742997 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3818427d-4da9-4347-8a90-83e34f7ed19d-istio-data\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-25spl\" (UID: \"3818427d-4da9-4347-8a90-83e34f7ed19d\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-25spl" Nov 02 06:38:01.743033 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:01.743005 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3818427d-4da9-4347-8a90-83e34f7ed19d-workload-certs\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-25spl\" (UID: \"3818427d-4da9-4347-8a90-83e34f7ed19d\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-25spl" Nov 02 06:38:01.743190 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:01.743065 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3818427d-4da9-4347-8a90-83e34f7ed19d-istio-envoy\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-25spl\" (UID: \"3818427d-4da9-4347-8a90-83e34f7ed19d\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-25spl" Nov 02 06:38:01.743190 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:01.743138 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3818427d-4da9-4347-8a90-83e34f7ed19d-istio-token\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-25spl\" (UID: \"3818427d-4da9-4347-8a90-83e34f7ed19d\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-25spl" Nov 02 06:38:01.743299 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:01.743283 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3818427d-4da9-4347-8a90-83e34f7ed19d-workload-certs\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-25spl\" (UID: \"3818427d-4da9-4347-8a90-83e34f7ed19d\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-25spl" Nov 02 06:38:01.743441 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:01.742998 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3818427d-4da9-4347-8a90-83e34f7ed19d-credential-socket\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-25spl\" (UID: \"3818427d-4da9-4347-8a90-83e34f7ed19d\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-25spl" Nov 02 06:38:01.743525 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:01.743502 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3818427d-4da9-4347-8a90-83e34f7ed19d-workload-socket\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-25spl\" (UID: \"3818427d-4da9-4347-8a90-83e34f7ed19d\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-25spl" Nov 02 06:38:01.743645 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:01.743624 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3818427d-4da9-4347-8a90-83e34f7ed19d-istiod-ca-cert\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-25spl\" (UID: \"3818427d-4da9-4347-8a90-83e34f7ed19d\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-25spl" Nov 02 06:38:01.745531 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:01.745489 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3818427d-4da9-4347-8a90-83e34f7ed19d-istio-envoy\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-25spl\" (UID: \"3818427d-4da9-4347-8a90-83e34f7ed19d\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-25spl" Nov 02 06:38:01.745707 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:01.745686 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3818427d-4da9-4347-8a90-83e34f7ed19d-istio-podinfo\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-25spl\" (UID: \"3818427d-4da9-4347-8a90-83e34f7ed19d\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-25spl" Nov 02 06:38:01.755631 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:01.755610 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-w4bhs" Nov 02 06:38:01.800743 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:01.800709 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwjbv\" (UniqueName: \"kubernetes.io/projected/3818427d-4da9-4347-8a90-83e34f7ed19d-kube-api-access-cwjbv\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-25spl\" (UID: \"3818427d-4da9-4347-8a90-83e34f7ed19d\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-25spl" Nov 02 06:38:01.800888 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:01.800865 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3818427d-4da9-4347-8a90-83e34f7ed19d-istio-token\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-25spl\" (UID: \"3818427d-4da9-4347-8a90-83e34f7ed19d\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-25spl" Nov 02 06:38:01.812482 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:01.812454 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-flk8r" Nov 02 06:38:01.866211 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:01.863252 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hbwqc"] Nov 02 06:38:01.875437 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:01.875415 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hbwqc" Nov 02 06:38:01.920432 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:01.920403 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-25spl" Nov 02 06:38:01.947558 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:01.947524 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hbwqc"] Nov 02 06:38:01.948217 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:01.948187 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2871e54e-8f8f-47d6-a16b-72d986b3835e-istio-data\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hbwqc\" (UID: \"2871e54e-8f8f-47d6-a16b-72d986b3835e\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hbwqc" Nov 02 06:38:01.948298 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:01.948272 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2871e54e-8f8f-47d6-a16b-72d986b3835e-workload-certs\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hbwqc\" (UID: \"2871e54e-8f8f-47d6-a16b-72d986b3835e\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hbwqc" Nov 02 06:38:01.948347 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:01.948307 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2871e54e-8f8f-47d6-a16b-72d986b3835e-istio-podinfo\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hbwqc\" (UID: \"2871e54e-8f8f-47d6-a16b-72d986b3835e\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hbwqc" Nov 02 06:38:01.948347 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:01.948333 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2871e54e-8f8f-47d6-a16b-72d986b3835e-istiod-ca-cert\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hbwqc\" (UID: \"2871e54e-8f8f-47d6-a16b-72d986b3835e\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hbwqc" Nov 02 06:38:01.948453 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:01.948382 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2871e54e-8f8f-47d6-a16b-72d986b3835e-istio-envoy\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hbwqc\" (UID: \"2871e54e-8f8f-47d6-a16b-72d986b3835e\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hbwqc" Nov 02 06:38:01.948453 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:01.948409 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjp57\" (UniqueName: \"kubernetes.io/projected/2871e54e-8f8f-47d6-a16b-72d986b3835e-kube-api-access-cjp57\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hbwqc\" (UID: \"2871e54e-8f8f-47d6-a16b-72d986b3835e\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hbwqc" Nov 02 06:38:01.948547 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:01.948508 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2871e54e-8f8f-47d6-a16b-72d986b3835e-credential-socket\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hbwqc\" (UID: \"2871e54e-8f8f-47d6-a16b-72d986b3835e\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hbwqc" Nov 02 06:38:01.948599 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:01.948548 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2871e54e-8f8f-47d6-a16b-72d986b3835e-workload-socket\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hbwqc\" (UID: \"2871e54e-8f8f-47d6-a16b-72d986b3835e\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hbwqc" Nov 02 06:38:01.948655 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:01.948623 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2871e54e-8f8f-47d6-a16b-72d986b3835e-istio-token\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hbwqc\" (UID: \"2871e54e-8f8f-47d6-a16b-72d986b3835e\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hbwqc" Nov 02 06:38:02.050230 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:02.049802 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2871e54e-8f8f-47d6-a16b-72d986b3835e-istio-data\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hbwqc\" (UID: \"2871e54e-8f8f-47d6-a16b-72d986b3835e\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hbwqc" Nov 02 06:38:02.050230 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:02.049958 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2871e54e-8f8f-47d6-a16b-72d986b3835e-workload-certs\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hbwqc\" (UID: \"2871e54e-8f8f-47d6-a16b-72d986b3835e\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hbwqc" Nov 02 06:38:02.050230 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:02.049996 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2871e54e-8f8f-47d6-a16b-72d986b3835e-istio-podinfo\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hbwqc\" (UID: \"2871e54e-8f8f-47d6-a16b-72d986b3835e\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hbwqc" Nov 02 06:38:02.050230 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:02.050028 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2871e54e-8f8f-47d6-a16b-72d986b3835e-istiod-ca-cert\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hbwqc\" (UID: \"2871e54e-8f8f-47d6-a16b-72d986b3835e\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hbwqc" Nov 02 06:38:02.050230 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:02.050074 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2871e54e-8f8f-47d6-a16b-72d986b3835e-istio-envoy\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hbwqc\" (UID: \"2871e54e-8f8f-47d6-a16b-72d986b3835e\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hbwqc" Nov 02 06:38:02.050230 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:02.050100 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjp57\" (UniqueName: \"kubernetes.io/projected/2871e54e-8f8f-47d6-a16b-72d986b3835e-kube-api-access-cjp57\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hbwqc\" (UID: \"2871e54e-8f8f-47d6-a16b-72d986b3835e\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hbwqc" Nov 02 06:38:02.050230 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:02.050194 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2871e54e-8f8f-47d6-a16b-72d986b3835e-credential-socket\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hbwqc\" (UID: \"2871e54e-8f8f-47d6-a16b-72d986b3835e\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hbwqc" Nov 02 06:38:02.050230 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:02.050232 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2871e54e-8f8f-47d6-a16b-72d986b3835e-workload-socket\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hbwqc\" (UID: \"2871e54e-8f8f-47d6-a16b-72d986b3835e\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hbwqc" Nov 02 06:38:02.050737 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:02.050292 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2871e54e-8f8f-47d6-a16b-72d986b3835e-istio-data\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hbwqc\" (UID: \"2871e54e-8f8f-47d6-a16b-72d986b3835e\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hbwqc" Nov 02 06:38:02.050737 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:02.050307 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2871e54e-8f8f-47d6-a16b-72d986b3835e-istio-token\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hbwqc\" (UID: \"2871e54e-8f8f-47d6-a16b-72d986b3835e\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hbwqc" Nov 02 06:38:02.051246 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:02.051040 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2871e54e-8f8f-47d6-a16b-72d986b3835e-credential-socket\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hbwqc\" (UID: \"2871e54e-8f8f-47d6-a16b-72d986b3835e\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hbwqc" Nov 02 06:38:02.051246 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:02.051121 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2871e54e-8f8f-47d6-a16b-72d986b3835e-workload-socket\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hbwqc\" (UID: \"2871e54e-8f8f-47d6-a16b-72d986b3835e\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hbwqc" Nov 02 06:38:02.051454 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:02.051368 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2871e54e-8f8f-47d6-a16b-72d986b3835e-workload-certs\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hbwqc\" (UID: \"2871e54e-8f8f-47d6-a16b-72d986b3835e\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hbwqc" Nov 02 06:38:02.051773 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:02.051720 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2871e54e-8f8f-47d6-a16b-72d986b3835e-istiod-ca-cert\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hbwqc\" (UID: \"2871e54e-8f8f-47d6-a16b-72d986b3835e\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hbwqc" Nov 02 06:38:02.054247 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:02.054218 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2871e54e-8f8f-47d6-a16b-72d986b3835e-istio-envoy\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hbwqc\" (UID: \"2871e54e-8f8f-47d6-a16b-72d986b3835e\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hbwqc" Nov 02 06:38:02.054481 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:02.054461 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2871e54e-8f8f-47d6-a16b-72d986b3835e-istio-podinfo\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hbwqc\" (UID: \"2871e54e-8f8f-47d6-a16b-72d986b3835e\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hbwqc" Nov 02 06:38:02.113428 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:02.113394 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjp57\" (UniqueName: \"kubernetes.io/projected/2871e54e-8f8f-47d6-a16b-72d986b3835e-kube-api-access-cjp57\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hbwqc\" (UID: \"2871e54e-8f8f-47d6-a16b-72d986b3835e\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hbwqc" Nov 02 06:38:02.113579 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:02.113407 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2871e54e-8f8f-47d6-a16b-72d986b3835e-istio-token\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hbwqc\" (UID: \"2871e54e-8f8f-47d6-a16b-72d986b3835e\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hbwqc" Nov 02 06:38:02.196536 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:38:02.196500 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb1d132e_7237_4c50_b70b_8f97862d20c3.slice/crio-d59ee75a833e6f8afb3dd1db766f9e47e04c9b59bd5ffb02c5fe9f9d88ce111d WatchSource:0}: Error finding container d59ee75a833e6f8afb3dd1db766f9e47e04c9b59bd5ffb02c5fe9f9d88ce111d: Status 404 returned error can't find the container with id d59ee75a833e6f8afb3dd1db766f9e47e04c9b59bd5ffb02c5fe9f9d88ce111d Nov 02 06:38:02.197435 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:02.197412 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hbwqc" Nov 02 06:38:02.198841 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:02.198813 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-w4bhs"] Nov 02 06:38:02.199462 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:02.199433 5916 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 06:38:02.199537 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:02.199491 5916 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 06:38:02.199537 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:02.199518 5916 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 06:38:02.254462 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:02.251357 5916 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 06:38:02.254462 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:02.251442 5916 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 06:38:02.254462 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:02.251473 5916 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 06:38:02.254462 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:02.252729 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-flk8r"] Nov 02 06:38:02.342350 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:02.342316 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-25spl"] Nov 02 06:38:02.356525 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:02.356492 5916 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 06:38:02.356708 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:02.356693 5916 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 06:38:02.356794 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:02.356783 5916 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 06:38:02.625247 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:02.625145 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hbwqc"] Nov 02 06:38:02.629166 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:02.629087 5916 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 06:38:02.629166 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:02.629149 5916 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 06:38:02.629345 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:02.629178 5916 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 06:38:02.965980 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:02.953656 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hbwqc" event={"ID":"2871e54e-8f8f-47d6-a16b-72d986b3835e","Type":"ContainerStarted","Data":"97aff99737a873d789e9cf6c6b6d78eb202c7fd93c2df833cafd1dc7497e7005"} Nov 02 06:38:02.965980 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:02.953740 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hbwqc" event={"ID":"2871e54e-8f8f-47d6-a16b-72d986b3835e","Type":"ContainerStarted","Data":"6606399d4fcd046a8a4a6cc17e36192072421560ec31d6b1a675e108c709c540"} Nov 02 06:38:02.965980 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:02.957938 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-25spl" event={"ID":"3818427d-4da9-4347-8a90-83e34f7ed19d","Type":"ContainerStarted","Data":"6fb33bb629b56824d71e353bda7d0af5f016efc68d498ac0ce4736efda682b08"} Nov 02 06:38:02.965980 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:02.957996 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-25spl" event={"ID":"3818427d-4da9-4347-8a90-83e34f7ed19d","Type":"ContainerStarted","Data":"bcbf8a12ca30e35cc71f078c86a7460044efa08161cdd00ac67ab1ec61d6090d"} Nov 02 06:38:02.966341 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:02.966029 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-flk8r" event={"ID":"4c10335d-e121-4ccd-949d-964cf21650ca","Type":"ContainerStarted","Data":"c11f67bbf7a865f85db82cdeae7847c747172a05959bbe9cd68e14ba886a53b3"} Nov 02 06:38:02.966341 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:02.966090 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-flk8r" event={"ID":"4c10335d-e121-4ccd-949d-964cf21650ca","Type":"ContainerStarted","Data":"b69bc230833ed96783908f62494c00f475d26791f1ad6b0ba14836afb6935c61"} Nov 02 06:38:02.976925 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:02.971797 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-w4bhs" event={"ID":"eb1d132e-7237-4c50-b70b-8f97862d20c3","Type":"ContainerStarted","Data":"43c8d7a3bf910eefad89dc39e2172de86c4222c34f2651eca5f8f11f9ab43cf6"} Nov 02 06:38:02.976925 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:02.971852 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-w4bhs" event={"ID":"eb1d132e-7237-4c50-b70b-8f97862d20c3","Type":"ContainerStarted","Data":"d59ee75a833e6f8afb3dd1db766f9e47e04c9b59bd5ffb02c5fe9f9d88ce111d"} Nov 02 06:38:03.202927 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:03.191634 5916 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hbwqc" podStartSLOduration=2.191609754 podStartE2EDuration="2.191609754s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:38:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:38:03.119820088 +0000 UTC m=+1271.391640478" watchObservedRunningTime="2026-11-02 06:38:03.191609754 +0000 UTC m=+1271.463430149" Nov 02 06:38:03.208438 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:03.207842 5916 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hbwqc" Nov 02 06:38:03.208438 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:03.208168 5916 patch_prober.go:29] interesting pod/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hbwqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Nov 02 06:38:03.208438 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:03.208207 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hbwqc" podUID="2871e54e-8f8f-47d6-a16b-72d986b3835e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Nov 02 06:38:03.400484 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:03.400449 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7de0cfea-dcgfp"] Nov 02 06:38:03.468251 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:03.437202 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7de0cfea-dcgfp" Nov 02 06:38:03.495373 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:03.495297 5916 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-flk8r" podStartSLOduration=2.495277969 podStartE2EDuration="2.495277969s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:38:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:38:03.342196488 +0000 UTC m=+1271.614016883" watchObservedRunningTime="2026-11-02 06:38:03.495277969 +0000 UTC m=+1271.767098364" Nov 02 06:38:03.528939 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:03.497501 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 06:38:03.528939 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:03.493963 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-68l4r\"" Nov 02 06:38:03.565928 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:03.559179 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqjhk\" (UniqueName: \"kubernetes.io/projected/851e0293-99c4-4ed8-a9ce-7bff1204e30e-kube-api-access-zqjhk\") pod \"maas-api-revoke-sub-7de0cfea-dcgfp\" (UID: \"851e0293-99c4-4ed8-a9ce-7bff1204e30e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7de0cfea-dcgfp" Nov 02 06:38:03.565928 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:03.559397 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/851e0293-99c4-4ed8-a9ce-7bff1204e30e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7de0cfea-dcgfp\" (UID: \"851e0293-99c4-4ed8-a9ce-7bff1204e30e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7de0cfea-dcgfp" Nov 02 06:38:03.565928 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:03.559453 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/851e0293-99c4-4ed8-a9ce-7bff1204e30e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7de0cfea-dcgfp\" (UID: \"851e0293-99c4-4ed8-a9ce-7bff1204e30e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7de0cfea-dcgfp" Nov 02 06:38:03.704033 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:03.661661 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqjhk\" (UniqueName: \"kubernetes.io/projected/851e0293-99c4-4ed8-a9ce-7bff1204e30e-kube-api-access-zqjhk\") pod \"maas-api-revoke-sub-7de0cfea-dcgfp\" (UID: \"851e0293-99c4-4ed8-a9ce-7bff1204e30e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7de0cfea-dcgfp" Nov 02 06:38:03.704033 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:03.661824 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/851e0293-99c4-4ed8-a9ce-7bff1204e30e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7de0cfea-dcgfp\" (UID: \"851e0293-99c4-4ed8-a9ce-7bff1204e30e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7de0cfea-dcgfp" Nov 02 06:38:03.704033 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:03.661865 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/851e0293-99c4-4ed8-a9ce-7bff1204e30e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7de0cfea-dcgfp\" (UID: \"851e0293-99c4-4ed8-a9ce-7bff1204e30e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7de0cfea-dcgfp" Nov 02 06:38:03.704033 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:03.662722 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/851e0293-99c4-4ed8-a9ce-7bff1204e30e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7de0cfea-dcgfp\" (UID: \"851e0293-99c4-4ed8-a9ce-7bff1204e30e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7de0cfea-dcgfp" Nov 02 06:38:03.721646 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:03.720399 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/851e0293-99c4-4ed8-a9ce-7bff1204e30e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7de0cfea-dcgfp\" (UID: \"851e0293-99c4-4ed8-a9ce-7bff1204e30e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7de0cfea-dcgfp" Nov 02 06:38:03.741966 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:03.721985 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7de0cfea-dcgfp"] Nov 02 06:38:03.761996 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:03.758286 5916 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-w4bhs" Nov 02 06:38:03.783973 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:03.783500 5916 patch_prober.go:29] interesting pod/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-w4bhs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Nov 02 06:38:03.783973 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:03.783563 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-w4bhs" podUID="eb1d132e-7237-4c50-b70b-8f97862d20c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Nov 02 06:38:03.808035 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:03.799272 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqjhk\" (UniqueName: \"kubernetes.io/projected/851e0293-99c4-4ed8-a9ce-7bff1204e30e-kube-api-access-zqjhk\") pod \"maas-api-revoke-sub-7de0cfea-dcgfp\" (UID: \"851e0293-99c4-4ed8-a9ce-7bff1204e30e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7de0cfea-dcgfp" Nov 02 06:38:03.905823 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:03.815982 5916 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-flk8r" Nov 02 06:38:03.905823 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:03.829085 5916 patch_prober.go:29] interesting pod/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-flk8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Nov 02 06:38:03.905823 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:03.829138 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-flk8r" podUID="4c10335d-e121-4ccd-949d-964cf21650ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Nov 02 06:38:03.969950 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:03.899109 5916 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-w4bhs" podStartSLOduration=2.899086992 podStartE2EDuration="2.899086992s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:38:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:38:03.512981389 +0000 UTC m=+1271.784801782" watchObservedRunningTime="2026-11-02 06:38:03.899086992 +0000 UTC m=+1272.170907386" Nov 02 06:38:03.969950 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:03.955194 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7de0cfea-dcgfp" Nov 02 06:38:04.079176 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:04.079137 5916 patch_prober.go:29] interesting pod/e2e-worker-w2-581608-openshift-default-cd7c896df-25spl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body= Nov 02 06:38:04.079447 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:04.079418 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-25spl" podUID="3818427d-4da9-4347-8a90-83e34f7ed19d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" Nov 02 06:38:04.138794 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:04.138716 5916 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-25spl" podStartSLOduration=3.138691967 podStartE2EDuration="3.138691967s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:38:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:38:03.660538382 +0000 UTC m=+1271.932358777" watchObservedRunningTime="2026-11-02 06:38:04.138691967 +0000 UTC m=+1272.410512361" Nov 02 06:38:04.257157 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:04.257115 5916 patch_prober.go:29] interesting pod/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hbwqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Nov 02 06:38:04.257398 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:04.257371 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hbwqc" podUID="2871e54e-8f8f-47d6-a16b-72d986b3835e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Nov 02 06:38:04.277928 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:04.273190 5916 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-25spl" Nov 02 06:38:04.320702 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:04.320659 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dcf9c3/distinct-2-w5-dcf9c3-kserve-64747dbd8f-wlwng" podUID="e37d42ab-7b62-4f07-8a29-cdedeb0fe89a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:04.451558 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:04.451513 5916 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-dcf9c3/distinct-2-w5-dcf9c3-kserve-64747dbd8f-wlwng" podUID="e37d42ab-7b62-4f07-8a29-cdedeb0fe89a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:04.476718 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:04.452477 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-581608/distinct-w2-581608-kserve-6bb6bb7b65-v7kfj" podUID="dcf2f562-3c50-4de7-9c84-cb659d3bc25b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded" Nov 02 06:38:04.517105 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:04.517012 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fb1398/distinct-2-w3-fb1398-kserve-597b7dc99b-dp4jf" podUID="523a31c4-4aea-4e9f-9ec4-7c28faefbf7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:04.565049 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:04.565001 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-680b7b/simulator-w4-680b7b-kserve-6c4f574c56-p5xtc" podUID="be8a1039-5934-4885-93fe-070575f936ec" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:04.737044 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:04.736997 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bdcd80/distinct-w0-bdcd80-kserve-5d9df79d48-bn5d4" podUID="1a7c7a03-e0cf-44ca-8eee-6c3e16112b3a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:04.757346 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:04.757302 5916 patch_prober.go:29] interesting pod/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-w4bhs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Nov 02 06:38:04.766930 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:04.759314 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-w4bhs" podUID="eb1d132e-7237-4c50-b70b-8f97862d20c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Nov 02 06:38:04.812930 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:04.765047 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6ff85e/simulator-w6-6ff85e-kserve-9b9fd978f-q76nl" podUID="30114285-d730-4abb-a618-6833d5746feb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:04.850224 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:04.849019 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cdfce9/simulator-w1-cdfce9-kserve-7ffd68f78-5vwl2" podUID="b8b219aa-6a59-4d31-96f6-8d50ab910aca" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Nov 02 06:38:04.900973 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:04.897109 5916 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-581608/distinct-2-w2-581608-kserve-57dd49c4b-qnt9k" podUID="837780ad-8ea3-4ed2-922f-917cffc7bb93" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:04.953948 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:04.945674 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6ff85e/premium-w6-6ff85e-kserve-59848ff6d4-szbxx" podUID="eb386976-f898-4f1d-b27c-77bccc735a3b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:05.007852 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:05.006142 5916 patch_prober.go:29] interesting pod/e2e-worker-w2-581608-openshift-default-cd7c896df-25spl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body= Nov 02 06:38:05.007852 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:05.006200 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-25spl" podUID="3818427d-4da9-4347-8a90-83e34f7ed19d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" Nov 02 06:38:05.053984 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:05.050082 5916 patch_prober.go:29] interesting pod/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-flk8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Nov 02 06:38:05.053984 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:05.050141 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-flk8r" podUID="4c10335d-e121-4ccd-949d-964cf21650ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Nov 02 06:38:05.519184 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:05.519116 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-581608/simulator-w2-581608-kserve-68f68d8559-xkcwp" podUID="529e25e6-4479-49d6-85c7-85dad6c2193e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Nov 02 06:38:05.520635 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:05.519480 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-581608/distinct-w2-581608-kserve-6bb6bb7b65-v7kfj" podUID="dcf2f562-3c50-4de7-9c84-cb659d3bc25b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded" Nov 02 06:38:05.520635 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:05.519989 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-581608/distinct-w2-581608-kserve-6bb6bb7b65-v7kfj" Nov 02 06:38:05.571529 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:05.541849 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bdcd80/simulator-w0-bdcd80-kserve-5749c847cb-9ktcr" podUID="2043ff0a-8a71-4b71-9750-beb566a12bad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Nov 02 06:38:05.609427 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:05.608732 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cdfce9/premium-w1-cdfce9-kserve-598cb9d57f-mrzrz" podUID="f9c5c1f7-cdca-46d6-8573-c85af1cd1946" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:05.680301 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:05.668791 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-baa3f47a-mhvfc"] Nov 02 06:38:05.775218 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:05.687033 5916 patch_prober.go:29] interesting pod/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hbwqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Nov 02 06:38:05.775811 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:05.775775 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hbwqc" podUID="2871e54e-8f8f-47d6-a16b-72d986b3835e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Nov 02 06:38:05.776070 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:05.723003 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fb1398/distinct-2-w3-fb1398-kserve-597b7dc99b-dp4jf" podUID="523a31c4-4aea-4e9f-9ec4-7c28faefbf7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:05.829474 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:05.689593 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-680b7b/simulator-w4-680b7b-kserve-6c4f574c56-p5xtc" podUID="be8a1039-5934-4885-93fe-070575f936ec" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Nov 02 06:38:05.829474 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:05.764179 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bdcd80/distinct-w0-bdcd80-kserve-5d9df79d48-bn5d4" podUID="1a7c7a03-e0cf-44ca-8eee-6c3e16112b3a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:05.829474 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:05.810766 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-fb1398/distinct-2-w3-fb1398-kserve-597b7dc99b-dp4jf" Nov 02 06:38:05.829474 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:05.810880 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-680b7b/simulator-w4-680b7b-kserve-6c4f574c56-p5xtc" Nov 02 06:38:05.829474 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:05.810949 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-bdcd80/distinct-w0-bdcd80-kserve-5d9df79d48-bn5d4" Nov 02 06:38:05.905155 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:05.894752 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cdfce9/simulator-w1-cdfce9-kserve-7ffd68f78-5vwl2" podUID="b8b219aa-6a59-4d31-96f6-8d50ab910aca" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:05.937332 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:05.930041 5916 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-fb1398/distinct-2-w3-fb1398-kserve-597b7dc99b-dp4jf" podUID="523a31c4-4aea-4e9f-9ec4-7c28faefbf7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:05.990751 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:05.990715 5916 patch_prober.go:29] interesting pod/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-flk8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Nov 02 06:38:06.017083 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:06.017037 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-flk8r" podUID="4c10335d-e121-4ccd-949d-964cf21650ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Nov 02 06:38:06.020090 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:06.019252 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-581608/distinct-2-w2-581608-kserve-57dd49c4b-qnt9k" podUID="837780ad-8ea3-4ed2-922f-917cffc7bb93" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:06.178018 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:06.109583 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-cdfce9/simulator-w1-cdfce9-kserve-7ffd68f78-5vwl2" Nov 02 06:38:06.241046 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:06.138522 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6ff85e/premium-w6-6ff85e-kserve-59848ff6d4-szbxx" podUID="eb386976-f898-4f1d-b27c-77bccc735a3b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:06.241381 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:06.156068 5916 patch_prober.go:29] interesting pod/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-w4bhs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Nov 02 06:38:06.241677 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:06.241643 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-w4bhs" podUID="eb1d132e-7237-4c50-b70b-8f97862d20c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Nov 02 06:38:06.241806 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:06.214171 5916 patch_prober.go:29] interesting pod/e2e-worker-w2-581608-openshift-default-cd7c896df-25spl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body= Nov 02 06:38:06.241953 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:06.241933 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-25spl" podUID="3818427d-4da9-4347-8a90-83e34f7ed19d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" Nov 02 06:38:06.242097 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:06.178933 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6ff85e/simulator-w6-6ff85e-kserve-9b9fd978f-q76nl" podUID="30114285-d730-4abb-a618-6833d5746feb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Nov 02 06:38:06.539928 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:06.519108 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-6ff85e/premium-w6-6ff85e-kserve-59848ff6d4-szbxx" Nov 02 06:38:06.539928 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:06.537096 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-6ff85e/simulator-w6-6ff85e-kserve-9b9fd978f-q76nl" Nov 02 06:38:06.651895 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:06.639734 5916 patch_prober.go:29] interesting pod/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hbwqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Nov 02 06:38:06.671933 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:06.671868 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hbwqc" podUID="2871e54e-8f8f-47d6-a16b-72d986b3835e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Nov 02 06:38:06.672250 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:06.649526 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cdfce9/premium-w1-cdfce9-kserve-598cb9d57f-mrzrz" podUID="f9c5c1f7-cdca-46d6-8573-c85af1cd1946" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:06.672943 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:06.649573 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bdcd80/simulator-w0-bdcd80-kserve-5749c847cb-9ktcr" podUID="2043ff0a-8a71-4b71-9750-beb566a12bad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Nov 02 06:38:06.672943 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:06.649600 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dcf9c3/distinct-2-w5-dcf9c3-kserve-64747dbd8f-wlwng" podUID="e37d42ab-7b62-4f07-8a29-cdedeb0fe89a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:06.672943 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:06.647619 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-581608/simulator-w2-581608-kserve-68f68d8559-xkcwp" podUID="529e25e6-4479-49d6-85c7-85dad6c2193e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:06.778983 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:06.777077 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fb1398/distinct-2-w3-fb1398-kserve-597b7dc99b-dp4jf" podUID="523a31c4-4aea-4e9f-9ec4-7c28faefbf7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:06.849578 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:06.795360 5916 patch_prober.go:29] interesting pod/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-w4bhs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Nov 02 06:38:06.849578 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:06.795424 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-w4bhs" podUID="eb1d132e-7237-4c50-b70b-8f97862d20c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Nov 02 06:38:06.849578 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:06.815643 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-680b7b/simulator-w4-680b7b-kserve-6c4f574c56-p5xtc" podUID="be8a1039-5934-4885-93fe-070575f936ec" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Nov 02 06:38:06.849578 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:06.816016 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bdcd80/distinct-w0-bdcd80-kserve-5d9df79d48-bn5d4" podUID="1a7c7a03-e0cf-44ca-8eee-6c3e16112b3a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded" Nov 02 06:38:06.849578 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:06.835985 5916 patch_prober.go:29] interesting pod/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-flk8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Nov 02 06:38:06.849578 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:06.836040 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-flk8r" podUID="4c10335d-e121-4ccd-949d-964cf21650ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Nov 02 06:38:06.920935 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:06.919079 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cdfce9/simulator-w1-cdfce9-kserve-7ffd68f78-5vwl2" podUID="b8b219aa-6a59-4d31-96f6-8d50ab910aca" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:06.944362 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:06.944287 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-baa3f47a-mhvfc"] Nov 02 06:38:06.944362 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:06.944342 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7de0cfea-dcgfp"] Nov 02 06:38:06.944728 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:06.944466 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-581608/distinct-w2-581608-kserve-6bb6bb7b65-v7kfj" Nov 02 06:38:06.944728 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:06.944500 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-6ff85e/premium-w6-6ff85e-kserve-59848ff6d4-szbxx" Nov 02 06:38:06.944728 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:06.944538 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-cdfce9/premium-w1-cdfce9-kserve-598cb9d57f-mrzrz" Nov 02 06:38:06.944728 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:06.944572 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-bdcd80/simulator-w0-bdcd80-kserve-5749c847cb-9ktcr" Nov 02 06:38:06.944728 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:06.944605 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-581608/simulator-w2-581608-kserve-68f68d8559-xkcwp" Nov 02 06:38:06.944728 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:06.944632 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-bdcd80/simulator-w0-bdcd80-kserve-5749c847cb-9ktcr" Nov 02 06:38:06.944728 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:06.944659 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-680b7b/simulator-w4-680b7b-kserve-6c4f574c56-p5xtc" Nov 02 06:38:06.990201 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:06.990166 5916 patch_prober.go:29] interesting pod/e2e-worker-w2-581608-openshift-default-cd7c896df-25spl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body= Nov 02 06:38:07.022166 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:06.994097 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-25spl" podUID="3818427d-4da9-4347-8a90-83e34f7ed19d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" Nov 02 06:38:07.224940 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:07.221137 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-581608/distinct-2-w2-581608-kserve-57dd49c4b-qnt9k" podUID="837780ad-8ea3-4ed2-922f-917cffc7bb93" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:07.224940 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:07.222547 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-581608/distinct-2-w2-581608-kserve-57dd49c4b-qnt9k" Nov 02 06:38:07.295931 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:07.275190 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6ff85e/simulator-w6-6ff85e-kserve-9b9fd978f-q76nl" podUID="30114285-d730-4abb-a618-6833d5746feb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:07.398928 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:07.398723 5916 patch_prober.go:29] interesting pod/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hbwqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Nov 02 06:38:07.398928 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:07.398786 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hbwqc" podUID="2871e54e-8f8f-47d6-a16b-72d986b3835e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Nov 02 06:38:07.727481 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:07.727417 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cdfce9/premium-w1-cdfce9-kserve-598cb9d57f-mrzrz" podUID="f9c5c1f7-cdca-46d6-8573-c85af1cd1946" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:07.728094 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:07.728060 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dcf9c3/distinct-2-w5-dcf9c3-kserve-64747dbd8f-wlwng" podUID="e37d42ab-7b62-4f07-8a29-cdedeb0fe89a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Nov 02 06:38:07.728530 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:07.728510 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-581608/simulator-w2-581608-kserve-68f68d8559-xkcwp" podUID="529e25e6-4479-49d6-85c7-85dad6c2193e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Nov 02 06:38:07.729035 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:07.729018 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-dcf9c3/distinct-2-w5-dcf9c3-kserve-64747dbd8f-wlwng" Nov 02 06:38:07.797104 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:07.773261 5916 patch_prober.go:29] interesting pod/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-w4bhs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Nov 02 06:38:07.797104 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:07.773318 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-w4bhs" podUID="eb1d132e-7237-4c50-b70b-8f97862d20c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Nov 02 06:38:07.808929 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:07.797670 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fb1398/distinct-2-w3-fb1398-kserve-597b7dc99b-dp4jf" podUID="523a31c4-4aea-4e9f-9ec4-7c28faefbf7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": context deadline exceeded" Nov 02 06:38:07.918161 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:07.845190 5916 patch_prober.go:29] interesting pod/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-flk8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Nov 02 06:38:07.947372 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:07.947330 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-flk8r" podUID="4c10335d-e121-4ccd-949d-964cf21650ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Nov 02 06:38:08.017314 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:07.945559 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bdcd80/distinct-w0-bdcd80-kserve-5d9df79d48-bn5d4" podUID="1a7c7a03-e0cf-44ca-8eee-6c3e16112b3a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:08.043708 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:07.946216 5916 patch_prober.go:29] interesting pod/e2e-worker-w2-581608-openshift-default-cd7c896df-25spl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body= Nov 02 06:38:08.043963 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:08.043934 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-25spl" podUID="3818427d-4da9-4347-8a90-83e34f7ed19d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" Nov 02 06:38:08.044083 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:07.947680 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cdfce9/simulator-w1-cdfce9-kserve-7ffd68f78-5vwl2" podUID="b8b219aa-6a59-4d31-96f6-8d50ab910aca" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:08.241064 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:08.241016 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-581608/distinct-2-w2-581608-kserve-57dd49c4b-qnt9k" podUID="837780ad-8ea3-4ed2-922f-917cffc7bb93" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": context deadline exceeded" Nov 02 06:38:08.302529 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:08.302019 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6ff85e/simulator-w6-6ff85e-kserve-9b9fd978f-q76nl" podUID="30114285-d730-4abb-a618-6833d5746feb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:08.320318 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:08.320275 5916 patch_prober.go:29] interesting pod/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hbwqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Nov 02 06:38:08.320566 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:08.320541 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hbwqc" podUID="2871e54e-8f8f-47d6-a16b-72d986b3835e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Nov 02 06:38:08.448807 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:08.448756 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bdcd80/simulator-w0-bdcd80-kserve-5749c847cb-9ktcr" podUID="2043ff0a-8a71-4b71-9750-beb566a12bad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:08.526955 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:08.526414 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff33efe1-03ac-4157-acda-a302dd262e10" path="/var/lib/kubelet/pods/ff33efe1-03ac-4157-acda-a302dd262e10/volumes" Nov 02 06:38:08.630073 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:08.610875 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7de0cfea-dcgfp" event={"ID":"851e0293-99c4-4ed8-a9ce-7bff1204e30e","Type":"ContainerStarted","Data":"5134fbc7fc36d89eda1da11f82678034687dc64d47196f7ab33a5528700805cb"} Nov 02 06:38:08.680046 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:08.679997 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-581608/distinct-w2-581608-kserve-6bb6bb7b65-v7kfj" podUID="dcf2f562-3c50-4de7-9c84-cb659d3bc25b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded" Nov 02 06:38:08.713049 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:08.713001 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6ff85e/premium-w6-6ff85e-kserve-59848ff6d4-szbxx" podUID="eb386976-f898-4f1d-b27c-77bccc735a3b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Nov 02 06:38:08.792239 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:08.758237 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-581608/simulator-w2-581608-kserve-68f68d8559-xkcwp" podUID="529e25e6-4479-49d6-85c7-85dad6c2193e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:08.792239 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:08.758773 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cdfce9/premium-w1-cdfce9-kserve-598cb9d57f-mrzrz" podUID="f9c5c1f7-cdca-46d6-8573-c85af1cd1946" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Nov 02 06:38:08.792239 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:08.759033 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dcf9c3/distinct-2-w5-dcf9c3-kserve-64747dbd8f-wlwng" podUID="e37d42ab-7b62-4f07-8a29-cdedeb0fe89a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Nov 02 06:38:08.819545 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:08.819414 5916 patch_prober.go:29] interesting pod/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-flk8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Nov 02 06:38:08.819545 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:08.819474 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-flk8r" podUID="4c10335d-e121-4ccd-949d-964cf21650ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Nov 02 06:38:08.869261 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:08.869207 5916 patch_prober.go:29] interesting pod/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-w4bhs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Nov 02 06:38:08.876936 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:08.869482 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-w4bhs" podUID="eb1d132e-7237-4c50-b70b-8f97862d20c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Nov 02 06:38:08.893985 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:08.891305 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fb1398/distinct-2-w3-fb1398-kserve-597b7dc99b-dp4jf" podUID="523a31c4-4aea-4e9f-9ec4-7c28faefbf7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": context deadline exceeded" Nov 02 06:38:08.916403 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:08.894245 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-680b7b/simulator-w4-680b7b-kserve-6c4f574c56-p5xtc" podUID="be8a1039-5934-4885-93fe-070575f936ec" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:08.930558 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:08.930471 5916 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-bdcd80/simulator-w0-bdcd80-kserve-5749c847cb-9ktcr" podUID="2043ff0a-8a71-4b71-9750-beb566a12bad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:08.959098 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:08.958982 5916 patch_prober.go:29] interesting pod/e2e-worker-w2-581608-openshift-default-cd7c896df-25spl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body= Nov 02 06:38:08.959098 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:08.959038 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-25spl" podUID="3818427d-4da9-4347-8a90-83e34f7ed19d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" Nov 02 06:38:09.096193 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:09.096141 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bdcd80/distinct-w0-bdcd80-kserve-5d9df79d48-bn5d4" podUID="1a7c7a03-e0cf-44ca-8eee-6c3e16112b3a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded" Nov 02 06:38:09.096812 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:09.096774 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cdfce9/simulator-w1-cdfce9-kserve-7ffd68f78-5vwl2" podUID="b8b219aa-6a59-4d31-96f6-8d50ab910aca" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:09.213591 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:09.213142 5916 patch_prober.go:29] interesting pod/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hbwqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Nov 02 06:38:09.213591 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:09.213202 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hbwqc" podUID="2871e54e-8f8f-47d6-a16b-72d986b3835e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Nov 02 06:38:09.243707 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:09.243664 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-581608/distinct-2-w2-581608-kserve-57dd49c4b-qnt9k" podUID="837780ad-8ea3-4ed2-922f-917cffc7bb93" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:09.323100 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:09.323005 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6ff85e/simulator-w6-6ff85e-kserve-9b9fd978f-q76nl" podUID="30114285-d730-4abb-a618-6833d5746feb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:09.359648 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:09.359613 5916 generic.go:356] "Generic (PLEG): container finished" podID="851e0293-99c4-4ed8-a9ce-7bff1204e30e" containerID="20fd1e3d7d3d4c7d1c853fddb05b01f9fb89cb809daf8c3f4c617cb6a7778e08" exitCode=0 Nov 02 06:38:09.373518 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:09.373480 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7de0cfea-dcgfp" event={"ID":"851e0293-99c4-4ed8-a9ce-7bff1204e30e","Type":"ContainerDied","Data":"20fd1e3d7d3d4c7d1c853fddb05b01f9fb89cb809daf8c3f4c617cb6a7778e08"} Nov 02 06:38:09.463189 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:09.463138 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bdcd80/simulator-w0-bdcd80-kserve-5749c847cb-9ktcr" podUID="2043ff0a-8a71-4b71-9750-beb566a12bad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:09.463895 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:09.463823 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-bdcd80/simulator-w0-bdcd80-kserve-5749c847cb-9ktcr" Nov 02 06:38:09.692050 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:09.691999 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-581608/distinct-w2-581608-kserve-6bb6bb7b65-v7kfj" podUID="dcf2f562-3c50-4de7-9c84-cb659d3bc25b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:09.693220 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:09.693166 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-581608/distinct-w2-581608-kserve-6bb6bb7b65-v7kfj" Nov 02 06:38:09.740528 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:09.738047 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6ff85e/premium-w6-6ff85e-kserve-59848ff6d4-szbxx" podUID="eb386976-f898-4f1d-b27c-77bccc735a3b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:09.784939 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:09.765950 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-6ff85e/premium-w6-6ff85e-kserve-59848ff6d4-szbxx" Nov 02 06:38:09.912936 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:09.910457 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-680b7b/simulator-w4-680b7b-kserve-6c4f574c56-p5xtc" podUID="be8a1039-5934-4885-93fe-070575f936ec" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:09.948967 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:09.922024 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fb1398/distinct-2-w3-fb1398-kserve-597b7dc99b-dp4jf" podUID="523a31c4-4aea-4e9f-9ec4-7c28faefbf7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": context deadline exceeded" Nov 02 06:38:09.948967 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:09.929777 5916 patch_prober.go:29] interesting pod/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-w4bhs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Nov 02 06:38:09.948967 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:09.929826 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-w4bhs" podUID="eb1d132e-7237-4c50-b70b-8f97862d20c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Nov 02 06:38:09.948967 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:09.929926 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-581608/simulator-w2-581608-kserve-68f68d8559-xkcwp" podUID="529e25e6-4479-49d6-85c7-85dad6c2193e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Nov 02 06:38:09.948967 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:09.930106 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cdfce9/premium-w1-cdfce9-kserve-598cb9d57f-mrzrz" podUID="f9c5c1f7-cdca-46d6-8573-c85af1cd1946" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Nov 02 06:38:09.948967 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:09.930245 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dcf9c3/distinct-2-w5-dcf9c3-kserve-64747dbd8f-wlwng" podUID="e37d42ab-7b62-4f07-8a29-cdedeb0fe89a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Nov 02 06:38:09.948967 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:09.944208 5916 patch_prober.go:29] interesting pod/e2e-worker-w2-581608-openshift-default-cd7c896df-25spl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body= Nov 02 06:38:09.948967 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:09.944256 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-25spl" podUID="3818427d-4da9-4347-8a90-83e34f7ed19d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" Nov 02 06:38:10.375972 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:10.297042 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-581608/distinct-2-w2-581608-kserve-57dd49c4b-qnt9k" podUID="837780ad-8ea3-4ed2-922f-917cffc7bb93" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:10.375972 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:10.321021 5916 patch_prober.go:29] interesting pod/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-flk8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Nov 02 06:38:10.375972 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:10.321100 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-flk8r" podUID="4c10335d-e121-4ccd-949d-964cf21650ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Nov 02 06:38:10.375972 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:10.372312 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6ff85e/simulator-w6-6ff85e-kserve-9b9fd978f-q76nl" podUID="30114285-d730-4abb-a618-6833d5746feb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:10.397843 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:10.397781 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cdfce9/simulator-w1-cdfce9-kserve-7ffd68f78-5vwl2" podUID="b8b219aa-6a59-4d31-96f6-8d50ab910aca" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Nov 02 06:38:10.401279 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:10.400035 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bdcd80/distinct-w0-bdcd80-kserve-5d9df79d48-bn5d4" podUID="1a7c7a03-e0cf-44ca-8eee-6c3e16112b3a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:10.474510 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:10.466243 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bdcd80/simulator-w0-bdcd80-kserve-5749c847cb-9ktcr" podUID="2043ff0a-8a71-4b71-9750-beb566a12bad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:10.758930 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:10.754786 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6ff85e/premium-w6-6ff85e-kserve-59848ff6d4-szbxx" podUID="eb386976-f898-4f1d-b27c-77bccc735a3b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:11.168930 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:11.161973 5916 patch_prober.go:29] interesting pod/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hbwqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Nov 02 06:38:11.168930 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:11.162033 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hbwqc" podUID="2871e54e-8f8f-47d6-a16b-72d986b3835e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Nov 02 06:38:11.168930 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:11.162196 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fb1398/distinct-2-w3-fb1398-kserve-597b7dc99b-dp4jf" podUID="523a31c4-4aea-4e9f-9ec4-7c28faefbf7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:11.207222 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:38:11.197082 5916 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.241s" Nov 02 06:38:11.207222 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:11.197260 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-680b7b/simulator-w4-680b7b-kserve-6c4f574c56-p5xtc" Nov 02 06:38:11.207222 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:11.203234 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-581608/distinct-w2-581608-kserve-6bb6bb7b65-v7kfj" podUID="dcf2f562-3c50-4de7-9c84-cb659d3bc25b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:11.220271 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:11.220228 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-680b7b/simulator-w4-680b7b-kserve-6c4f574c56-p5xtc" podUID="be8a1039-5934-4885-93fe-070575f936ec" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:11.221042 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:11.220829 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-581608/simulator-w2-581608-kserve-68f68d8559-xkcwp" podUID="529e25e6-4479-49d6-85c7-85dad6c2193e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:11.221251 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:11.221219 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cdfce9/premium-w1-cdfce9-kserve-598cb9d57f-mrzrz" podUID="f9c5c1f7-cdca-46d6-8573-c85af1cd1946" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:11.239538 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:11.222075 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dcf9c3/distinct-2-w5-dcf9c3-kserve-64747dbd8f-wlwng" podUID="e37d42ab-7b62-4f07-8a29-cdedeb0fe89a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:11.299313 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:11.299254 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-581608/distinct-2-w2-581608-kserve-57dd49c4b-qnt9k" podUID="837780ad-8ea3-4ed2-922f-917cffc7bb93" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:11.328331 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:11.328106 5916 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-581608/simulator-w2-581608-kserve-68f68d8559-xkcwp" podUID="529e25e6-4479-49d6-85c7-85dad6c2193e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:11.337270 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:11.335945 5916 patch_prober.go:29] interesting pod/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hbwqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Nov 02 06:38:11.337270 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:11.335999 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hbwqc" podUID="2871e54e-8f8f-47d6-a16b-72d986b3835e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Nov 02 06:38:11.338940 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:11.338361 5916 patch_prober.go:29] interesting pod/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-w4bhs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Nov 02 06:38:11.338940 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:11.338417 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-w4bhs" podUID="eb1d132e-7237-4c50-b70b-8f97862d20c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Nov 02 06:38:11.365841 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:11.365812 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-6ff85e/simulator-w6-6ff85e-kserve-9b9fd978f-q76nl" Nov 02 06:38:11.402993 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:11.400452 5916 patch_prober.go:29] interesting pod/e2e-worker-w2-581608-openshift-default-cd7c896df-25spl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body= Nov 02 06:38:11.402993 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:11.400506 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-25spl" podUID="3818427d-4da9-4347-8a90-83e34f7ed19d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" Nov 02 06:38:11.441011 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:11.411751 5916 patch_prober.go:29] interesting pod/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-flk8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Nov 02 06:38:11.441011 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:11.411820 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-flk8r" podUID="4c10335d-e121-4ccd-949d-964cf21650ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Nov 02 06:38:11.441011 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:11.421490 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bdcd80/distinct-w0-bdcd80-kserve-5d9df79d48-bn5d4" podUID="1a7c7a03-e0cf-44ca-8eee-6c3e16112b3a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:11.441011 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:11.421589 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cdfce9/simulator-w1-cdfce9-kserve-7ffd68f78-5vwl2" podUID="b8b219aa-6a59-4d31-96f6-8d50ab910aca" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Nov 02 06:38:11.490786 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:11.490756 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-581608/distinct-w2-581608-kserve-6bb6bb7b65-v7kfj" Nov 02 06:38:11.565177 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:11.565134 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bdcd80/simulator-w0-bdcd80-kserve-5749c847cb-9ktcr" podUID="2043ff0a-8a71-4b71-9750-beb566a12bad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Nov 02 06:38:11.567128 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:11.567104 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-6ff85e/premium-w6-6ff85e-kserve-59848ff6d4-szbxx" Nov 02 06:38:11.760966 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:11.760867 5916 patch_prober.go:29] interesting pod/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-w4bhs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Nov 02 06:38:11.781770 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:11.761258 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-w4bhs" podUID="eb1d132e-7237-4c50-b70b-8f97862d20c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Nov 02 06:38:11.847948 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:11.847892 5916 patch_prober.go:29] interesting pod/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-flk8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Nov 02 06:38:11.848214 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:11.848183 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-flk8r" podUID="4c10335d-e121-4ccd-949d-964cf21650ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Nov 02 06:38:11.947637 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:11.947593 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-bdcd80/simulator-w0-bdcd80-kserve-5749c847cb-9ktcr" Nov 02 06:38:11.977246 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:11.977198 5916 patch_prober.go:29] interesting pod/e2e-worker-w2-581608-openshift-default-cd7c896df-25spl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body= Nov 02 06:38:11.977533 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:11.977490 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-25spl" podUID="3818427d-4da9-4347-8a90-83e34f7ed19d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" Nov 02 06:38:12.076622 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:12.076530 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-cdfce9/simulator-w1-cdfce9-kserve-7ffd68f78-5vwl2" Nov 02 06:38:12.128709 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:12.123680 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-fb1398/distinct-2-w3-fb1398-kserve-597b7dc99b-dp4jf" Nov 02 06:38:12.128709 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:12.123736 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-581608/distinct-2-w2-581608-kserve-57dd49c4b-qnt9k" Nov 02 06:38:12.128709 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:12.128142 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-dcf9c3/distinct-2-w5-dcf9c3-kserve-64747dbd8f-wlwng" Nov 02 06:38:12.205233 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:12.205203 5916 patch_prober.go:29] interesting pod/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hbwqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Nov 02 06:38:12.205798 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:12.205770 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hbwqc" podUID="2871e54e-8f8f-47d6-a16b-72d986b3835e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Nov 02 06:38:12.211119 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:12.211100 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-bdcd80/distinct-w0-bdcd80-kserve-5d9df79d48-bn5d4" Nov 02 06:38:12.218968 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:12.214172 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-cdfce9/premium-w1-cdfce9-kserve-598cb9d57f-mrzrz" Nov 02 06:38:12.251004 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:12.220834 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-680b7b/simulator-w4-680b7b-kserve-6c4f574c56-p5xtc" podUID="be8a1039-5934-4885-93fe-070575f936ec" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:12.251004 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:12.221177 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-581608/simulator-w2-581608-kserve-68f68d8559-xkcwp" podUID="529e25e6-4479-49d6-85c7-85dad6c2193e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:12.260989 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:12.260959 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s6n9q"] Nov 02 06:38:12.291446 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:12.289091 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7de0cfea-dcgfp" Nov 02 06:38:12.300957 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:12.298847 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-680b7b/simulator-w4-680b7b-kserve-6c4f574c56-p5xtc" Nov 02 06:38:12.304655 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:12.299068 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s6n9q" Nov 02 06:38:12.312184 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:12.309971 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/851e0293-99c4-4ed8-a9ce-7bff1204e30e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/851e0293-99c4-4ed8-a9ce-7bff1204e30e-maas-api-xks-ca\") pod \"851e0293-99c4-4ed8-a9ce-7bff1204e30e\" (UID: \"851e0293-99c4-4ed8-a9ce-7bff1204e30e\") " Nov 02 06:38:12.312184 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:12.311615 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/851e0293-99c4-4ed8-a9ce-7bff1204e30e-kube-api-access-zqjhk\" (UniqueName: \"kubernetes.io/projected/851e0293-99c4-4ed8-a9ce-7bff1204e30e-kube-api-access-zqjhk\") pod \"851e0293-99c4-4ed8-a9ce-7bff1204e30e\" (UID: \"851e0293-99c4-4ed8-a9ce-7bff1204e30e\") " Nov 02 06:38:12.312184 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:12.311669 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/851e0293-99c4-4ed8-a9ce-7bff1204e30e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/851e0293-99c4-4ed8-a9ce-7bff1204e30e-maas-api-openshift-ca\") pod \"851e0293-99c4-4ed8-a9ce-7bff1204e30e\" (UID: \"851e0293-99c4-4ed8-a9ce-7bff1204e30e\") " Nov 02 06:38:12.313102 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:12.312882 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/851e0293-99c4-4ed8-a9ce-7bff1204e30e-maas-api-openshift-ca" pod "851e0293-99c4-4ed8-a9ce-7bff1204e30e" (UID: "851e0293-99c4-4ed8-a9ce-7bff1204e30e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:38:12.315158 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:12.313444 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ed18eacc-a61b-470d-8cd9-1d46f742fabd-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-s6n9q\" (UID: \"ed18eacc-a61b-470d-8cd9-1d46f742fabd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s6n9q" Nov 02 06:38:12.315158 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:12.313479 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed18eacc-a61b-470d-8cd9-1d46f742fabd-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-s6n9q\" (UID: \"ed18eacc-a61b-470d-8cd9-1d46f742fabd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s6n9q" Nov 02 06:38:12.315158 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:12.313545 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzjzt\" (UniqueName: \"kubernetes.io/projected/ed18eacc-a61b-470d-8cd9-1d46f742fabd-kube-api-access-nzjzt\") pod \"istiod-openshift-gateway-7ddd9cfdc8-s6n9q\" (UID: \"ed18eacc-a61b-470d-8cd9-1d46f742fabd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s6n9q" Nov 02 06:38:12.315158 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:12.313570 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ed18eacc-a61b-470d-8cd9-1d46f742fabd-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-s6n9q\" (UID: \"ed18eacc-a61b-470d-8cd9-1d46f742fabd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s6n9q" Nov 02 06:38:12.315158 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:12.313757 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed18eacc-a61b-470d-8cd9-1d46f742fabd-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-s6n9q\" (UID: \"ed18eacc-a61b-470d-8cd9-1d46f742fabd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s6n9q" Nov 02 06:38:12.315158 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:12.313818 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ed18eacc-a61b-470d-8cd9-1d46f742fabd-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-s6n9q\" (UID: \"ed18eacc-a61b-470d-8cd9-1d46f742fabd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s6n9q" Nov 02 06:38:12.315158 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:12.313860 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ed18eacc-a61b-470d-8cd9-1d46f742fabd-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-s6n9q\" (UID: \"ed18eacc-a61b-470d-8cd9-1d46f742fabd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s6n9q" Nov 02 06:38:12.315158 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:12.314062 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/851e0293-99c4-4ed8-a9ce-7bff1204e30e-maas-api-openshift-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:38:12.318056 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:12.318030 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/851e0293-99c4-4ed8-a9ce-7bff1204e30e-maas-api-xks-ca" pod "851e0293-99c4-4ed8-a9ce-7bff1204e30e" (UID: "851e0293-99c4-4ed8-a9ce-7bff1204e30e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:38:12.318550 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:12.318521 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s6n9q"] Nov 02 06:38:12.321262 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:12.321237 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/851e0293-99c4-4ed8-a9ce-7bff1204e30e-kube-api-access-zqjhk" pod "851e0293-99c4-4ed8-a9ce-7bff1204e30e" (UID: "851e0293-99c4-4ed8-a9ce-7bff1204e30e"). InnerVolumeSpecName "kube-api-access-zqjhk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:38:12.396976 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:12.394549 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-581608/simulator-w2-581608-kserve-68f68d8559-xkcwp" Nov 02 06:38:12.401981 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:12.401599 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Nov 02 06:38:12.401981 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:12.401699 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Nov 02 06:38:12.401981 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:12.401797 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Nov 02 06:38:12.401981 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:12.401966 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-vk64l\"" Nov 02 06:38:12.441937 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:12.438089 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzjzt\" (UniqueName: \"kubernetes.io/projected/ed18eacc-a61b-470d-8cd9-1d46f742fabd-kube-api-access-nzjzt\") pod \"istiod-openshift-gateway-7ddd9cfdc8-s6n9q\" (UID: \"ed18eacc-a61b-470d-8cd9-1d46f742fabd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s6n9q" Nov 02 06:38:12.441937 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:12.438146 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ed18eacc-a61b-470d-8cd9-1d46f742fabd-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-s6n9q\" (UID: \"ed18eacc-a61b-470d-8cd9-1d46f742fabd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s6n9q" Nov 02 06:38:12.441937 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:12.438253 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed18eacc-a61b-470d-8cd9-1d46f742fabd-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-s6n9q\" (UID: \"ed18eacc-a61b-470d-8cd9-1d46f742fabd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s6n9q" Nov 02 06:38:12.441937 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:12.438303 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ed18eacc-a61b-470d-8cd9-1d46f742fabd-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-s6n9q\" (UID: \"ed18eacc-a61b-470d-8cd9-1d46f742fabd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s6n9q" Nov 02 06:38:12.441937 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:12.438344 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ed18eacc-a61b-470d-8cd9-1d46f742fabd-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-s6n9q\" (UID: \"ed18eacc-a61b-470d-8cd9-1d46f742fabd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s6n9q" Nov 02 06:38:12.441937 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:12.438598 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ed18eacc-a61b-470d-8cd9-1d46f742fabd-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-s6n9q\" (UID: \"ed18eacc-a61b-470d-8cd9-1d46f742fabd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s6n9q" Nov 02 06:38:12.441937 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:12.438640 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed18eacc-a61b-470d-8cd9-1d46f742fabd-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-s6n9q\" (UID: \"ed18eacc-a61b-470d-8cd9-1d46f742fabd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s6n9q" Nov 02 06:38:12.441937 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:12.438786 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/851e0293-99c4-4ed8-a9ce-7bff1204e30e-maas-api-xks-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:38:12.441937 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:12.438807 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqjhk\" (UniqueName: \"kubernetes.io/projected/851e0293-99c4-4ed8-a9ce-7bff1204e30e-kube-api-access-zqjhk\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:38:12.445381 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:12.445342 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ed18eacc-a61b-470d-8cd9-1d46f742fabd-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-s6n9q\" (UID: \"ed18eacc-a61b-470d-8cd9-1d46f742fabd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s6n9q" Nov 02 06:38:12.457781 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:12.456727 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ed18eacc-a61b-470d-8cd9-1d46f742fabd-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-s6n9q\" (UID: \"ed18eacc-a61b-470d-8cd9-1d46f742fabd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s6n9q" Nov 02 06:38:12.477674 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:12.477616 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ed18eacc-a61b-470d-8cd9-1d46f742fabd-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-s6n9q\" (UID: \"ed18eacc-a61b-470d-8cd9-1d46f742fabd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s6n9q" Nov 02 06:38:12.485407 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:12.485376 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ed18eacc-a61b-470d-8cd9-1d46f742fabd-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-s6n9q\" (UID: \"ed18eacc-a61b-470d-8cd9-1d46f742fabd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s6n9q" Nov 02 06:38:12.494160 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:12.494120 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed18eacc-a61b-470d-8cd9-1d46f742fabd-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-s6n9q\" (UID: \"ed18eacc-a61b-470d-8cd9-1d46f742fabd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s6n9q" Nov 02 06:38:12.509614 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:12.509573 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed18eacc-a61b-470d-8cd9-1d46f742fabd-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-s6n9q\" (UID: \"ed18eacc-a61b-470d-8cd9-1d46f742fabd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s6n9q" Nov 02 06:38:12.511298 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:12.511252 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzjzt\" (UniqueName: \"kubernetes.io/projected/ed18eacc-a61b-470d-8cd9-1d46f742fabd-kube-api-access-nzjzt\") pod \"istiod-openshift-gateway-7ddd9cfdc8-s6n9q\" (UID: \"ed18eacc-a61b-470d-8cd9-1d46f742fabd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s6n9q" Nov 02 06:38:12.683900 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:12.683867 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s6n9q" Nov 02 06:38:13.101373 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:13.101278 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7de0cfea-dcgfp" event={"ID":"851e0293-99c4-4ed8-a9ce-7bff1204e30e","Type":"ContainerDied","Data":"5134fbc7fc36d89eda1da11f82678034687dc64d47196f7ab33a5528700805cb"} Nov 02 06:38:13.101630 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:13.101610 5916 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5134fbc7fc36d89eda1da11f82678034687dc64d47196f7ab33a5528700805cb" Nov 02 06:38:13.101813 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:13.101797 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7de0cfea-dcgfp" Nov 02 06:38:13.281920 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:13.281874 5916 patch_prober.go:29] interesting pod/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hbwqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Nov 02 06:38:13.282529 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:13.282497 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hbwqc" podUID="2871e54e-8f8f-47d6-a16b-72d986b3835e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Nov 02 06:38:13.478146 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:13.478113 5916 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-flk8r" Nov 02 06:38:13.479625 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:13.479604 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-flk8r" Nov 02 06:38:13.484434 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:13.484416 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-flk8r" Nov 02 06:38:13.682944 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:13.675208 5916 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-w4bhs" Nov 02 06:38:13.682944 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:13.679081 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-w4bhs" Nov 02 06:38:13.691379 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:13.686982 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-w4bhs" Nov 02 06:38:13.705861 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:13.705829 5916 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-25spl" Nov 02 06:38:13.707167 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:13.707142 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-25spl" Nov 02 06:38:13.712553 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:13.712533 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-25spl" Nov 02 06:38:13.890134 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:13.890094 5916 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 06:38:13.902695 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:13.902657 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s6n9q"] Nov 02 06:38:14.113479 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:14.113441 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s6n9q" event={"ID":"ed18eacc-a61b-470d-8cd9-1d46f742fabd","Type":"ContainerStarted","Data":"5fb756db6e9c34fddaf9c0fc14bf15f109ac09f551f4eb64863c65ff49d9093a"} Nov 02 06:38:14.205867 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:14.205840 5916 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hbwqc" Nov 02 06:38:14.206155 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:14.206136 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hbwqc" Nov 02 06:38:14.206840 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:14.206825 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hbwqc" Nov 02 06:38:14.319575 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:14.319544 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98e3f2eb-9rds2"] Nov 02 06:38:14.321019 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:14.320987 5916 cpu_manager.go:414] "containerMap: removing container" podUID="851e0293-99c4-4ed8-a9ce-7bff1204e30e" containerName="revoke-keys" Nov 02 06:38:14.321019 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:14.321019 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="851e0293-99c4-4ed8-a9ce-7bff1204e30e" containerName="revoke-keys" Nov 02 06:38:14.321502 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:14.321483 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="851e0293-99c4-4ed8-a9ce-7bff1204e30e" containerName="revoke-keys" Nov 02 06:38:14.333969 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:14.333895 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98e3f2eb-9rds2" Nov 02 06:38:14.387982 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:14.386858 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 06:38:14.387982 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:14.387012 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-68l4r\"" Nov 02 06:38:14.387982 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:14.387695 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98e3f2eb-9rds2"] Nov 02 06:38:14.516305 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:14.516220 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0b8ef83-4677-405d-be44-4351cca0f47f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-98e3f2eb-9rds2\" (UID: \"f0b8ef83-4677-405d-be44-4351cca0f47f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98e3f2eb-9rds2" Nov 02 06:38:14.516305 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:14.516265 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0b8ef83-4677-405d-be44-4351cca0f47f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-98e3f2eb-9rds2\" (UID: \"f0b8ef83-4677-405d-be44-4351cca0f47f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98e3f2eb-9rds2" Nov 02 06:38:14.516305 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:14.516290 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2828\" (UniqueName: \"kubernetes.io/projected/f0b8ef83-4677-405d-be44-4351cca0f47f-kube-api-access-f2828\") pod \"maas-api-revoke-sub-98e3f2eb-9rds2\" (UID: \"f0b8ef83-4677-405d-be44-4351cca0f47f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98e3f2eb-9rds2" Nov 02 06:38:14.617764 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:14.617732 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0b8ef83-4677-405d-be44-4351cca0f47f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-98e3f2eb-9rds2\" (UID: \"f0b8ef83-4677-405d-be44-4351cca0f47f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98e3f2eb-9rds2" Nov 02 06:38:14.617969 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:14.617780 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0b8ef83-4677-405d-be44-4351cca0f47f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-98e3f2eb-9rds2\" (UID: \"f0b8ef83-4677-405d-be44-4351cca0f47f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98e3f2eb-9rds2" Nov 02 06:38:14.617969 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:14.617800 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2828\" (UniqueName: \"kubernetes.io/projected/f0b8ef83-4677-405d-be44-4351cca0f47f-kube-api-access-f2828\") pod \"maas-api-revoke-sub-98e3f2eb-9rds2\" (UID: \"f0b8ef83-4677-405d-be44-4351cca0f47f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98e3f2eb-9rds2" Nov 02 06:38:14.618541 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:14.618515 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0b8ef83-4677-405d-be44-4351cca0f47f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-98e3f2eb-9rds2\" (UID: \"f0b8ef83-4677-405d-be44-4351cca0f47f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98e3f2eb-9rds2" Nov 02 06:38:14.620235 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:14.620211 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0b8ef83-4677-405d-be44-4351cca0f47f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-98e3f2eb-9rds2\" (UID: \"f0b8ef83-4677-405d-be44-4351cca0f47f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98e3f2eb-9rds2" Nov 02 06:38:14.678034 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:14.678002 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2828\" (UniqueName: \"kubernetes.io/projected/f0b8ef83-4677-405d-be44-4351cca0f47f-kube-api-access-f2828\") pod \"maas-api-revoke-sub-98e3f2eb-9rds2\" (UID: \"f0b8ef83-4677-405d-be44-4351cca0f47f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98e3f2eb-9rds2" Nov 02 06:38:14.952484 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:14.952449 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98e3f2eb-9rds2" Nov 02 06:38:15.359268 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:38:15.359238 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0b8ef83_4677_405d_be44_4351cca0f47f.slice/crio-6be8baee5c8047a2df2507eb377ea7d7106a82604c3022c3f86d6b6633cda917 WatchSource:0}: Error finding container 6be8baee5c8047a2df2507eb377ea7d7106a82604c3022c3f86d6b6633cda917: Status 404 returned error can't find the container with id 6be8baee5c8047a2df2507eb377ea7d7106a82604c3022c3f86d6b6633cda917 Nov 02 06:38:15.361311 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:15.361283 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98e3f2eb-9rds2"] Nov 02 06:38:16.139167 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:16.139125 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98e3f2eb-9rds2" event={"ID":"f0b8ef83-4677-405d-be44-4351cca0f47f","Type":"ContainerStarted","Data":"26b089f54ed089640fbc8fb80ffc3cb88195f912df29c6c21273e310c8f4dec8"} Nov 02 06:38:16.139377 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:16.139183 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98e3f2eb-9rds2" event={"ID":"f0b8ef83-4677-405d-be44-4351cca0f47f","Type":"ContainerStarted","Data":"6be8baee5c8047a2df2507eb377ea7d7106a82604c3022c3f86d6b6633cda917"} Nov 02 06:38:16.253338 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:16.253282 5916 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98e3f2eb-9rds2" podStartSLOduration=2.2532639899999998 podStartE2EDuration="2.25326399s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:38:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:38:16.252669835 +0000 UTC m=+1284.524490229" watchObservedRunningTime="2026-11-02 06:38:16.25326399 +0000 UTC m=+1284.525084384" Nov 02 06:38:17.155077 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:17.155036 5916 generic.go:356] "Generic (PLEG): container finished" podID="f0b8ef83-4677-405d-be44-4351cca0f47f" containerID="26b089f54ed089640fbc8fb80ffc3cb88195f912df29c6c21273e310c8f4dec8" exitCode=0 Nov 02 06:38:17.155559 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:17.155117 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98e3f2eb-9rds2" event={"ID":"f0b8ef83-4677-405d-be44-4351cca0f47f","Type":"ContainerDied","Data":"26b089f54ed089640fbc8fb80ffc3cb88195f912df29c6c21273e310c8f4dec8"} Nov 02 06:38:18.366996 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:18.366965 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98e3f2eb-9rds2" Nov 02 06:38:18.380244 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:18.380209 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0b8ef83-4677-405d-be44-4351cca0f47f-kube-api-access-f2828\" (UniqueName: \"kubernetes.io/projected/f0b8ef83-4677-405d-be44-4351cca0f47f-kube-api-access-f2828\") pod \"f0b8ef83-4677-405d-be44-4351cca0f47f\" (UID: \"f0b8ef83-4677-405d-be44-4351cca0f47f\") " Nov 02 06:38:18.380389 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:18.380266 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f0b8ef83-4677-405d-be44-4351cca0f47f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0b8ef83-4677-405d-be44-4351cca0f47f-maas-api-xks-ca\") pod \"f0b8ef83-4677-405d-be44-4351cca0f47f\" (UID: \"f0b8ef83-4677-405d-be44-4351cca0f47f\") " Nov 02 06:38:18.380479 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:18.380458 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0b8ef83-4677-405d-be44-4351cca0f47f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0b8ef83-4677-405d-be44-4351cca0f47f-maas-api-openshift-ca\") pod \"f0b8ef83-4677-405d-be44-4351cca0f47f\" (UID: \"f0b8ef83-4677-405d-be44-4351cca0f47f\") " Nov 02 06:38:18.380940 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:18.380886 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0b8ef83-4677-405d-be44-4351cca0f47f-maas-api-openshift-ca" pod "f0b8ef83-4677-405d-be44-4351cca0f47f" (UID: "f0b8ef83-4677-405d-be44-4351cca0f47f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:38:18.381131 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:18.381109 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0b8ef83-4677-405d-be44-4351cca0f47f-maas-api-openshift-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:38:18.383443 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:18.383415 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0b8ef83-4677-405d-be44-4351cca0f47f-maas-api-xks-ca" pod "f0b8ef83-4677-405d-be44-4351cca0f47f" (UID: "f0b8ef83-4677-405d-be44-4351cca0f47f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:38:18.384027 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:18.383975 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0b8ef83-4677-405d-be44-4351cca0f47f-kube-api-access-f2828" pod "f0b8ef83-4677-405d-be44-4351cca0f47f" (UID: "f0b8ef83-4677-405d-be44-4351cca0f47f"). InnerVolumeSpecName "kube-api-access-f2828". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:38:18.493653 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:18.488145 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2828\" (UniqueName: \"kubernetes.io/projected/f0b8ef83-4677-405d-be44-4351cca0f47f-kube-api-access-f2828\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:38:18.493653 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:18.488180 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0b8ef83-4677-405d-be44-4351cca0f47f-maas-api-xks-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:38:19.010598 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:19.010559 5916 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 06:38:19.010693 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:19.010653 5916 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 06:38:19.182231 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:19.182196 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98e3f2eb-9rds2" Nov 02 06:38:19.182608 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:19.182225 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98e3f2eb-9rds2" event={"ID":"f0b8ef83-4677-405d-be44-4351cca0f47f","Type":"ContainerDied","Data":"6be8baee5c8047a2df2507eb377ea7d7106a82604c3022c3f86d6b6633cda917"} Nov 02 06:38:19.182608 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:19.182581 5916 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6be8baee5c8047a2df2507eb377ea7d7106a82604c3022c3f86d6b6633cda917" Nov 02 06:38:19.186561 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:19.185919 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s6n9q" event={"ID":"ed18eacc-a61b-470d-8cd9-1d46f742fabd","Type":"ContainerStarted","Data":"a21a0b93a292549a41fd8b3ea404a92291828fa217d3c069dc1fa840bbacf326"} Nov 02 06:38:19.186561 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:19.185995 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s6n9q" Nov 02 06:38:19.316986 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:19.316840 5916 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s6n9q" podStartSLOduration=4.198534983 podStartE2EDuration="9.31682134s" totalImagesPullingTime="5.118286357s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:38:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:38:19.308406107 +0000 UTC m=+1287.580226515" watchObservedRunningTime="2026-11-02 06:38:19.31682134 +0000 UTC m=+1287.588641734" Nov 02 06:38:20.106163 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:38:20.105110 5916 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p98cq" containerName="extractor" Nov 02 06:38:20.385982 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:20.367200 5916 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-s6n9q container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 06:38:20.385982 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:20.367291 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s6n9q" podUID="ed18eacc-a61b-470d-8cd9-1d46f742fabd" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 06:38:21.106099 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:21.106021 5916 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-581608/simulator-w2-581608-kserve-68f68d8559-xkcwp" podUID="529e25e6-4479-49d6-85c7-85dad6c2193e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:21.249028 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:21.248982 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-581608/distinct-2-w2-581608-kserve-57dd49c4b-qnt9k" podUID="837780ad-8ea3-4ed2-922f-917cffc7bb93" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": context deadline exceeded" Nov 02 06:38:21.317194 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:21.317141 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6ff85e/simulator-w6-6ff85e-kserve-9b9fd978f-q76nl" podUID="30114285-d730-4abb-a618-6833d5746feb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:21.317731 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:21.317707 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-680b7b/simulator-w4-680b7b-kserve-6c4f574c56-p5xtc" podUID="be8a1039-5934-4885-93fe-070575f936ec" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Nov 02 06:38:21.437047 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:21.437000 5916 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-581608/distinct-w2-581608-kserve-6bb6bb7b65-v7kfj" podUID="dcf2f562-3c50-4de7-9c84-cb659d3bc25b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:21.472042 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:21.471991 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bdcd80/simulator-w0-bdcd80-kserve-5749c847cb-9ktcr" podUID="2043ff0a-8a71-4b71-9750-beb566a12bad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:21.517039 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:21.516995 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-581608/distinct-w2-581608-kserve-6bb6bb7b65-v7kfj" podUID="dcf2f562-3c50-4de7-9c84-cb659d3bc25b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:21.590188 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:21.590141 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6ff85e/premium-w6-6ff85e-kserve-59848ff6d4-szbxx" podUID="eb386976-f898-4f1d-b27c-77bccc735a3b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:21.743146 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:21.743048 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-581608/simulator-w2-581608-kserve-68f68d8559-xkcwp" podUID="529e25e6-4479-49d6-85c7-85dad6c2193e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Nov 02 06:38:21.811059 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:21.811016 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cdfce9/premium-w1-cdfce9-kserve-598cb9d57f-mrzrz" podUID="f9c5c1f7-cdca-46d6-8573-c85af1cd1946" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:21.933930 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:21.933455 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fb1398/distinct-2-w3-fb1398-kserve-597b7dc99b-dp4jf" podUID="523a31c4-4aea-4e9f-9ec4-7c28faefbf7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:21.965999 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:21.952142 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cdfce9/simulator-w1-cdfce9-kserve-7ffd68f78-5vwl2" podUID="b8b219aa-6a59-4d31-96f6-8d50ab910aca" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:22.088006 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:22.056437 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bdcd80/distinct-w0-bdcd80-kserve-5d9df79d48-bn5d4" podUID="1a7c7a03-e0cf-44ca-8eee-6c3e16112b3a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:22.252067 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:22.243131 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dcf9c3/distinct-2-w5-dcf9c3-kserve-64747dbd8f-wlwng" podUID="e37d42ab-7b62-4f07-8a29-cdedeb0fe89a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Nov 02 06:38:22.253734 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:22.253042 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-581608/distinct-2-w2-581608-kserve-57dd49c4b-qnt9k" podUID="837780ad-8ea3-4ed2-922f-917cffc7bb93" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:22.351992 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:22.329283 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-680b7b/simulator-w4-680b7b-kserve-6c4f574c56-p5xtc" podUID="be8a1039-5934-4885-93fe-070575f936ec" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:22.351992 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:22.330385 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6ff85e/simulator-w6-6ff85e-kserve-9b9fd978f-q76nl" podUID="30114285-d730-4abb-a618-6833d5746feb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Nov 02 06:38:22.504959 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:22.485886 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bdcd80/simulator-w0-bdcd80-kserve-5749c847cb-9ktcr" podUID="2043ff0a-8a71-4b71-9750-beb566a12bad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:22.504959 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:22.486286 5916 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-bdcd80/distinct-w0-bdcd80-kserve-5d9df79d48-bn5d4" podUID="1a7c7a03-e0cf-44ca-8eee-6c3e16112b3a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/health\": context deadline exceeded" Nov 02 06:38:22.532260 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:22.526159 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-581608/distinct-w2-581608-kserve-6bb6bb7b65-v7kfj" podUID="dcf2f562-3c50-4de7-9c84-cb659d3bc25b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:22.532260 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:22.527063 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-581608/distinct-2-w2-581608-kserve-57dd49c4b-qnt9k" Nov 02 06:38:22.532260 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:22.527119 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-680b7b/simulator-w4-680b7b-kserve-6c4f574c56-p5xtc" Nov 02 06:38:22.532260 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:22.527155 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-6ff85e/simulator-w6-6ff85e-kserve-9b9fd978f-q76nl" Nov 02 06:38:22.532260 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:22.527187 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-bdcd80/simulator-w0-bdcd80-kserve-5749c847cb-9ktcr" Nov 02 06:38:22.532260 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:22.527219 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-581608/distinct-w2-581608-kserve-6bb6bb7b65-v7kfj" Nov 02 06:38:22.735888 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:22.732006 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6ff85e/premium-w6-6ff85e-kserve-59848ff6d4-szbxx" podUID="eb386976-f898-4f1d-b27c-77bccc735a3b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Nov 02 06:38:22.735888 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:22.732568 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-6ff85e/premium-w6-6ff85e-kserve-59848ff6d4-szbxx" Nov 02 06:38:22.760724 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:22.760011 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-581608/simulator-w2-581608-kserve-68f68d8559-xkcwp" podUID="529e25e6-4479-49d6-85c7-85dad6c2193e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:22.760724 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:22.760369 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-581608/simulator-w2-581608-kserve-68f68d8559-xkcwp" Nov 02 06:38:22.814353 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:22.814310 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cdfce9/premium-w1-cdfce9-kserve-598cb9d57f-mrzrz" podUID="f9c5c1f7-cdca-46d6-8573-c85af1cd1946" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:22.814844 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:22.814822 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-cdfce9/premium-w1-cdfce9-kserve-598cb9d57f-mrzrz" Nov 02 06:38:22.956351 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:22.924429 5916 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-6ff85e/premium-w6-6ff85e-kserve-59848ff6d4-szbxx" podUID="eb386976-f898-4f1d-b27c-77bccc735a3b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:22.967942 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:22.957316 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fb1398/distinct-2-w3-fb1398-kserve-597b7dc99b-dp4jf" podUID="523a31c4-4aea-4e9f-9ec4-7c28faefbf7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:22.967942 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:22.957566 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-fb1398/distinct-2-w3-fb1398-kserve-597b7dc99b-dp4jf" Nov 02 06:38:22.967942 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:22.957273 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cdfce9/simulator-w1-cdfce9-kserve-7ffd68f78-5vwl2" podUID="b8b219aa-6a59-4d31-96f6-8d50ab910aca" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:23.144075 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:23.144033 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bdcd80/distinct-w0-bdcd80-kserve-5d9df79d48-bn5d4" podUID="1a7c7a03-e0cf-44ca-8eee-6c3e16112b3a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:23.264930 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:23.247481 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dcf9c3/distinct-2-w5-dcf9c3-kserve-64747dbd8f-wlwng" podUID="e37d42ab-7b62-4f07-8a29-cdedeb0fe89a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Nov 02 06:38:23.265893 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:23.265855 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-581608/distinct-2-w2-581608-kserve-57dd49c4b-qnt9k" podUID="837780ad-8ea3-4ed2-922f-917cffc7bb93" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:23.336088 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:23.336009 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6ff85e/simulator-w6-6ff85e-kserve-9b9fd978f-q76nl" podUID="30114285-d730-4abb-a618-6833d5746feb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:23.336361 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:23.336333 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-680b7b/simulator-w4-680b7b-kserve-6c4f574c56-p5xtc" podUID="be8a1039-5934-4885-93fe-070575f936ec" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Nov 02 06:38:23.345332 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:23.345307 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s6n9q" Nov 02 06:38:23.499207 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:23.499111 5916 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-cdfce9/premium-w1-cdfce9-kserve-598cb9d57f-mrzrz" podUID="f9c5c1f7-cdca-46d6-8573-c85af1cd1946" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": context deadline exceeded" Nov 02 06:38:23.547019 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:23.516207 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-cdfce9/simulator-w1-cdfce9-kserve-7ffd68f78-5vwl2" Nov 02 06:38:23.547214 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:23.540353 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-581608/distinct-w2-581608-kserve-6bb6bb7b65-v7kfj" podUID="dcf2f562-3c50-4de7-9c84-cb659d3bc25b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:23.547930 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:23.544838 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bdcd80/simulator-w0-bdcd80-kserve-5749c847cb-9ktcr" podUID="2043ff0a-8a71-4b71-9750-beb566a12bad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Nov 02 06:38:23.564060 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:23.564035 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-bdcd80/distinct-w0-bdcd80-kserve-5d9df79d48-bn5d4" Nov 02 06:38:23.564311 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:23.564279 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-dcf9c3/distinct-2-w5-dcf9c3-kserve-64747dbd8f-wlwng" Nov 02 06:38:23.835983 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:23.834864 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cdfce9/premium-w1-cdfce9-kserve-598cb9d57f-mrzrz" podUID="f9c5c1f7-cdca-46d6-8573-c85af1cd1946" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:23.835983 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:23.835317 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-581608/simulator-w2-581608-kserve-68f68d8559-xkcwp" podUID="529e25e6-4479-49d6-85c7-85dad6c2193e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Nov 02 06:38:24.000931 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:23.985042 5916 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-6ff85e/simulator-w6-6ff85e-kserve-9b9fd978f-q76nl" podUID="30114285-d730-4abb-a618-6833d5746feb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": context deadline exceeded" Nov 02 06:38:24.004525 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:24.004396 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fb1398/distinct-2-w3-fb1398-kserve-597b7dc99b-dp4jf" podUID="523a31c4-4aea-4e9f-9ec4-7c28faefbf7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": context deadline exceeded" Nov 02 06:38:24.685449 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:24.623779 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bdcd80/distinct-w0-bdcd80-kserve-5d9df79d48-bn5d4" podUID="1a7c7a03-e0cf-44ca-8eee-6c3e16112b3a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded" Nov 02 06:38:24.694928 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:24.624140 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dcf9c3/distinct-2-w5-dcf9c3-kserve-64747dbd8f-wlwng" podUID="e37d42ab-7b62-4f07-8a29-cdedeb0fe89a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Nov 02 06:38:24.695303 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:24.624332 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-581608/distinct-2-w2-581608-kserve-57dd49c4b-qnt9k" podUID="837780ad-8ea3-4ed2-922f-917cffc7bb93" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": context deadline exceeded" Nov 02 06:38:24.695550 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:24.624532 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6ff85e/simulator-w6-6ff85e-kserve-9b9fd978f-q76nl" podUID="30114285-d730-4abb-a618-6833d5746feb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Nov 02 06:38:24.695791 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:24.624691 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-680b7b/simulator-w4-680b7b-kserve-6c4f574c56-p5xtc" podUID="be8a1039-5934-4885-93fe-070575f936ec" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Nov 02 06:38:24.696026 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:24.634122 5916 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-dcf9c3/distinct-2-w5-dcf9c3-kserve-64747dbd8f-wlwng" podUID="e37d42ab-7b62-4f07-8a29-cdedeb0fe89a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/health\": context deadline exceeded" Nov 02 06:38:24.696232 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:24.656719 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cdfce9/simulator-w1-cdfce9-kserve-7ffd68f78-5vwl2" podUID="b8b219aa-6a59-4d31-96f6-8d50ab910aca" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:24.707882 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:24.707846 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-581608/distinct-w2-581608-kserve-6bb6bb7b65-v7kfj" podUID="dcf2f562-3c50-4de7-9c84-cb659d3bc25b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded" Nov 02 06:38:24.840085 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:24.840038 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cdfce9/premium-w1-cdfce9-kserve-598cb9d57f-mrzrz" podUID="f9c5c1f7-cdca-46d6-8573-c85af1cd1946" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Nov 02 06:38:24.980375 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:24.980265 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6ff85e/premium-w6-6ff85e-kserve-59848ff6d4-szbxx" podUID="eb386976-f898-4f1d-b27c-77bccc735a3b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Nov 02 06:38:25.021089 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:25.020368 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bdcd80/simulator-w0-bdcd80-kserve-5749c847cb-9ktcr" podUID="2043ff0a-8a71-4b71-9750-beb566a12bad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:25.124344 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:38:25.121924 5916 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.242s" Nov 02 06:38:25.124344 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:25.122058 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-fb1398/distinct-2-w3-fb1398-kserve-597b7dc99b-dp4jf" Nov 02 06:38:25.124344 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:25.122100 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-581608/simulator-w2-581608-kserve-68f68d8559-xkcwp" Nov 02 06:38:25.124344 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:25.122132 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-cdfce9/simulator-w1-cdfce9-kserve-7ffd68f78-5vwl2" Nov 02 06:38:25.124344 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:25.122164 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-dcf9c3/distinct-2-w5-dcf9c3-kserve-64747dbd8f-wlwng" Nov 02 06:38:25.124344 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:25.122193 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-6ff85e/simulator-w6-6ff85e-kserve-9b9fd978f-q76nl" Nov 02 06:38:25.124769 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:25.124733 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-680b7b/simulator-w4-680b7b-kserve-6c4f574c56-p5xtc" Nov 02 06:38:25.128288 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:25.128267 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-581608/distinct-2-w2-581608-kserve-57dd49c4b-qnt9k" Nov 02 06:38:25.132213 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:25.132191 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-bdcd80/simulator-w0-bdcd80-kserve-5749c847cb-9ktcr" Nov 02 06:38:25.142996 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:25.142974 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-cdfce9/premium-w1-cdfce9-kserve-598cb9d57f-mrzrz" Nov 02 06:38:25.148306 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:25.148287 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-6ff85e/premium-w6-6ff85e-kserve-59848ff6d4-szbxx" Nov 02 06:38:25.171319 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:25.171289 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-bdcd80/distinct-w0-bdcd80-kserve-5d9df79d48-bn5d4" Nov 02 06:38:25.200036 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:25.200001 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-581608/distinct-w2-581608-kserve-6bb6bb7b65-v7kfj" Nov 02 06:38:33.263662 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:33.263624 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-v2lws"] Nov 02 06:38:33.264610 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:33.264588 5916 cpu_manager.go:414] "containerMap: removing container" podUID="f0b8ef83-4677-405d-be44-4351cca0f47f" containerName="revoke-keys" Nov 02 06:38:33.264678 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:33.264613 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0b8ef83-4677-405d-be44-4351cca0f47f" containerName="revoke-keys" Nov 02 06:38:33.264813 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:33.264800 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0b8ef83-4677-405d-be44-4351cca0f47f" containerName="revoke-keys" Nov 02 06:38:33.270600 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:33.270575 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-v2lws" Nov 02 06:38:33.349728 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:33.349693 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e0634b1-6e2e-480b-9e74-27cc8525ff6b-istio-data\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-v2lws\" (UID: \"9e0634b1-6e2e-480b-9e74-27cc8525ff6b\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-v2lws" Nov 02 06:38:33.349890 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:33.349745 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e0634b1-6e2e-480b-9e74-27cc8525ff6b-credential-socket\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-v2lws\" (UID: \"9e0634b1-6e2e-480b-9e74-27cc8525ff6b\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-v2lws" Nov 02 06:38:33.349952 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:33.349875 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vss9x\" (UniqueName: \"kubernetes.io/projected/9e0634b1-6e2e-480b-9e74-27cc8525ff6b-kube-api-access-vss9x\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-v2lws\" (UID: \"9e0634b1-6e2e-480b-9e74-27cc8525ff6b\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-v2lws" Nov 02 06:38:33.350001 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:33.349977 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e0634b1-6e2e-480b-9e74-27cc8525ff6b-istio-podinfo\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-v2lws\" (UID: \"9e0634b1-6e2e-480b-9e74-27cc8525ff6b\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-v2lws" Nov 02 06:38:33.350037 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:33.350007 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e0634b1-6e2e-480b-9e74-27cc8525ff6b-istiod-ca-cert\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-v2lws\" (UID: \"9e0634b1-6e2e-480b-9e74-27cc8525ff6b\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-v2lws" Nov 02 06:38:33.350068 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:33.350059 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e0634b1-6e2e-480b-9e74-27cc8525ff6b-workload-certs\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-v2lws\" (UID: \"9e0634b1-6e2e-480b-9e74-27cc8525ff6b\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-v2lws" Nov 02 06:38:33.350134 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:33.350119 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e0634b1-6e2e-480b-9e74-27cc8525ff6b-istio-envoy\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-v2lws\" (UID: \"9e0634b1-6e2e-480b-9e74-27cc8525ff6b\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-v2lws" Nov 02 06:38:33.350182 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:33.350171 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e0634b1-6e2e-480b-9e74-27cc8525ff6b-istio-token\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-v2lws\" (UID: \"9e0634b1-6e2e-480b-9e74-27cc8525ff6b\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-v2lws" Nov 02 06:38:33.350359 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:33.350339 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e0634b1-6e2e-480b-9e74-27cc8525ff6b-workload-socket\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-v2lws\" (UID: \"9e0634b1-6e2e-480b-9e74-27cc8525ff6b\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-v2lws" Nov 02 06:38:33.381123 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:33.381087 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-v2lws"] Nov 02 06:38:33.451174 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:33.451137 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e0634b1-6e2e-480b-9e74-27cc8525ff6b-workload-socket\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-v2lws\" (UID: \"9e0634b1-6e2e-480b-9e74-27cc8525ff6b\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-v2lws" Nov 02 06:38:33.451351 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:33.451262 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e0634b1-6e2e-480b-9e74-27cc8525ff6b-istio-data\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-v2lws\" (UID: \"9e0634b1-6e2e-480b-9e74-27cc8525ff6b\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-v2lws" Nov 02 06:38:33.451351 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:33.451308 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e0634b1-6e2e-480b-9e74-27cc8525ff6b-credential-socket\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-v2lws\" (UID: \"9e0634b1-6e2e-480b-9e74-27cc8525ff6b\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-v2lws" Nov 02 06:38:33.451476 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:33.451363 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vss9x\" (UniqueName: \"kubernetes.io/projected/9e0634b1-6e2e-480b-9e74-27cc8525ff6b-kube-api-access-vss9x\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-v2lws\" (UID: \"9e0634b1-6e2e-480b-9e74-27cc8525ff6b\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-v2lws" Nov 02 06:38:33.451476 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:33.451432 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e0634b1-6e2e-480b-9e74-27cc8525ff6b-istio-podinfo\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-v2lws\" (UID: \"9e0634b1-6e2e-480b-9e74-27cc8525ff6b\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-v2lws" Nov 02 06:38:33.451476 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:33.451463 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e0634b1-6e2e-480b-9e74-27cc8525ff6b-istiod-ca-cert\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-v2lws\" (UID: \"9e0634b1-6e2e-480b-9e74-27cc8525ff6b\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-v2lws" Nov 02 06:38:33.451615 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:33.451515 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e0634b1-6e2e-480b-9e74-27cc8525ff6b-workload-certs\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-v2lws\" (UID: \"9e0634b1-6e2e-480b-9e74-27cc8525ff6b\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-v2lws" Nov 02 06:38:33.451615 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:33.451545 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e0634b1-6e2e-480b-9e74-27cc8525ff6b-istio-envoy\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-v2lws\" (UID: \"9e0634b1-6e2e-480b-9e74-27cc8525ff6b\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-v2lws" Nov 02 06:38:33.451615 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:33.451607 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e0634b1-6e2e-480b-9e74-27cc8525ff6b-istio-token\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-v2lws\" (UID: \"9e0634b1-6e2e-480b-9e74-27cc8525ff6b\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-v2lws" Nov 02 06:38:33.451768 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:33.451621 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e0634b1-6e2e-480b-9e74-27cc8525ff6b-workload-socket\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-v2lws\" (UID: \"9e0634b1-6e2e-480b-9e74-27cc8525ff6b\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-v2lws" Nov 02 06:38:33.451768 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:33.451653 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e0634b1-6e2e-480b-9e74-27cc8525ff6b-istio-data\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-v2lws\" (UID: \"9e0634b1-6e2e-480b-9e74-27cc8525ff6b\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-v2lws" Nov 02 06:38:33.452019 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:33.451995 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e0634b1-6e2e-480b-9e74-27cc8525ff6b-workload-certs\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-v2lws\" (UID: \"9e0634b1-6e2e-480b-9e74-27cc8525ff6b\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-v2lws" Nov 02 06:38:33.452185 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:33.452161 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e0634b1-6e2e-480b-9e74-27cc8525ff6b-credential-socket\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-v2lws\" (UID: \"9e0634b1-6e2e-480b-9e74-27cc8525ff6b\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-v2lws" Nov 02 06:38:33.452450 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:33.452427 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e0634b1-6e2e-480b-9e74-27cc8525ff6b-istiod-ca-cert\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-v2lws\" (UID: \"9e0634b1-6e2e-480b-9e74-27cc8525ff6b\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-v2lws" Nov 02 06:38:33.454352 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:33.454329 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e0634b1-6e2e-480b-9e74-27cc8525ff6b-istio-envoy\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-v2lws\" (UID: \"9e0634b1-6e2e-480b-9e74-27cc8525ff6b\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-v2lws" Nov 02 06:38:33.454569 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:33.454551 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e0634b1-6e2e-480b-9e74-27cc8525ff6b-istio-podinfo\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-v2lws\" (UID: \"9e0634b1-6e2e-480b-9e74-27cc8525ff6b\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-v2lws" Nov 02 06:38:33.510259 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:33.510224 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e0634b1-6e2e-480b-9e74-27cc8525ff6b-istio-token\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-v2lws\" (UID: \"9e0634b1-6e2e-480b-9e74-27cc8525ff6b\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-v2lws" Nov 02 06:38:33.510430 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:33.510235 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vss9x\" (UniqueName: \"kubernetes.io/projected/9e0634b1-6e2e-480b-9e74-27cc8525ff6b-kube-api-access-vss9x\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-v2lws\" (UID: \"9e0634b1-6e2e-480b-9e74-27cc8525ff6b\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-v2lws" Nov 02 06:38:33.584947 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:33.584830 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-v2lws" Nov 02 06:38:34.033606 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:38:34.033572 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e0634b1_6e2e_480b_9e74_27cc8525ff6b.slice/crio-9816c54ef13dde72b930a74fa054e9b58d73f34506cd6a80808500eeec82415a WatchSource:0}: Error finding container 9816c54ef13dde72b930a74fa054e9b58d73f34506cd6a80808500eeec82415a: Status 404 returned error can't find the container with id 9816c54ef13dde72b930a74fa054e9b58d73f34506cd6a80808500eeec82415a Nov 02 06:38:34.034513 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:34.034376 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-v2lws"] Nov 02 06:38:34.036840 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:34.036797 5916 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 06:38:34.036970 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:34.036881 5916 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 06:38:34.036970 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:34.036953 5916 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 06:38:34.255715 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:34.247124 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-v2lws" event={"ID":"9e0634b1-6e2e-480b-9e74-27cc8525ff6b","Type":"ContainerStarted","Data":"28d667e56b297c4ba9d0b92fc1e6681613cec18fe69ff61c733c726a468bb970"} Nov 02 06:38:34.255715 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:34.247203 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-v2lws" event={"ID":"9e0634b1-6e2e-480b-9e74-27cc8525ff6b","Type":"ContainerStarted","Data":"9816c54ef13dde72b930a74fa054e9b58d73f34506cd6a80808500eeec82415a"} Nov 02 06:38:34.369100 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:34.368872 5916 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-v2lws" podStartSLOduration=1.368851339 podStartE2EDuration="1.368851339s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:38:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:38:34.363483718 +0000 UTC m=+1302.635304112" watchObservedRunningTime="2026-11-02 06:38:34.368851339 +0000 UTC m=+1302.640671735" Nov 02 06:38:34.585331 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:34.585288 5916 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-v2lws" Nov 02 06:38:35.587143 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:35.587109 5916 patch_prober.go:29] interesting pod/e2e-worker-w6-6ff85e-openshift-default-698585659c-v2lws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Nov 02 06:38:35.587554 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:35.587170 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-v2lws" podUID="9e0634b1-6e2e-480b-9e74-27cc8525ff6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Nov 02 06:38:36.585737 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:36.585703 5916 patch_prober.go:29] interesting pod/e2e-worker-w6-6ff85e-openshift-default-698585659c-v2lws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Nov 02 06:38:36.585883 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:36.585768 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-v2lws" podUID="9e0634b1-6e2e-480b-9e74-27cc8525ff6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Nov 02 06:38:37.585266 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:37.585233 5916 patch_prober.go:29] interesting pod/e2e-worker-w6-6ff85e-openshift-default-698585659c-v2lws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Nov 02 06:38:37.585778 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:37.585306 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-v2lws" podUID="9e0634b1-6e2e-480b-9e74-27cc8525ff6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Nov 02 06:38:38.585927 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:38.585877 5916 patch_prober.go:29] interesting pod/e2e-worker-w6-6ff85e-openshift-default-698585659c-v2lws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Nov 02 06:38:38.586448 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:38.585972 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-v2lws" podUID="9e0634b1-6e2e-480b-9e74-27cc8525ff6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Nov 02 06:38:39.585135 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:39.585102 5916 patch_prober.go:29] interesting pod/e2e-worker-w6-6ff85e-openshift-default-698585659c-v2lws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Nov 02 06:38:39.585306 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:39.585155 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-v2lws" podUID="9e0634b1-6e2e-480b-9e74-27cc8525ff6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Nov 02 06:38:40.585309 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:40.585269 5916 patch_prober.go:29] interesting pod/e2e-worker-w6-6ff85e-openshift-default-698585659c-v2lws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Nov 02 06:38:40.585809 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:40.585325 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-v2lws" podUID="9e0634b1-6e2e-480b-9e74-27cc8525ff6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Nov 02 06:38:41.701569 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:41.698755 5916 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-v2lws" Nov 02 06:38:41.701569 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:41.700387 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-v2lws" Nov 02 06:38:41.707932 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:41.705229 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-v2lws" Nov 02 06:38:47.187295 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:47.187259 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-p5j6w"] Nov 02 06:38:47.191739 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:47.191718 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-p5j6w" Nov 02 06:38:47.219178 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:47.219151 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/835047d2-d7ce-4dcf-95a4-449fb987d4a7-istio-data\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-p5j6w\" (UID: \"835047d2-d7ce-4dcf-95a4-449fb987d4a7\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-p5j6w" Nov 02 06:38:47.219272 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:47.219217 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/835047d2-d7ce-4dcf-95a4-449fb987d4a7-istiod-ca-cert\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-p5j6w\" (UID: \"835047d2-d7ce-4dcf-95a4-449fb987d4a7\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-p5j6w" Nov 02 06:38:47.219328 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:47.219309 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/835047d2-d7ce-4dcf-95a4-449fb987d4a7-credential-socket\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-p5j6w\" (UID: \"835047d2-d7ce-4dcf-95a4-449fb987d4a7\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-p5j6w" Nov 02 06:38:47.219367 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:47.219354 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/835047d2-d7ce-4dcf-95a4-449fb987d4a7-istio-token\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-p5j6w\" (UID: \"835047d2-d7ce-4dcf-95a4-449fb987d4a7\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-p5j6w" Nov 02 06:38:47.219406 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:47.219392 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/835047d2-d7ce-4dcf-95a4-449fb987d4a7-workload-socket\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-p5j6w\" (UID: \"835047d2-d7ce-4dcf-95a4-449fb987d4a7\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-p5j6w" Nov 02 06:38:47.219482 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:47.219471 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/835047d2-d7ce-4dcf-95a4-449fb987d4a7-istio-podinfo\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-p5j6w\" (UID: \"835047d2-d7ce-4dcf-95a4-449fb987d4a7\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-p5j6w" Nov 02 06:38:47.219527 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:47.219492 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czgnt\" (UniqueName: \"kubernetes.io/projected/835047d2-d7ce-4dcf-95a4-449fb987d4a7-kube-api-access-czgnt\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-p5j6w\" (UID: \"835047d2-d7ce-4dcf-95a4-449fb987d4a7\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-p5j6w" Nov 02 06:38:47.219579 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:47.219565 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/835047d2-d7ce-4dcf-95a4-449fb987d4a7-workload-certs\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-p5j6w\" (UID: \"835047d2-d7ce-4dcf-95a4-449fb987d4a7\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-p5j6w" Nov 02 06:38:47.219618 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:47.219599 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/835047d2-d7ce-4dcf-95a4-449fb987d4a7-istio-envoy\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-p5j6w\" (UID: \"835047d2-d7ce-4dcf-95a4-449fb987d4a7\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-p5j6w" Nov 02 06:38:47.321300 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:47.321257 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/835047d2-d7ce-4dcf-95a4-449fb987d4a7-istio-data\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-p5j6w\" (UID: \"835047d2-d7ce-4dcf-95a4-449fb987d4a7\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-p5j6w" Nov 02 06:38:47.321481 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:47.321346 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/835047d2-d7ce-4dcf-95a4-449fb987d4a7-istiod-ca-cert\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-p5j6w\" (UID: \"835047d2-d7ce-4dcf-95a4-449fb987d4a7\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-p5j6w" Nov 02 06:38:47.321481 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:47.321401 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/835047d2-d7ce-4dcf-95a4-449fb987d4a7-credential-socket\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-p5j6w\" (UID: \"835047d2-d7ce-4dcf-95a4-449fb987d4a7\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-p5j6w" Nov 02 06:38:47.321481 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:47.321458 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/835047d2-d7ce-4dcf-95a4-449fb987d4a7-istio-token\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-p5j6w\" (UID: \"835047d2-d7ce-4dcf-95a4-449fb987d4a7\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-p5j6w" Nov 02 06:38:47.321662 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:47.321487 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/835047d2-d7ce-4dcf-95a4-449fb987d4a7-workload-socket\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-p5j6w\" (UID: \"835047d2-d7ce-4dcf-95a4-449fb987d4a7\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-p5j6w" Nov 02 06:38:47.321662 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:47.321574 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/835047d2-d7ce-4dcf-95a4-449fb987d4a7-istio-podinfo\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-p5j6w\" (UID: \"835047d2-d7ce-4dcf-95a4-449fb987d4a7\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-p5j6w" Nov 02 06:38:47.321662 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:47.321605 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czgnt\" (UniqueName: \"kubernetes.io/projected/835047d2-d7ce-4dcf-95a4-449fb987d4a7-kube-api-access-czgnt\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-p5j6w\" (UID: \"835047d2-d7ce-4dcf-95a4-449fb987d4a7\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-p5j6w" Nov 02 06:38:47.321811 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:47.321676 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/835047d2-d7ce-4dcf-95a4-449fb987d4a7-workload-certs\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-p5j6w\" (UID: \"835047d2-d7ce-4dcf-95a4-449fb987d4a7\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-p5j6w" Nov 02 06:38:47.321811 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:47.321675 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/835047d2-d7ce-4dcf-95a4-449fb987d4a7-istio-data\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-p5j6w\" (UID: \"835047d2-d7ce-4dcf-95a4-449fb987d4a7\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-p5j6w" Nov 02 06:38:47.321811 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:47.321743 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/835047d2-d7ce-4dcf-95a4-449fb987d4a7-istio-envoy\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-p5j6w\" (UID: \"835047d2-d7ce-4dcf-95a4-449fb987d4a7\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-p5j6w" Nov 02 06:38:47.321975 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:47.321807 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/835047d2-d7ce-4dcf-95a4-449fb987d4a7-credential-socket\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-p5j6w\" (UID: \"835047d2-d7ce-4dcf-95a4-449fb987d4a7\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-p5j6w" Nov 02 06:38:47.322095 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:47.322077 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/835047d2-d7ce-4dcf-95a4-449fb987d4a7-workload-certs\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-p5j6w\" (UID: \"835047d2-d7ce-4dcf-95a4-449fb987d4a7\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-p5j6w" Nov 02 06:38:47.322209 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:47.322189 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/835047d2-d7ce-4dcf-95a4-449fb987d4a7-workload-socket\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-p5j6w\" (UID: \"835047d2-d7ce-4dcf-95a4-449fb987d4a7\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-p5j6w" Nov 02 06:38:47.322266 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:47.322221 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/835047d2-d7ce-4dcf-95a4-449fb987d4a7-istiod-ca-cert\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-p5j6w\" (UID: \"835047d2-d7ce-4dcf-95a4-449fb987d4a7\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-p5j6w" Nov 02 06:38:47.323879 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:47.323860 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/835047d2-d7ce-4dcf-95a4-449fb987d4a7-istio-envoy\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-p5j6w\" (UID: \"835047d2-d7ce-4dcf-95a4-449fb987d4a7\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-p5j6w" Nov 02 06:38:47.324344 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:47.324322 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/835047d2-d7ce-4dcf-95a4-449fb987d4a7-istio-podinfo\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-p5j6w\" (UID: \"835047d2-d7ce-4dcf-95a4-449fb987d4a7\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-p5j6w" Nov 02 06:38:47.382724 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:47.382694 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/835047d2-d7ce-4dcf-95a4-449fb987d4a7-istio-token\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-p5j6w\" (UID: \"835047d2-d7ce-4dcf-95a4-449fb987d4a7\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-p5j6w" Nov 02 06:38:47.382979 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:47.382856 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czgnt\" (UniqueName: \"kubernetes.io/projected/835047d2-d7ce-4dcf-95a4-449fb987d4a7-kube-api-access-czgnt\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-p5j6w\" (UID: \"835047d2-d7ce-4dcf-95a4-449fb987d4a7\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-p5j6w" Nov 02 06:38:47.407635 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:47.407603 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-p5j6w"] Nov 02 06:38:47.508412 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:47.508340 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-p5j6w" Nov 02 06:38:47.947807 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:38:47.947780 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod835047d2_d7ce_4dcf_95a4_449fb987d4a7.slice/crio-4b5d6fd272be8c2610d4284a2975d3239ac2b24e3f0e99a8ef96a8fffdcd7905 WatchSource:0}: Error finding container 4b5d6fd272be8c2610d4284a2975d3239ac2b24e3f0e99a8ef96a8fffdcd7905: Status 404 returned error can't find the container with id 4b5d6fd272be8c2610d4284a2975d3239ac2b24e3f0e99a8ef96a8fffdcd7905 Nov 02 06:38:47.948586 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:47.948561 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-p5j6w"] Nov 02 06:38:47.950556 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:47.950523 5916 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 06:38:47.950643 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:47.950586 5916 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 06:38:47.950643 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:47.950626 5916 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 06:38:48.397976 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:48.397866 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-p5j6w" event={"ID":"835047d2-d7ce-4dcf-95a4-449fb987d4a7","Type":"ContainerStarted","Data":"e695b9e0fa78af668e201eefa11352870bb9c707c017064de549208bdb5e7388"} Nov 02 06:38:48.397976 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:48.397973 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-p5j6w" event={"ID":"835047d2-d7ce-4dcf-95a4-449fb987d4a7","Type":"ContainerStarted","Data":"4b5d6fd272be8c2610d4284a2975d3239ac2b24e3f0e99a8ef96a8fffdcd7905"} Nov 02 06:38:48.508752 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:48.508706 5916 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-p5j6w" Nov 02 06:38:48.523621 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:48.523558 5916 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-p5j6w" podStartSLOduration=1.523539361 podStartE2EDuration="1.523539361s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:38:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:38:48.520110877 +0000 UTC m=+1316.791931271" watchObservedRunningTime="2026-11-02 06:38:48.523539361 +0000 UTC m=+1316.795359755" Nov 02 06:38:49.510076 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:49.510039 5916 patch_prober.go:29] interesting pod/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-p5j6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Nov 02 06:38:49.510472 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:49.510100 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-p5j6w" podUID="835047d2-d7ce-4dcf-95a4-449fb987d4a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Nov 02 06:38:49.895673 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:38:49.895641 5916 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p98cq" containerName="extractor" Nov 02 06:38:50.509253 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:50.509218 5916 patch_prober.go:29] interesting pod/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-p5j6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Nov 02 06:38:50.509429 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:50.509289 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-p5j6w" podUID="835047d2-d7ce-4dcf-95a4-449fb987d4a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Nov 02 06:38:51.509000 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:51.508966 5916 patch_prober.go:29] interesting pod/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-p5j6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Nov 02 06:38:51.509449 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:51.509026 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-p5j6w" podUID="835047d2-d7ce-4dcf-95a4-449fb987d4a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Nov 02 06:38:52.509552 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:52.509519 5916 patch_prober.go:29] interesting pod/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-p5j6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Nov 02 06:38:52.509981 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:52.509581 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-p5j6w" podUID="835047d2-d7ce-4dcf-95a4-449fb987d4a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Nov 02 06:38:53.301820 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:53.301793 5916 scope.go:122] "RemoveContainer" containerID="4e77001acde5fbe21c1888db1488f3ce8c25bcc4839153ba714547d7bc1020f7" Nov 02 06:38:53.313558 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:53.313534 5916 scope.go:122] "RemoveContainer" containerID="44fc1b54d1a6c81c5029b5f28100a193e9bf2088944f9961337ed9e3124ed787" Nov 02 06:38:53.323838 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:53.323813 5916 scope.go:122] "RemoveContainer" containerID="63fa2ffb3d304e2025ededdbd05f59c7b5f10a819e93379a48e1076b52df072e" Nov 02 06:38:53.369232 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:53.369205 5916 scope.go:122] "RemoveContainer" containerID="0d968e7ca96e5944c2f67ec7d9904b8b71c8a2337b26a236b19c978e50577e73" Nov 02 06:38:53.381075 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:53.381052 5916 scope.go:122] "RemoveContainer" containerID="e852ba9a877b46c96e7e512bec2fee1de3f6e2c1a1504cfcaeab5d7ccf65e1ef" Nov 02 06:38:53.391541 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:53.391516 5916 scope.go:122] "RemoveContainer" containerID="fadb7833ba0092324d5ea7df340ac1f4a989404522061fab3c1dccf120a7977b" Nov 02 06:38:53.402085 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:53.402058 5916 scope.go:122] "RemoveContainer" containerID="ec69be554402404194189fa794deae4d5f190aa498b36f57b88ae236feabfdc9" Nov 02 06:38:53.412709 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:53.412688 5916 scope.go:122] "RemoveContainer" containerID="41c963fefe4b4c30e0ca41187723250553da6f5a56dbdc215f0541443e815ef2" Nov 02 06:38:53.424076 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:53.424049 5916 scope.go:122] "RemoveContainer" containerID="6642695f0202413e2ddb181b9e21d3ecd366fc0d5b4023eb6af7564f6a89c7fc" Nov 02 06:38:53.475457 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:53.475436 5916 scope.go:122] "RemoveContainer" containerID="e27ed737da6f8f9053ffbb759fc738062685d3238d9ad9f745f727635bc9b18a" Nov 02 06:38:53.509498 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:53.509470 5916 patch_prober.go:29] interesting pod/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-p5j6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Nov 02 06:38:53.509610 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:53.509556 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-p5j6w" podUID="835047d2-d7ce-4dcf-95a4-449fb987d4a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Nov 02 06:38:54.509415 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:54.509384 5916 patch_prober.go:29] interesting pod/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-p5j6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Nov 02 06:38:54.509597 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:54.509440 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-p5j6w" podUID="835047d2-d7ce-4dcf-95a4-449fb987d4a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Nov 02 06:38:55.514581 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:55.514548 5916 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-p5j6w" Nov 02 06:38:55.515113 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:55.515011 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-p5j6w" Nov 02 06:38:55.515738 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:38:55.515716 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-p5j6w" Nov 02 06:39:19.895445 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:39:19.895399 5916 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p98cq" containerName="extractor" Nov 02 06:39:21.670716 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:21.670683 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-r9cmb"] Nov 02 06:39:21.675027 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:21.675007 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-r9cmb" Nov 02 06:39:21.764996 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:21.764966 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d018c073-2cde-4cbc-8209-a2d788777d2a-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-r9cmb\" (UID: \"d018c073-2cde-4cbc-8209-a2d788777d2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-r9cmb" Nov 02 06:39:21.765183 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:21.765094 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d018c073-2cde-4cbc-8209-a2d788777d2a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-r9cmb\" (UID: \"d018c073-2cde-4cbc-8209-a2d788777d2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-r9cmb" Nov 02 06:39:21.765183 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:21.765171 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d018c073-2cde-4cbc-8209-a2d788777d2a-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-r9cmb\" (UID: \"d018c073-2cde-4cbc-8209-a2d788777d2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-r9cmb" Nov 02 06:39:21.765309 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:21.765263 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d018c073-2cde-4cbc-8209-a2d788777d2a-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-r9cmb\" (UID: \"d018c073-2cde-4cbc-8209-a2d788777d2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-r9cmb" Nov 02 06:39:21.765368 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:21.765309 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d018c073-2cde-4cbc-8209-a2d788777d2a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-r9cmb\" (UID: \"d018c073-2cde-4cbc-8209-a2d788777d2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-r9cmb" Nov 02 06:39:21.765368 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:21.765330 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d018c073-2cde-4cbc-8209-a2d788777d2a-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-r9cmb\" (UID: \"d018c073-2cde-4cbc-8209-a2d788777d2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-r9cmb" Nov 02 06:39:21.765491 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:21.765467 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d018c073-2cde-4cbc-8209-a2d788777d2a-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-r9cmb\" (UID: \"d018c073-2cde-4cbc-8209-a2d788777d2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-r9cmb" Nov 02 06:39:21.765549 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:21.765497 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spr4h\" (UniqueName: \"kubernetes.io/projected/d018c073-2cde-4cbc-8209-a2d788777d2a-kube-api-access-spr4h\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-r9cmb\" (UID: \"d018c073-2cde-4cbc-8209-a2d788777d2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-r9cmb" Nov 02 06:39:21.765549 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:21.765517 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d018c073-2cde-4cbc-8209-a2d788777d2a-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-r9cmb\" (UID: \"d018c073-2cde-4cbc-8209-a2d788777d2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-r9cmb" Nov 02 06:39:21.866435 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:21.866400 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d018c073-2cde-4cbc-8209-a2d788777d2a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-r9cmb\" (UID: \"d018c073-2cde-4cbc-8209-a2d788777d2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-r9cmb" Nov 02 06:39:21.866663 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:21.866504 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d018c073-2cde-4cbc-8209-a2d788777d2a-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-r9cmb\" (UID: \"d018c073-2cde-4cbc-8209-a2d788777d2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-r9cmb" Nov 02 06:39:21.866733 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:21.866687 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d018c073-2cde-4cbc-8209-a2d788777d2a-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-r9cmb\" (UID: \"d018c073-2cde-4cbc-8209-a2d788777d2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-r9cmb" Nov 02 06:39:21.866780 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:21.866769 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d018c073-2cde-4cbc-8209-a2d788777d2a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-r9cmb\" (UID: \"d018c073-2cde-4cbc-8209-a2d788777d2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-r9cmb" Nov 02 06:39:21.866825 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:21.866804 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d018c073-2cde-4cbc-8209-a2d788777d2a-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-r9cmb\" (UID: \"d018c073-2cde-4cbc-8209-a2d788777d2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-r9cmb" Nov 02 06:39:21.866954 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:21.866900 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d018c073-2cde-4cbc-8209-a2d788777d2a-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-r9cmb\" (UID: \"d018c073-2cde-4cbc-8209-a2d788777d2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-r9cmb" Nov 02 06:39:21.866954 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:21.866942 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spr4h\" (UniqueName: \"kubernetes.io/projected/d018c073-2cde-4cbc-8209-a2d788777d2a-kube-api-access-spr4h\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-r9cmb\" (UID: \"d018c073-2cde-4cbc-8209-a2d788777d2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-r9cmb" Nov 02 06:39:21.867695 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:21.866974 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d018c073-2cde-4cbc-8209-a2d788777d2a-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-r9cmb\" (UID: \"d018c073-2cde-4cbc-8209-a2d788777d2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-r9cmb" Nov 02 06:39:21.867695 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:21.867001 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d018c073-2cde-4cbc-8209-a2d788777d2a-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-r9cmb\" (UID: \"d018c073-2cde-4cbc-8209-a2d788777d2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-r9cmb" Nov 02 06:39:21.867695 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:21.867283 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d018c073-2cde-4cbc-8209-a2d788777d2a-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-r9cmb\" (UID: \"d018c073-2cde-4cbc-8209-a2d788777d2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-r9cmb" Nov 02 06:39:21.867695 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:21.867403 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d018c073-2cde-4cbc-8209-a2d788777d2a-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-r9cmb\" (UID: \"d018c073-2cde-4cbc-8209-a2d788777d2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-r9cmb" Nov 02 06:39:21.867695 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:21.867476 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d018c073-2cde-4cbc-8209-a2d788777d2a-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-r9cmb\" (UID: \"d018c073-2cde-4cbc-8209-a2d788777d2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-r9cmb" Nov 02 06:39:21.867695 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:21.867603 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d018c073-2cde-4cbc-8209-a2d788777d2a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-r9cmb\" (UID: \"d018c073-2cde-4cbc-8209-a2d788777d2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-r9cmb" Nov 02 06:39:21.868572 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:21.868549 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d018c073-2cde-4cbc-8209-a2d788777d2a-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-r9cmb\" (UID: \"d018c073-2cde-4cbc-8209-a2d788777d2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-r9cmb" Nov 02 06:39:21.869585 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:21.869541 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d018c073-2cde-4cbc-8209-a2d788777d2a-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-r9cmb\" (UID: \"d018c073-2cde-4cbc-8209-a2d788777d2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-r9cmb" Nov 02 06:39:21.870406 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:21.870379 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d018c073-2cde-4cbc-8209-a2d788777d2a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-r9cmb\" (UID: \"d018c073-2cde-4cbc-8209-a2d788777d2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-r9cmb" Nov 02 06:39:21.875928 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:21.875893 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-r9cmb"] Nov 02 06:39:21.924445 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:21.924378 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d018c073-2cde-4cbc-8209-a2d788777d2a-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-r9cmb\" (UID: \"d018c073-2cde-4cbc-8209-a2d788777d2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-r9cmb" Nov 02 06:39:21.924445 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:21.924428 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spr4h\" (UniqueName: \"kubernetes.io/projected/d018c073-2cde-4cbc-8209-a2d788777d2a-kube-api-access-spr4h\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-r9cmb\" (UID: \"d018c073-2cde-4cbc-8209-a2d788777d2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-r9cmb" Nov 02 06:39:21.989422 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:21.989390 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-r9cmb" Nov 02 06:39:22.425074 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:22.425046 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-r9cmb"] Nov 02 06:39:22.425845 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:39:22.425819 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd018c073_2cde_4cbc_8209_a2d788777d2a.slice/crio-c24f9ff71ffb135ad9d598ba5e15d7e08ced816daefef7f7df59adedfa680e43 WatchSource:0}: Error finding container c24f9ff71ffb135ad9d598ba5e15d7e08ced816daefef7f7df59adedfa680e43: Status 404 returned error can't find the container with id c24f9ff71ffb135ad9d598ba5e15d7e08ced816daefef7f7df59adedfa680e43 Nov 02 06:39:22.428208 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:22.428175 5916 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 06:39:22.428317 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:22.428249 5916 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 06:39:22.428317 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:22.428292 5916 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 06:39:22.818164 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:22.818066 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-r9cmb" event={"ID":"d018c073-2cde-4cbc-8209-a2d788777d2a","Type":"ContainerStarted","Data":"e708b1b42904654b3c111a2692d66305deb1c742f3a17fa3dbd74235b3bd878a"} Nov 02 06:39:22.818164 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:22.818140 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-r9cmb" event={"ID":"d018c073-2cde-4cbc-8209-a2d788777d2a","Type":"ContainerStarted","Data":"c24f9ff71ffb135ad9d598ba5e15d7e08ced816daefef7f7df59adedfa680e43"} Nov 02 06:39:22.991170 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:22.991087 5916 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-r9cmb" Nov 02 06:39:22.994243 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:22.994192 5916 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-r9cmb" podStartSLOduration=1.994178658 podStartE2EDuration="1.994178658s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:39:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:39:22.991477186 +0000 UTC m=+1351.263297581" watchObservedRunningTime="2026-11-02 06:39:22.994178658 +0000 UTC m=+1351.265999050" Nov 02 06:39:23.992422 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:23.992394 5916 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-r9cmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Nov 02 06:39:23.992829 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:23.992442 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-r9cmb" podUID="d018c073-2cde-4cbc-8209-a2d788777d2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Nov 02 06:39:24.990937 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:24.990796 5916 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-r9cmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Nov 02 06:39:24.990937 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:24.990883 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-r9cmb" podUID="d018c073-2cde-4cbc-8209-a2d788777d2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Nov 02 06:39:25.990329 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:25.990293 5916 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-r9cmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Nov 02 06:39:25.990819 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:25.990381 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-r9cmb" podUID="d018c073-2cde-4cbc-8209-a2d788777d2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Nov 02 06:39:26.990048 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:26.990011 5916 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-r9cmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Nov 02 06:39:26.990259 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:26.990071 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-r9cmb" podUID="d018c073-2cde-4cbc-8209-a2d788777d2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Nov 02 06:39:27.990031 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:27.989999 5916 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-r9cmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Nov 02 06:39:27.990428 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:27.990064 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-r9cmb" podUID="d018c073-2cde-4cbc-8209-a2d788777d2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Nov 02 06:39:28.990352 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:28.990317 5916 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-r9cmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Nov 02 06:39:28.990770 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:28.990380 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-r9cmb" podUID="d018c073-2cde-4cbc-8209-a2d788777d2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Nov 02 06:39:29.989981 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:29.989946 5916 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-r9cmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Nov 02 06:39:29.990198 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:29.990006 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-r9cmb" podUID="d018c073-2cde-4cbc-8209-a2d788777d2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Nov 02 06:39:30.994997 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:30.994915 5916 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-r9cmb" Nov 02 06:39:30.995450 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:30.995429 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-r9cmb" Nov 02 06:39:30.996174 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:30.996151 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-r9cmb" Nov 02 06:39:38.571288 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:38.571241 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-15867003-dcp8l"] Nov 02 06:39:38.576415 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:38.576391 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-15867003-dcp8l" Nov 02 06:39:38.634431 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:38.634397 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-15867003-dcp8l"] Nov 02 06:39:38.643997 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:38.643964 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qc5xk\" (UniqueName: \"kubernetes.io/projected/2b8402d1-7cc6-4072-835e-5987d2dd2c50-kube-api-access-qc5xk\") pod \"maas-api-revoke-keys-e2e-ait-15867003-dcp8l\" (UID: \"2b8402d1-7cc6-4072-835e-5987d2dd2c50\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-15867003-dcp8l" Nov 02 06:39:38.644192 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:38.644170 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b8402d1-7cc6-4072-835e-5987d2dd2c50-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-15867003-dcp8l\" (UID: \"2b8402d1-7cc6-4072-835e-5987d2dd2c50\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-15867003-dcp8l" Nov 02 06:39:38.644256 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:38.644226 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b8402d1-7cc6-4072-835e-5987d2dd2c50-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-15867003-dcp8l\" (UID: \"2b8402d1-7cc6-4072-835e-5987d2dd2c50\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-15867003-dcp8l" Nov 02 06:39:38.676590 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:38.676559 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 06:39:38.676590 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:38.676581 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-68l4r\"" Nov 02 06:39:38.781929 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:38.781418 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qc5xk\" (UniqueName: \"kubernetes.io/projected/2b8402d1-7cc6-4072-835e-5987d2dd2c50-kube-api-access-qc5xk\") pod \"maas-api-revoke-keys-e2e-ait-15867003-dcp8l\" (UID: \"2b8402d1-7cc6-4072-835e-5987d2dd2c50\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-15867003-dcp8l" Nov 02 06:39:38.781929 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:38.781645 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b8402d1-7cc6-4072-835e-5987d2dd2c50-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-15867003-dcp8l\" (UID: \"2b8402d1-7cc6-4072-835e-5987d2dd2c50\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-15867003-dcp8l" Nov 02 06:39:38.781929 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:38.781702 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b8402d1-7cc6-4072-835e-5987d2dd2c50-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-15867003-dcp8l\" (UID: \"2b8402d1-7cc6-4072-835e-5987d2dd2c50\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-15867003-dcp8l" Nov 02 06:39:38.785929 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:38.782640 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b8402d1-7cc6-4072-835e-5987d2dd2c50-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-15867003-dcp8l\" (UID: \"2b8402d1-7cc6-4072-835e-5987d2dd2c50\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-15867003-dcp8l" Nov 02 06:39:38.795421 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:38.795386 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b8402d1-7cc6-4072-835e-5987d2dd2c50-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-15867003-dcp8l\" (UID: \"2b8402d1-7cc6-4072-835e-5987d2dd2c50\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-15867003-dcp8l" Nov 02 06:39:38.842439 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:38.842364 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qc5xk\" (UniqueName: \"kubernetes.io/projected/2b8402d1-7cc6-4072-835e-5987d2dd2c50-kube-api-access-qc5xk\") pod \"maas-api-revoke-keys-e2e-ait-15867003-dcp8l\" (UID: \"2b8402d1-7cc6-4072-835e-5987d2dd2c50\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-15867003-dcp8l" Nov 02 06:39:38.893139 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:38.893092 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-15867003-dcp8l" Nov 02 06:39:39.299721 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:39:39.299692 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b8402d1_7cc6_4072_835e_5987d2dd2c50.slice/crio-426ce991279800c61093a8501719cfa1f79257f571db55e67bb5262459f85507 WatchSource:0}: Error finding container 426ce991279800c61093a8501719cfa1f79257f571db55e67bb5262459f85507: Status 404 returned error can't find the container with id 426ce991279800c61093a8501719cfa1f79257f571db55e67bb5262459f85507 Nov 02 06:39:39.301161 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:39.301133 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-15867003-dcp8l"] Nov 02 06:39:40.042896 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:40.042797 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-15867003-dcp8l" event={"ID":"2b8402d1-7cc6-4072-835e-5987d2dd2c50","Type":"ContainerStarted","Data":"d76dedb075972834d8183084e1734fb0d70e7ae3421839837177d51653bb2112"} Nov 02 06:39:40.042896 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:40.042868 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-15867003-dcp8l" event={"ID":"2b8402d1-7cc6-4072-835e-5987d2dd2c50","Type":"ContainerStarted","Data":"426ce991279800c61093a8501719cfa1f79257f571db55e67bb5262459f85507"} Nov 02 06:39:40.159796 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:40.159736 5916 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-15867003-dcp8l" podStartSLOduration=2.159718836 podStartE2EDuration="2.159718836s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:39:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:39:40.158062176 +0000 UTC m=+1368.429882571" watchObservedRunningTime="2026-11-02 06:39:40.159718836 +0000 UTC m=+1368.431539232" Nov 02 06:39:41.053812 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:41.053774 5916 generic.go:356] "Generic (PLEG): container finished" podID="2b8402d1-7cc6-4072-835e-5987d2dd2c50" containerID="d76dedb075972834d8183084e1734fb0d70e7ae3421839837177d51653bb2112" exitCode=0 Nov 02 06:39:41.054229 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:41.053842 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-15867003-dcp8l" event={"ID":"2b8402d1-7cc6-4072-835e-5987d2dd2c50","Type":"ContainerDied","Data":"d76dedb075972834d8183084e1734fb0d70e7ae3421839837177d51653bb2112"} Nov 02 06:39:42.238227 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:42.238194 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-15867003-dcp8l" Nov 02 06:39:42.342094 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:42.342019 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b8402d1-7cc6-4072-835e-5987d2dd2c50-kube-api-access-qc5xk\" (UniqueName: \"kubernetes.io/projected/2b8402d1-7cc6-4072-835e-5987d2dd2c50-kube-api-access-qc5xk\") pod \"2b8402d1-7cc6-4072-835e-5987d2dd2c50\" (UID: \"2b8402d1-7cc6-4072-835e-5987d2dd2c50\") " Nov 02 06:39:42.342094 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:42.342087 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b8402d1-7cc6-4072-835e-5987d2dd2c50-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b8402d1-7cc6-4072-835e-5987d2dd2c50-maas-api-xks-ca\") pod \"2b8402d1-7cc6-4072-835e-5987d2dd2c50\" (UID: \"2b8402d1-7cc6-4072-835e-5987d2dd2c50\") " Nov 02 06:39:42.342275 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:42.342259 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b8402d1-7cc6-4072-835e-5987d2dd2c50-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b8402d1-7cc6-4072-835e-5987d2dd2c50-maas-api-openshift-ca\") pod \"2b8402d1-7cc6-4072-835e-5987d2dd2c50\" (UID: \"2b8402d1-7cc6-4072-835e-5987d2dd2c50\") " Nov 02 06:39:42.342651 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:42.342628 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b8402d1-7cc6-4072-835e-5987d2dd2c50-maas-api-openshift-ca" pod "2b8402d1-7cc6-4072-835e-5987d2dd2c50" (UID: "2b8402d1-7cc6-4072-835e-5987d2dd2c50"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:39:42.342863 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:42.342846 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b8402d1-7cc6-4072-835e-5987d2dd2c50-maas-api-openshift-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:39:42.345758 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:42.345729 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b8402d1-7cc6-4072-835e-5987d2dd2c50-kube-api-access-qc5xk" pod "2b8402d1-7cc6-4072-835e-5987d2dd2c50" (UID: "2b8402d1-7cc6-4072-835e-5987d2dd2c50"). InnerVolumeSpecName "kube-api-access-qc5xk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:39:42.348381 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:42.348333 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b8402d1-7cc6-4072-835e-5987d2dd2c50-maas-api-xks-ca" pod "2b8402d1-7cc6-4072-835e-5987d2dd2c50" (UID: "2b8402d1-7cc6-4072-835e-5987d2dd2c50"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:39:42.444249 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:42.444219 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qc5xk\" (UniqueName: \"kubernetes.io/projected/2b8402d1-7cc6-4072-835e-5987d2dd2c50-kube-api-access-qc5xk\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:39:42.444249 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:42.444246 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b8402d1-7cc6-4072-835e-5987d2dd2c50-maas-api-xks-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:39:43.075816 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:43.075784 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-15867003-dcp8l" Nov 02 06:39:43.075816 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:43.075810 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-15867003-dcp8l" event={"ID":"2b8402d1-7cc6-4072-835e-5987d2dd2c50","Type":"ContainerDied","Data":"426ce991279800c61093a8501719cfa1f79257f571db55e67bb5262459f85507"} Nov 02 06:39:43.076106 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:43.075850 5916 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="426ce991279800c61093a8501719cfa1f79257f571db55e67bb5262459f85507" Nov 02 06:39:44.687750 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:44.687715 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d690ed79-867lw"] Nov 02 06:39:44.688454 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:44.688430 5916 cpu_manager.go:414] "containerMap: removing container" podUID="2b8402d1-7cc6-4072-835e-5987d2dd2c50" containerName="revoke-keys" Nov 02 06:39:44.688454 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:44.688453 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b8402d1-7cc6-4072-835e-5987d2dd2c50" containerName="revoke-keys" Nov 02 06:39:44.688702 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:44.688629 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b8402d1-7cc6-4072-835e-5987d2dd2c50" containerName="revoke-keys" Nov 02 06:39:44.692347 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:44.692328 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d690ed79-867lw" Nov 02 06:39:44.750627 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:44.750276 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d690ed79-867lw"] Nov 02 06:39:44.791390 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:44.791358 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92698bcd-5f10-4723-882e-e2be400e6259-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-d690ed79-867lw\" (UID: \"92698bcd-5f10-4723-882e-e2be400e6259\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d690ed79-867lw" Nov 02 06:39:44.791583 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:44.791420 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92698bcd-5f10-4723-882e-e2be400e6259-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-d690ed79-867lw\" (UID: \"92698bcd-5f10-4723-882e-e2be400e6259\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d690ed79-867lw" Nov 02 06:39:44.791646 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:44.791579 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpfcz\" (UniqueName: \"kubernetes.io/projected/92698bcd-5f10-4723-882e-e2be400e6259-kube-api-access-mpfcz\") pod \"maas-api-revoke-keys-e2e-ait-d690ed79-867lw\" (UID: \"92698bcd-5f10-4723-882e-e2be400e6259\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d690ed79-867lw" Nov 02 06:39:44.793838 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:44.793813 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 06:39:44.793981 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:44.793843 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-68l4r\"" Nov 02 06:39:44.893408 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:44.893373 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92698bcd-5f10-4723-882e-e2be400e6259-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-d690ed79-867lw\" (UID: \"92698bcd-5f10-4723-882e-e2be400e6259\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d690ed79-867lw" Nov 02 06:39:44.893599 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:44.893428 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92698bcd-5f10-4723-882e-e2be400e6259-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-d690ed79-867lw\" (UID: \"92698bcd-5f10-4723-882e-e2be400e6259\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d690ed79-867lw" Nov 02 06:39:44.893599 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:44.893482 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpfcz\" (UniqueName: \"kubernetes.io/projected/92698bcd-5f10-4723-882e-e2be400e6259-kube-api-access-mpfcz\") pod \"maas-api-revoke-keys-e2e-ait-d690ed79-867lw\" (UID: \"92698bcd-5f10-4723-882e-e2be400e6259\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d690ed79-867lw" Nov 02 06:39:44.894257 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:44.894229 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92698bcd-5f10-4723-882e-e2be400e6259-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-d690ed79-867lw\" (UID: \"92698bcd-5f10-4723-882e-e2be400e6259\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d690ed79-867lw" Nov 02 06:39:44.895853 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:44.895834 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92698bcd-5f10-4723-882e-e2be400e6259-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-d690ed79-867lw\" (UID: \"92698bcd-5f10-4723-882e-e2be400e6259\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d690ed79-867lw" Nov 02 06:39:44.950978 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:44.950896 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpfcz\" (UniqueName: \"kubernetes.io/projected/92698bcd-5f10-4723-882e-e2be400e6259-kube-api-access-mpfcz\") pod \"maas-api-revoke-keys-e2e-ait-d690ed79-867lw\" (UID: \"92698bcd-5f10-4723-882e-e2be400e6259\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d690ed79-867lw" Nov 02 06:39:45.008584 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:45.008557 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d690ed79-867lw" Nov 02 06:39:45.398022 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:39:45.397988 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92698bcd_5f10_4723_882e_e2be400e6259.slice/crio-5ea67b3b18acbec57ab6fdde80a8dc34e4364845efcbd10a1e07318423e664f9 WatchSource:0}: Error finding container 5ea67b3b18acbec57ab6fdde80a8dc34e4364845efcbd10a1e07318423e664f9: Status 404 returned error can't find the container with id 5ea67b3b18acbec57ab6fdde80a8dc34e4364845efcbd10a1e07318423e664f9 Nov 02 06:39:45.399949 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:45.399921 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d690ed79-867lw"] Nov 02 06:39:46.109510 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:46.109407 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d690ed79-867lw" event={"ID":"92698bcd-5f10-4723-882e-e2be400e6259","Type":"ContainerStarted","Data":"cffe55ce0d9e19e10f54d01ecced5cc89a635087d64a8a59fa5a98ba11445bc8"} Nov 02 06:39:46.109510 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:46.109463 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d690ed79-867lw" event={"ID":"92698bcd-5f10-4723-882e-e2be400e6259","Type":"ContainerStarted","Data":"5ea67b3b18acbec57ab6fdde80a8dc34e4364845efcbd10a1e07318423e664f9"} Nov 02 06:39:46.237227 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:46.237155 5916 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d690ed79-867lw" podStartSLOduration=2.237135679 podStartE2EDuration="2.237135679s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:39:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:39:46.230618447 +0000 UTC m=+1374.502438863" watchObservedRunningTime="2026-11-02 06:39:46.237135679 +0000 UTC m=+1374.508956078" Nov 02 06:39:47.119599 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:47.119565 5916 generic.go:356] "Generic (PLEG): container finished" podID="92698bcd-5f10-4723-882e-e2be400e6259" containerID="cffe55ce0d9e19e10f54d01ecced5cc89a635087d64a8a59fa5a98ba11445bc8" exitCode=0 Nov 02 06:39:47.120020 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:47.119647 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d690ed79-867lw" event={"ID":"92698bcd-5f10-4723-882e-e2be400e6259","Type":"ContainerDied","Data":"cffe55ce0d9e19e10f54d01ecced5cc89a635087d64a8a59fa5a98ba11445bc8"} Nov 02 06:39:48.484479 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:48.484440 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d690ed79-867lw" Nov 02 06:39:48.525633 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:48.525598 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/92698bcd-5f10-4723-882e-e2be400e6259-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92698bcd-5f10-4723-882e-e2be400e6259-maas-api-xks-ca\") pod \"92698bcd-5f10-4723-882e-e2be400e6259\" (UID: \"92698bcd-5f10-4723-882e-e2be400e6259\") " Nov 02 06:39:48.525834 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:48.525769 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92698bcd-5f10-4723-882e-e2be400e6259-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92698bcd-5f10-4723-882e-e2be400e6259-maas-api-openshift-ca\") pod \"92698bcd-5f10-4723-882e-e2be400e6259\" (UID: \"92698bcd-5f10-4723-882e-e2be400e6259\") " Nov 02 06:39:48.525916 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:48.525889 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92698bcd-5f10-4723-882e-e2be400e6259-kube-api-access-mpfcz\" (UniqueName: \"kubernetes.io/projected/92698bcd-5f10-4723-882e-e2be400e6259-kube-api-access-mpfcz\") pod \"92698bcd-5f10-4723-882e-e2be400e6259\" (UID: \"92698bcd-5f10-4723-882e-e2be400e6259\") " Nov 02 06:39:48.526663 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:48.526620 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92698bcd-5f10-4723-882e-e2be400e6259-maas-api-openshift-ca" pod "92698bcd-5f10-4723-882e-e2be400e6259" (UID: "92698bcd-5f10-4723-882e-e2be400e6259"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:39:48.530975 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:48.528796 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92698bcd-5f10-4723-882e-e2be400e6259-kube-api-access-mpfcz" pod "92698bcd-5f10-4723-882e-e2be400e6259" (UID: "92698bcd-5f10-4723-882e-e2be400e6259"). InnerVolumeSpecName "kube-api-access-mpfcz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:39:48.536159 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:48.536115 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92698bcd-5f10-4723-882e-e2be400e6259-maas-api-xks-ca" pod "92698bcd-5f10-4723-882e-e2be400e6259" (UID: "92698bcd-5f10-4723-882e-e2be400e6259"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:39:48.627747 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:48.627710 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92698bcd-5f10-4723-882e-e2be400e6259-maas-api-xks-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:39:48.627747 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:48.627750 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92698bcd-5f10-4723-882e-e2be400e6259-maas-api-openshift-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:39:48.627985 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:48.627765 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mpfcz\" (UniqueName: \"kubernetes.io/projected/92698bcd-5f10-4723-882e-e2be400e6259-kube-api-access-mpfcz\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:39:49.152928 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:49.152509 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d690ed79-867lw" event={"ID":"92698bcd-5f10-4723-882e-e2be400e6259","Type":"ContainerDied","Data":"5ea67b3b18acbec57ab6fdde80a8dc34e4364845efcbd10a1e07318423e664f9"} Nov 02 06:39:49.152928 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:49.152572 5916 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5ea67b3b18acbec57ab6fdde80a8dc34e4364845efcbd10a1e07318423e664f9" Nov 02 06:39:49.152928 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:49.152651 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d690ed79-867lw" Nov 02 06:39:49.894812 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:39:49.894781 5916 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p98cq" containerName="extractor" Nov 02 06:39:52.247190 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:52.247152 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-14448910-8ggrp"] Nov 02 06:39:52.247807 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:52.247768 5916 cpu_manager.go:414] "containerMap: removing container" podUID="92698bcd-5f10-4723-882e-e2be400e6259" containerName="revoke-keys" Nov 02 06:39:52.247807 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:52.247780 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92698bcd-5f10-4723-882e-e2be400e6259" containerName="revoke-keys" Nov 02 06:39:52.247959 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:52.247922 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="92698bcd-5f10-4723-882e-e2be400e6259" containerName="revoke-keys" Nov 02 06:39:52.251876 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:52.251854 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-14448910-8ggrp" Nov 02 06:39:52.291743 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:52.291715 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ce243df-455d-4797-8492-3b3e403181e9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-14448910-8ggrp\" (UID: \"3ce243df-455d-4797-8492-3b3e403181e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-14448910-8ggrp" Nov 02 06:39:52.291942 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:52.291913 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffwc7\" (UniqueName: \"kubernetes.io/projected/3ce243df-455d-4797-8492-3b3e403181e9-kube-api-access-ffwc7\") pod \"maas-api-revoke-keys-e2e-derive-14448910-8ggrp\" (UID: \"3ce243df-455d-4797-8492-3b3e403181e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-14448910-8ggrp" Nov 02 06:39:52.292081 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:52.292067 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ce243df-455d-4797-8492-3b3e403181e9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-14448910-8ggrp\" (UID: \"3ce243df-455d-4797-8492-3b3e403181e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-14448910-8ggrp" Nov 02 06:39:52.315957 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:52.305931 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-14448910-8ggrp"] Nov 02 06:39:52.350080 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:52.350036 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 06:39:52.350080 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:52.350071 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-68l4r\"" Nov 02 06:39:52.393570 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:52.393527 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ce243df-455d-4797-8492-3b3e403181e9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-14448910-8ggrp\" (UID: \"3ce243df-455d-4797-8492-3b3e403181e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-14448910-8ggrp" Nov 02 06:39:52.393737 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:52.393663 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ce243df-455d-4797-8492-3b3e403181e9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-14448910-8ggrp\" (UID: \"3ce243df-455d-4797-8492-3b3e403181e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-14448910-8ggrp" Nov 02 06:39:52.393831 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:52.393814 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffwc7\" (UniqueName: \"kubernetes.io/projected/3ce243df-455d-4797-8492-3b3e403181e9-kube-api-access-ffwc7\") pod \"maas-api-revoke-keys-e2e-derive-14448910-8ggrp\" (UID: \"3ce243df-455d-4797-8492-3b3e403181e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-14448910-8ggrp" Nov 02 06:39:52.394293 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:52.394251 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ce243df-455d-4797-8492-3b3e403181e9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-14448910-8ggrp\" (UID: \"3ce243df-455d-4797-8492-3b3e403181e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-14448910-8ggrp" Nov 02 06:39:52.396597 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:52.396574 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ce243df-455d-4797-8492-3b3e403181e9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-14448910-8ggrp\" (UID: \"3ce243df-455d-4797-8492-3b3e403181e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-14448910-8ggrp" Nov 02 06:39:52.451655 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:52.451621 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffwc7\" (UniqueName: \"kubernetes.io/projected/3ce243df-455d-4797-8492-3b3e403181e9-kube-api-access-ffwc7\") pod \"maas-api-revoke-keys-e2e-derive-14448910-8ggrp\" (UID: \"3ce243df-455d-4797-8492-3b3e403181e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-14448910-8ggrp" Nov 02 06:39:52.565684 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:52.565601 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-14448910-8ggrp" Nov 02 06:39:53.002148 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:53.002116 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-14448910-8ggrp"] Nov 02 06:39:53.003330 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:39:53.003306 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ce243df_455d_4797_8492_3b3e403181e9.slice/crio-eae52c813f7f965fbc14133c8f3d4a948c22bec47465aba9e57e528963282e0b WatchSource:0}: Error finding container eae52c813f7f965fbc14133c8f3d4a948c22bec47465aba9e57e528963282e0b: Status 404 returned error can't find the container with id eae52c813f7f965fbc14133c8f3d4a948c22bec47465aba9e57e528963282e0b Nov 02 06:39:53.196544 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:53.196508 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-14448910-8ggrp" event={"ID":"3ce243df-455d-4797-8492-3b3e403181e9","Type":"ContainerStarted","Data":"eae52c813f7f965fbc14133c8f3d4a948c22bec47465aba9e57e528963282e0b"} Nov 02 06:39:53.596403 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:53.596314 5916 scope.go:122] "RemoveContainer" containerID="19a1bbf8ac6189361105f3753094e71be952e9093ea39e26b920272cb8f5917b" Nov 02 06:39:54.210804 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:54.210707 5916 generic.go:356] "Generic (PLEG): container finished" podID="3ce243df-455d-4797-8492-3b3e403181e9" containerID="67164a0c533d5a518b740df7d0112244aa85f7c16aa52e3d01808def74144292" exitCode=0 Nov 02 06:39:54.211403 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:54.211365 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-14448910-8ggrp" event={"ID":"3ce243df-455d-4797-8492-3b3e403181e9","Type":"ContainerDied","Data":"67164a0c533d5a518b740df7d0112244aa85f7c16aa52e3d01808def74144292"} Nov 02 06:39:55.400347 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:55.400320 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-14448910-8ggrp" Nov 02 06:39:55.476189 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:55.476104 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ce243df-455d-4797-8492-3b3e403181e9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ce243df-455d-4797-8492-3b3e403181e9-maas-api-xks-ca\") pod \"3ce243df-455d-4797-8492-3b3e403181e9\" (UID: \"3ce243df-455d-4797-8492-3b3e403181e9\") " Nov 02 06:39:55.476347 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:55.476242 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ce243df-455d-4797-8492-3b3e403181e9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ce243df-455d-4797-8492-3b3e403181e9-maas-api-openshift-ca\") pod \"3ce243df-455d-4797-8492-3b3e403181e9\" (UID: \"3ce243df-455d-4797-8492-3b3e403181e9\") " Nov 02 06:39:55.476405 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:55.476382 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ce243df-455d-4797-8492-3b3e403181e9-kube-api-access-ffwc7\" (UniqueName: \"kubernetes.io/projected/3ce243df-455d-4797-8492-3b3e403181e9-kube-api-access-ffwc7\") pod \"3ce243df-455d-4797-8492-3b3e403181e9\" (UID: \"3ce243df-455d-4797-8492-3b3e403181e9\") " Nov 02 06:39:55.476604 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:55.476570 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ce243df-455d-4797-8492-3b3e403181e9-maas-api-openshift-ca" pod "3ce243df-455d-4797-8492-3b3e403181e9" (UID: "3ce243df-455d-4797-8492-3b3e403181e9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:39:55.477028 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:55.477007 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ce243df-455d-4797-8492-3b3e403181e9-maas-api-openshift-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:39:55.478475 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:55.478451 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ce243df-455d-4797-8492-3b3e403181e9-kube-api-access-ffwc7" pod "3ce243df-455d-4797-8492-3b3e403181e9" (UID: "3ce243df-455d-4797-8492-3b3e403181e9"). InnerVolumeSpecName "kube-api-access-ffwc7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:39:55.478708 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:55.478690 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ce243df-455d-4797-8492-3b3e403181e9-maas-api-xks-ca" pod "3ce243df-455d-4797-8492-3b3e403181e9" (UID: "3ce243df-455d-4797-8492-3b3e403181e9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:39:55.578463 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:55.578422 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ffwc7\" (UniqueName: \"kubernetes.io/projected/3ce243df-455d-4797-8492-3b3e403181e9-kube-api-access-ffwc7\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:39:55.578463 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:55.578459 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ce243df-455d-4797-8492-3b3e403181e9-maas-api-xks-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:39:55.889019 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:55.888989 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24af19b3-lp25w"] Nov 02 06:39:55.889601 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:55.889589 5916 cpu_manager.go:414] "containerMap: removing container" podUID="3ce243df-455d-4797-8492-3b3e403181e9" containerName="revoke-keys" Nov 02 06:39:55.889643 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:55.889603 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ce243df-455d-4797-8492-3b3e403181e9" containerName="revoke-keys" Nov 02 06:39:55.889714 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:55.889705 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ce243df-455d-4797-8492-3b3e403181e9" containerName="revoke-keys" Nov 02 06:39:55.893492 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:55.893474 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24af19b3-lp25w" Nov 02 06:39:55.950711 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:55.950685 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24af19b3-lp25w"] Nov 02 06:39:55.982949 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:55.982893 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2816f619-278a-4693-a977-1261c31ea5a4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-24af19b3-lp25w\" (UID: \"2816f619-278a-4693-a977-1261c31ea5a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24af19b3-lp25w" Nov 02 06:39:55.983147 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:55.983120 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k57n8\" (UniqueName: \"kubernetes.io/projected/2816f619-278a-4693-a977-1261c31ea5a4-kube-api-access-k57n8\") pod \"maas-api-revoke-sub-24af19b3-lp25w\" (UID: \"2816f619-278a-4693-a977-1261c31ea5a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24af19b3-lp25w" Nov 02 06:39:55.983314 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:55.983296 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2816f619-278a-4693-a977-1261c31ea5a4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-24af19b3-lp25w\" (UID: \"2816f619-278a-4693-a977-1261c31ea5a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24af19b3-lp25w" Nov 02 06:39:56.084870 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:56.084840 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2816f619-278a-4693-a977-1261c31ea5a4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-24af19b3-lp25w\" (UID: \"2816f619-278a-4693-a977-1261c31ea5a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24af19b3-lp25w" Nov 02 06:39:56.085045 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:56.084928 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k57n8\" (UniqueName: \"kubernetes.io/projected/2816f619-278a-4693-a977-1261c31ea5a4-kube-api-access-k57n8\") pod \"maas-api-revoke-sub-24af19b3-lp25w\" (UID: \"2816f619-278a-4693-a977-1261c31ea5a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24af19b3-lp25w" Nov 02 06:39:56.085045 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:56.085002 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2816f619-278a-4693-a977-1261c31ea5a4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-24af19b3-lp25w\" (UID: \"2816f619-278a-4693-a977-1261c31ea5a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24af19b3-lp25w" Nov 02 06:39:56.085624 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:56.085603 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2816f619-278a-4693-a977-1261c31ea5a4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-24af19b3-lp25w\" (UID: \"2816f619-278a-4693-a977-1261c31ea5a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24af19b3-lp25w" Nov 02 06:39:56.087199 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:56.087178 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2816f619-278a-4693-a977-1261c31ea5a4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-24af19b3-lp25w\" (UID: \"2816f619-278a-4693-a977-1261c31ea5a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24af19b3-lp25w" Nov 02 06:39:56.142876 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:56.142852 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k57n8\" (UniqueName: \"kubernetes.io/projected/2816f619-278a-4693-a977-1261c31ea5a4-kube-api-access-k57n8\") pod \"maas-api-revoke-sub-24af19b3-lp25w\" (UID: \"2816f619-278a-4693-a977-1261c31ea5a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24af19b3-lp25w" Nov 02 06:39:56.207100 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:56.207080 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24af19b3-lp25w" Nov 02 06:39:56.231211 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:56.231180 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-14448910-8ggrp" Nov 02 06:39:56.231307 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:56.231175 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-14448910-8ggrp" event={"ID":"3ce243df-455d-4797-8492-3b3e403181e9","Type":"ContainerDied","Data":"eae52c813f7f965fbc14133c8f3d4a948c22bec47465aba9e57e528963282e0b"} Nov 02 06:39:56.231358 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:56.231319 5916 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eae52c813f7f965fbc14133c8f3d4a948c22bec47465aba9e57e528963282e0b" Nov 02 06:39:56.593038 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:39:56.593010 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2816f619_278a_4693_a977_1261c31ea5a4.slice/crio-17ec7c9beb2591ac683b954bbc3e53fd134919c8d1cc28ceeddabc96548e8da5 WatchSource:0}: Error finding container 17ec7c9beb2591ac683b954bbc3e53fd134919c8d1cc28ceeddabc96548e8da5: Status 404 returned error can't find the container with id 17ec7c9beb2591ac683b954bbc3e53fd134919c8d1cc28ceeddabc96548e8da5 Nov 02 06:39:56.593690 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:56.593661 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24af19b3-lp25w"] Nov 02 06:39:57.244245 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:57.244197 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24af19b3-lp25w" event={"ID":"2816f619-278a-4693-a977-1261c31ea5a4","Type":"ContainerStarted","Data":"2813e871dfb65bea620ed582fdb67b92a5e845d0da5db47b6770fe6763ef68f3"} Nov 02 06:39:57.244464 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:57.244296 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24af19b3-lp25w" event={"ID":"2816f619-278a-4693-a977-1261c31ea5a4","Type":"ContainerStarted","Data":"17ec7c9beb2591ac683b954bbc3e53fd134919c8d1cc28ceeddabc96548e8da5"} Nov 02 06:39:57.360989 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:57.360923 5916 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24af19b3-lp25w" podStartSLOduration=2.360896261 podStartE2EDuration="2.360896261s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:39:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:39:57.357047915 +0000 UTC m=+1385.628868313" watchObservedRunningTime="2026-11-02 06:39:57.360896261 +0000 UTC m=+1385.632716679" Nov 02 06:39:57.834085 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:57.834053 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5eed1bfb-wdr5w"] Nov 02 06:39:57.838802 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:57.838765 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5eed1bfb-wdr5w" Nov 02 06:39:57.895683 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:57.895656 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5eed1bfb-wdr5w"] Nov 02 06:39:58.016383 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:58.016289 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44b57332-91fe-483c-806f-f6576f18e461-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-5eed1bfb-wdr5w\" (UID: \"44b57332-91fe-483c-806f-f6576f18e461\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5eed1bfb-wdr5w" Nov 02 06:39:58.016383 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:58.016355 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ckwn\" (UniqueName: \"kubernetes.io/projected/44b57332-91fe-483c-806f-f6576f18e461-kube-api-access-7ckwn\") pod \"maas-api-revoke-keys-e2e-migrate-5eed1bfb-wdr5w\" (UID: \"44b57332-91fe-483c-806f-f6576f18e461\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5eed1bfb-wdr5w" Nov 02 06:39:58.016583 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:58.016504 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44b57332-91fe-483c-806f-f6576f18e461-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-5eed1bfb-wdr5w\" (UID: \"44b57332-91fe-483c-806f-f6576f18e461\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5eed1bfb-wdr5w" Nov 02 06:39:58.118214 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:58.118179 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44b57332-91fe-483c-806f-f6576f18e461-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-5eed1bfb-wdr5w\" (UID: \"44b57332-91fe-483c-806f-f6576f18e461\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5eed1bfb-wdr5w" Nov 02 06:39:58.118393 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:58.118331 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44b57332-91fe-483c-806f-f6576f18e461-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-5eed1bfb-wdr5w\" (UID: \"44b57332-91fe-483c-806f-f6576f18e461\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5eed1bfb-wdr5w" Nov 02 06:39:58.118393 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:58.118360 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ckwn\" (UniqueName: \"kubernetes.io/projected/44b57332-91fe-483c-806f-f6576f18e461-kube-api-access-7ckwn\") pod \"maas-api-revoke-keys-e2e-migrate-5eed1bfb-wdr5w\" (UID: \"44b57332-91fe-483c-806f-f6576f18e461\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5eed1bfb-wdr5w" Nov 02 06:39:58.118848 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:58.118826 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44b57332-91fe-483c-806f-f6576f18e461-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-5eed1bfb-wdr5w\" (UID: \"44b57332-91fe-483c-806f-f6576f18e461\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5eed1bfb-wdr5w" Nov 02 06:39:58.120654 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:58.120631 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44b57332-91fe-483c-806f-f6576f18e461-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-5eed1bfb-wdr5w\" (UID: \"44b57332-91fe-483c-806f-f6576f18e461\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5eed1bfb-wdr5w" Nov 02 06:39:58.176862 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:58.176835 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ckwn\" (UniqueName: \"kubernetes.io/projected/44b57332-91fe-483c-806f-f6576f18e461-kube-api-access-7ckwn\") pod \"maas-api-revoke-keys-e2e-migrate-5eed1bfb-wdr5w\" (UID: \"44b57332-91fe-483c-806f-f6576f18e461\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5eed1bfb-wdr5w" Nov 02 06:39:58.259596 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:58.259563 5916 generic.go:356] "Generic (PLEG): container finished" podID="2816f619-278a-4693-a977-1261c31ea5a4" containerID="2813e871dfb65bea620ed582fdb67b92a5e845d0da5db47b6770fe6763ef68f3" exitCode=0 Nov 02 06:39:58.259797 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:58.259648 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24af19b3-lp25w" event={"ID":"2816f619-278a-4693-a977-1261c31ea5a4","Type":"ContainerDied","Data":"2813e871dfb65bea620ed582fdb67b92a5e845d0da5db47b6770fe6763ef68f3"} Nov 02 06:39:58.457432 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:58.456064 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5eed1bfb-wdr5w" Nov 02 06:39:58.914357 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:39:58.914328 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44b57332_91fe_483c_806f_f6576f18e461.slice/crio-452eb493ebdbcdc31e59c9e2c224faca0a3403be71ff422f34f8edb874451afb WatchSource:0}: Error finding container 452eb493ebdbcdc31e59c9e2c224faca0a3403be71ff422f34f8edb874451afb: Status 404 returned error can't find the container with id 452eb493ebdbcdc31e59c9e2c224faca0a3403be71ff422f34f8edb874451afb Nov 02 06:39:58.914818 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:58.914793 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5eed1bfb-wdr5w"] Nov 02 06:39:59.271593 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:59.271559 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5eed1bfb-wdr5w" event={"ID":"44b57332-91fe-483c-806f-f6576f18e461","Type":"ContainerStarted","Data":"f4fd08872d3f9a860ecbd8cfe89ae32d70f6bda5f5d9990ff5ef4a008f698b70"} Nov 02 06:39:59.271736 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:59.271622 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5eed1bfb-wdr5w" event={"ID":"44b57332-91fe-483c-806f-f6576f18e461","Type":"ContainerStarted","Data":"452eb493ebdbcdc31e59c9e2c224faca0a3403be71ff422f34f8edb874451afb"} Nov 02 06:39:59.439613 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:59.439588 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24af19b3-lp25w" Nov 02 06:39:59.500966 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:59.500869 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2816f619-278a-4693-a977-1261c31ea5a4-kube-api-access-k57n8\" (UniqueName: \"kubernetes.io/projected/2816f619-278a-4693-a977-1261c31ea5a4-kube-api-access-k57n8\") pod \"2816f619-278a-4693-a977-1261c31ea5a4\" (UID: \"2816f619-278a-4693-a977-1261c31ea5a4\") " Nov 02 06:39:59.501086 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:59.501050 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2816f619-278a-4693-a977-1261c31ea5a4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2816f619-278a-4693-a977-1261c31ea5a4-maas-api-xks-ca\") pod \"2816f619-278a-4693-a977-1261c31ea5a4\" (UID: \"2816f619-278a-4693-a977-1261c31ea5a4\") " Nov 02 06:39:59.501132 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:59.501086 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2816f619-278a-4693-a977-1261c31ea5a4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2816f619-278a-4693-a977-1261c31ea5a4-maas-api-openshift-ca\") pod \"2816f619-278a-4693-a977-1261c31ea5a4\" (UID: \"2816f619-278a-4693-a977-1261c31ea5a4\") " Nov 02 06:39:59.501531 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:59.501504 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2816f619-278a-4693-a977-1261c31ea5a4-maas-api-openshift-ca" pod "2816f619-278a-4693-a977-1261c31ea5a4" (UID: "2816f619-278a-4693-a977-1261c31ea5a4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:39:59.501748 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:59.501732 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2816f619-278a-4693-a977-1261c31ea5a4-maas-api-openshift-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:39:59.502940 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:59.502916 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2816f619-278a-4693-a977-1261c31ea5a4-maas-api-xks-ca" pod "2816f619-278a-4693-a977-1261c31ea5a4" (UID: "2816f619-278a-4693-a977-1261c31ea5a4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:39:59.503029 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:59.502992 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2816f619-278a-4693-a977-1261c31ea5a4-kube-api-access-k57n8" pod "2816f619-278a-4693-a977-1261c31ea5a4" (UID: "2816f619-278a-4693-a977-1261c31ea5a4"). InnerVolumeSpecName "kube-api-access-k57n8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:39:59.572539 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:59.572476 5916 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5eed1bfb-wdr5w" podStartSLOduration=2.572456049 podStartE2EDuration="2.572456049s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:39:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:39:59.387927173 +0000 UTC m=+1387.659747592" watchObservedRunningTime="2026-11-02 06:39:59.572456049 +0000 UTC m=+1387.844276444" Nov 02 06:39:59.602798 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:59.602765 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k57n8\" (UniqueName: \"kubernetes.io/projected/2816f619-278a-4693-a977-1261c31ea5a4-kube-api-access-k57n8\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:39:59.602798 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:39:59.602797 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2816f619-278a-4693-a977-1261c31ea5a4-maas-api-xks-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:40:00.283784 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:40:00.283754 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24af19b3-lp25w" Nov 02 06:40:00.284325 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:40:00.283780 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24af19b3-lp25w" event={"ID":"2816f619-278a-4693-a977-1261c31ea5a4","Type":"ContainerDied","Data":"17ec7c9beb2591ac683b954bbc3e53fd134919c8d1cc28ceeddabc96548e8da5"} Nov 02 06:40:00.284325 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:40:00.283839 5916 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="17ec7c9beb2591ac683b954bbc3e53fd134919c8d1cc28ceeddabc96548e8da5" Nov 02 06:40:00.285882 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:40:00.285855 5916 generic.go:356] "Generic (PLEG): container finished" podID="44b57332-91fe-483c-806f-f6576f18e461" containerID="f4fd08872d3f9a860ecbd8cfe89ae32d70f6bda5f5d9990ff5ef4a008f698b70" exitCode=0 Nov 02 06:40:00.286068 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:40:00.285944 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5eed1bfb-wdr5w" event={"ID":"44b57332-91fe-483c-806f-f6576f18e461","Type":"ContainerDied","Data":"f4fd08872d3f9a860ecbd8cfe89ae32d70f6bda5f5d9990ff5ef4a008f698b70"} Nov 02 06:40:01.479122 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:40:01.479094 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5eed1bfb-wdr5w" Nov 02 06:40:01.528203 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:40:01.528175 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44b57332-91fe-483c-806f-f6576f18e461-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44b57332-91fe-483c-806f-f6576f18e461-maas-api-openshift-ca\") pod \"44b57332-91fe-483c-806f-f6576f18e461\" (UID: \"44b57332-91fe-483c-806f-f6576f18e461\") " Nov 02 06:40:01.528369 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:40:01.528354 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/44b57332-91fe-483c-806f-f6576f18e461-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44b57332-91fe-483c-806f-f6576f18e461-maas-api-xks-ca\") pod \"44b57332-91fe-483c-806f-f6576f18e461\" (UID: \"44b57332-91fe-483c-806f-f6576f18e461\") " Nov 02 06:40:01.528443 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:40:01.528432 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44b57332-91fe-483c-806f-f6576f18e461-kube-api-access-7ckwn\" (UniqueName: \"kubernetes.io/projected/44b57332-91fe-483c-806f-f6576f18e461-kube-api-access-7ckwn\") pod \"44b57332-91fe-483c-806f-f6576f18e461\" (UID: \"44b57332-91fe-483c-806f-f6576f18e461\") " Nov 02 06:40:01.528576 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:40:01.528549 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44b57332-91fe-483c-806f-f6576f18e461-maas-api-openshift-ca" pod "44b57332-91fe-483c-806f-f6576f18e461" (UID: "44b57332-91fe-483c-806f-f6576f18e461"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:40:01.528884 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:40:01.528866 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44b57332-91fe-483c-806f-f6576f18e461-maas-api-openshift-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:40:01.530517 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:40:01.530496 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44b57332-91fe-483c-806f-f6576f18e461-kube-api-access-7ckwn" pod "44b57332-91fe-483c-806f-f6576f18e461" (UID: "44b57332-91fe-483c-806f-f6576f18e461"). InnerVolumeSpecName "kube-api-access-7ckwn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:40:01.530850 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:40:01.530561 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44b57332-91fe-483c-806f-f6576f18e461-maas-api-xks-ca" pod "44b57332-91fe-483c-806f-f6576f18e461" (UID: "44b57332-91fe-483c-806f-f6576f18e461"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:40:01.630149 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:40:01.630072 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44b57332-91fe-483c-806f-f6576f18e461-maas-api-xks-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:40:01.630149 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:40:01.630102 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ckwn\" (UniqueName: \"kubernetes.io/projected/44b57332-91fe-483c-806f-f6576f18e461-kube-api-access-7ckwn\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:40:02.303619 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:40:02.303580 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5eed1bfb-wdr5w" event={"ID":"44b57332-91fe-483c-806f-f6576f18e461","Type":"ContainerDied","Data":"452eb493ebdbcdc31e59c9e2c224faca0a3403be71ff422f34f8edb874451afb"} Nov 02 06:40:02.303619 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:40:02.303614 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5eed1bfb-wdr5w" Nov 02 06:40:02.303810 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:40:02.303642 5916 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="452eb493ebdbcdc31e59c9e2c224faca0a3403be71ff422f34f8edb874451afb" Nov 02 06:40:19.905750 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:40:19.905713 5916 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p98cq" containerName="extractor" Nov 02 06:40:29.085487 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:40:29.085450 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-15867003-dcp8l"] Nov 02 06:40:29.136653 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:40:29.136621 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-15867003-dcp8l"] Nov 02 06:40:29.844791 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:40:29.844756 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b8402d1-7cc6-4072-835e-5987d2dd2c50" path="/var/lib/kubelet/pods/2b8402d1-7cc6-4072-835e-5987d2dd2c50/volumes" Nov 02 06:40:30.326217 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:40:30.326185 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d690ed79-867lw"] Nov 02 06:40:30.339945 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:40:30.339895 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d690ed79-867lw"] Nov 02 06:40:31.956440 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:40:31.956404 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92698bcd-5f10-4723-882e-e2be400e6259" path="/var/lib/kubelet/pods/92698bcd-5f10-4723-882e-e2be400e6259/volumes" Nov 02 06:40:36.843814 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:40:36.843776 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-14448910-8ggrp"] Nov 02 06:40:36.896223 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:40:36.896185 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-14448910-8ggrp"] Nov 02 06:40:37.235302 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:40:37.235262 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-gkz2f"] Nov 02 06:40:37.236024 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:40:37.236001 5916 cpu_manager.go:414] "containerMap: removing container" podUID="44b57332-91fe-483c-806f-f6576f18e461" containerName="revoke-keys" Nov 02 06:40:37.236024 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:40:37.236022 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44b57332-91fe-483c-806f-f6576f18e461" containerName="revoke-keys" Nov 02 06:40:37.236146 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:40:37.236043 5916 cpu_manager.go:414] "containerMap: removing container" podUID="2816f619-278a-4693-a977-1261c31ea5a4" containerName="revoke-keys" Nov 02 06:40:37.236146 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:40:37.236062 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2816f619-278a-4693-a977-1261c31ea5a4" containerName="revoke-keys" Nov 02 06:40:37.236321 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:40:37.236307 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="44b57332-91fe-483c-806f-f6576f18e461" containerName="revoke-keys" Nov 02 06:40:37.236360 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:40:37.236324 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="2816f619-278a-4693-a977-1261c31ea5a4" containerName="revoke-keys" Nov 02 06:40:37.242196 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:40:37.242172 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-gkz2f" Nov 02 06:40:37.337267 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:40:37.337188 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-gkz2f"] Nov 02 06:40:37.390043 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:40:37.390007 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc31d778-d117-4ca5-89e6-550d04804cd1-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-gkz2f\" (UID: \"bc31d778-d117-4ca5-89e6-550d04804cd1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-gkz2f" Nov 02 06:40:37.390261 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:40:37.390089 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc31d778-d117-4ca5-89e6-550d04804cd1-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-gkz2f\" (UID: \"bc31d778-d117-4ca5-89e6-550d04804cd1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-gkz2f" Nov 02 06:40:37.390261 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:40:37.390184 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc31d778-d117-4ca5-89e6-550d04804cd1-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-gkz2f\" (UID: \"bc31d778-d117-4ca5-89e6-550d04804cd1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-gkz2f" Nov 02 06:40:37.390381 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:40:37.390298 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcw5f\" (UniqueName: \"kubernetes.io/projected/bc31d778-d117-4ca5-89e6-550d04804cd1-kube-api-access-lcw5f\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-gkz2f\" (UID: \"bc31d778-d117-4ca5-89e6-550d04804cd1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-gkz2f" Nov 02 06:40:37.390381 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:40:37.390323 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc31d778-d117-4ca5-89e6-550d04804cd1-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-gkz2f\" (UID: \"bc31d778-d117-4ca5-89e6-550d04804cd1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-gkz2f" Nov 02 06:40:37.390463 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:40:37.390403 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc31d778-d117-4ca5-89e6-550d04804cd1-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-gkz2f\" (UID: \"bc31d778-d117-4ca5-89e6-550d04804cd1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-gkz2f" Nov 02 06:40:37.390463 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:40:37.390435 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc31d778-d117-4ca5-89e6-550d04804cd1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-gkz2f\" (UID: \"bc31d778-d117-4ca5-89e6-550d04804cd1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-gkz2f" Nov 02 06:40:37.390463 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:40:37.390460 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc31d778-d117-4ca5-89e6-550d04804cd1-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-gkz2f\" (UID: \"bc31d778-d117-4ca5-89e6-550d04804cd1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-gkz2f" Nov 02 06:40:37.390571 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:40:37.390536 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc31d778-d117-4ca5-89e6-550d04804cd1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-gkz2f\" (UID: \"bc31d778-d117-4ca5-89e6-550d04804cd1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-gkz2f" Nov 02 06:40:37.492241 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:40:37.492160 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc31d778-d117-4ca5-89e6-550d04804cd1-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-gkz2f\" (UID: \"bc31d778-d117-4ca5-89e6-550d04804cd1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-gkz2f" Nov 02 06:40:37.492241 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:40:37.492208 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc31d778-d117-4ca5-89e6-550d04804cd1-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-gkz2f\" (UID: \"bc31d778-d117-4ca5-89e6-550d04804cd1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-gkz2f" Nov 02 06:40:37.492462 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:40:37.492267 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcw5f\" (UniqueName: \"kubernetes.io/projected/bc31d778-d117-4ca5-89e6-550d04804cd1-kube-api-access-lcw5f\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-gkz2f\" (UID: \"bc31d778-d117-4ca5-89e6-550d04804cd1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-gkz2f" Nov 02 06:40:37.492462 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:40:37.492288 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc31d778-d117-4ca5-89e6-550d04804cd1-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-gkz2f\" (UID: \"bc31d778-d117-4ca5-89e6-550d04804cd1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-gkz2f" Nov 02 06:40:37.492462 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:40:37.492315 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc31d778-d117-4ca5-89e6-550d04804cd1-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-gkz2f\" (UID: \"bc31d778-d117-4ca5-89e6-550d04804cd1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-gkz2f" Nov 02 06:40:37.492462 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:40:37.492355 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc31d778-d117-4ca5-89e6-550d04804cd1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-gkz2f\" (UID: \"bc31d778-d117-4ca5-89e6-550d04804cd1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-gkz2f" Nov 02 06:40:37.492462 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:40:37.492394 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc31d778-d117-4ca5-89e6-550d04804cd1-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-gkz2f\" (UID: \"bc31d778-d117-4ca5-89e6-550d04804cd1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-gkz2f" Nov 02 06:40:37.492462 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:40:37.492422 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc31d778-d117-4ca5-89e6-550d04804cd1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-gkz2f\" (UID: \"bc31d778-d117-4ca5-89e6-550d04804cd1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-gkz2f" Nov 02 06:40:37.492896 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:40:37.492547 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc31d778-d117-4ca5-89e6-550d04804cd1-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-gkz2f\" (UID: \"bc31d778-d117-4ca5-89e6-550d04804cd1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-gkz2f" Nov 02 06:40:37.492896 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:40:37.492621 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc31d778-d117-4ca5-89e6-550d04804cd1-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-gkz2f\" (UID: \"bc31d778-d117-4ca5-89e6-550d04804cd1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-gkz2f" Nov 02 06:40:37.492896 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:40:37.492660 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc31d778-d117-4ca5-89e6-550d04804cd1-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-gkz2f\" (UID: \"bc31d778-d117-4ca5-89e6-550d04804cd1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-gkz2f" Nov 02 06:40:37.493125 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:40:37.493097 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc31d778-d117-4ca5-89e6-550d04804cd1-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-gkz2f\" (UID: \"bc31d778-d117-4ca5-89e6-550d04804cd1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-gkz2f" Nov 02 06:40:37.493253 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:40:37.493187 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc31d778-d117-4ca5-89e6-550d04804cd1-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-gkz2f\" (UID: \"bc31d778-d117-4ca5-89e6-550d04804cd1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-gkz2f" Nov 02 06:40:37.493253 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:40:37.493215 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc31d778-d117-4ca5-89e6-550d04804cd1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-gkz2f\" (UID: \"bc31d778-d117-4ca5-89e6-550d04804cd1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-gkz2f" Nov 02 06:40:37.494947 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:40:37.494922 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc31d778-d117-4ca5-89e6-550d04804cd1-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-gkz2f\" (UID: \"bc31d778-d117-4ca5-89e6-550d04804cd1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-gkz2f" Nov 02 06:40:37.495040 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:40:37.494949 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc31d778-d117-4ca5-89e6-550d04804cd1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-gkz2f\" (UID: \"bc31d778-d117-4ca5-89e6-550d04804cd1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-gkz2f" Nov 02 06:40:37.549079 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:40:37.549050 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc31d778-d117-4ca5-89e6-550d04804cd1-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-gkz2f\" (UID: \"bc31d778-d117-4ca5-89e6-550d04804cd1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-gkz2f" Nov 02 06:40:37.549230 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:40:37.549158 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcw5f\" (UniqueName: \"kubernetes.io/projected/bc31d778-d117-4ca5-89e6-550d04804cd1-kube-api-access-lcw5f\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-gkz2f\" (UID: \"bc31d778-d117-4ca5-89e6-550d04804cd1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-gkz2f" Nov 02 06:40:37.558127 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:40:37.558104 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-gkz2f" Nov 02 06:40:37.848292 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:40:37.848179 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ce243df-455d-4797-8492-3b3e403181e9" path="/var/lib/kubelet/pods/3ce243df-455d-4797-8492-3b3e403181e9/volumes" Nov 02 06:40:38.000138 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:40:38.000116 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-gkz2f"] Nov 02 06:40:38.000238 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:40:38.000161 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc31d778_d117_4ca5_89e6_550d04804cd1.slice/crio-e4f8dc887ad0953f328e3747fbfbce791301321ed53a4ca5d1fd7afe39763ee8 WatchSource:0}: Error finding container e4f8dc887ad0953f328e3747fbfbce791301321ed53a4ca5d1fd7afe39763ee8: Status 404 returned error can't find the container with id e4f8dc887ad0953f328e3747fbfbce791301321ed53a4ca5d1fd7afe39763ee8 Nov 02 06:40:38.002874 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:40:38.002845 5916 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 06:40:38.002981 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:40:38.002931 5916 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 06:40:38.002981 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:40:38.002960 5916 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 06:40:38.704006 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:40:38.703950 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-gkz2f" event={"ID":"bc31d778-d117-4ca5-89e6-550d04804cd1","Type":"ContainerStarted","Data":"5ebf6a023e04a1543efb593d498ddd475ad4ae4b9e09edda81a4a03d0d1d13d8"} Nov 02 06:40:38.704224 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:40:38.704036 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-gkz2f" event={"ID":"bc31d778-d117-4ca5-89e6-550d04804cd1","Type":"ContainerStarted","Data":"e4f8dc887ad0953f328e3747fbfbce791301321ed53a4ca5d1fd7afe39763ee8"} Nov 02 06:40:38.881477 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:40:38.881426 5916 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-gkz2f" podStartSLOduration=1.88141234 podStartE2EDuration="1.88141234s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:40:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:40:38.876975588 +0000 UTC m=+1427.148795982" watchObservedRunningTime="2026-11-02 06:40:38.88141234 +0000 UTC m=+1427.153232734" Nov 02 06:40:39.110985 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:40:39.107394 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5eed1bfb-wdr5w"] Nov 02 06:40:39.136927 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:40:39.136414 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5eed1bfb-wdr5w"] Nov 02 06:40:39.579632 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:40:39.566238 5916 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-gkz2f" Nov 02 06:40:39.579847 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:40:39.579698 5916 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-gkz2f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Nov 02 06:40:39.579847 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:40:39.579756 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-gkz2f" podUID="bc31d778-d117-4ca5-89e6-550d04804cd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Nov 02 06:40:39.866521 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:40:39.866439 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44b57332-91fe-483c-806f-f6576f18e461" path="/var/lib/kubelet/pods/44b57332-91fe-483c-806f-f6576f18e461/volumes" Nov 02 06:40:40.559464 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:40:40.558764 5916 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-gkz2f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Nov 02 06:40:40.559464 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:40:40.558823 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-gkz2f" podUID="bc31d778-d117-4ca5-89e6-550d04804cd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Nov 02 06:40:41.561270 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:40:41.561231 5916 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-gkz2f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Nov 02 06:40:41.561880 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:40:41.561846 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-gkz2f" podUID="bc31d778-d117-4ca5-89e6-550d04804cd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Nov 02 06:40:42.563928 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:40:42.561208 5916 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-gkz2f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Nov 02 06:40:42.563928 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:40:42.561282 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-gkz2f" podUID="bc31d778-d117-4ca5-89e6-550d04804cd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Nov 02 06:40:43.563930 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:40:43.558494 5916 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-gkz2f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Nov 02 06:40:43.563930 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:40:43.558554 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-gkz2f" podUID="bc31d778-d117-4ca5-89e6-550d04804cd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Nov 02 06:40:44.560049 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:40:44.560011 5916 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-gkz2f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Nov 02 06:40:44.570060 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:40:44.570015 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-gkz2f" podUID="bc31d778-d117-4ca5-89e6-550d04804cd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Nov 02 06:40:45.559289 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:40:45.559248 5916 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-gkz2f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Nov 02 06:40:45.559510 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:40:45.559309 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-gkz2f" podUID="bc31d778-d117-4ca5-89e6-550d04804cd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Nov 02 06:40:46.914674 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:40:46.914635 5916 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-gkz2f" Nov 02 06:40:46.915214 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:40:46.915146 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-gkz2f" Nov 02 06:40:46.916188 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:40:46.916170 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-gkz2f" Nov 02 06:40:49.905261 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:40:49.905204 5916 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p98cq" containerName="extractor" Nov 02 06:41:04.052351 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:41:04.052268 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a418ce9-tf7rb"] Nov 02 06:41:04.101003 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:41:04.100971 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a418ce9-tf7rb"] Nov 02 06:41:05.844261 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:41:05.844228 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d838d23-c56b-4439-ab37-a6b95e3317db" path="/var/lib/kubelet/pods/7d838d23-c56b-4439-ab37-a6b95e3317db/volumes" Nov 02 06:41:11.055307 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:41:11.055272 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0fe9b4ab-984fr"] Nov 02 06:41:11.101682 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:41:11.101652 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0fe9b4ab-984fr"] Nov 02 06:41:11.846246 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:41:11.846203 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ac2c97b-96e6-421c-8503-06eb94532e6a" path="/var/lib/kubelet/pods/7ac2c97b-96e6-421c-8503-06eb94532e6a/volumes" Nov 02 06:41:19.051123 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:41:19.051087 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7f7deb8-rql6s"] Nov 02 06:41:19.100354 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:41:19.100322 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7f7deb8-rql6s"] Nov 02 06:41:19.848890 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:41:19.848856 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fef5332f-da6e-4a02-9ee5-b50499a1c373" path="/var/lib/kubelet/pods/fef5332f-da6e-4a02-9ee5-b50499a1c373/volumes" Nov 02 06:41:19.901539 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:41:19.901500 5916 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p98cq" containerName="extractor" Nov 02 06:41:25.051510 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:41:25.051475 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10585146-mkldd"] Nov 02 06:41:25.101069 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:41:25.101039 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10585146-mkldd"] Nov 02 06:41:25.844147 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:41:25.844114 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd3cc14b-9cd3-4500-bf18-8a0832e357cd" path="/var/lib/kubelet/pods/fd3cc14b-9cd3-4500-bf18-8a0832e357cd/volumes" Nov 02 06:41:29.553552 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:41:29.553513 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5151c1f-7q6dl"] Nov 02 06:41:29.559956 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:41:29.559933 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5151c1f-7q6dl" Nov 02 06:41:29.618802 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:41:29.618771 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5151c1f-7q6dl"] Nov 02 06:41:29.632127 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:41:29.632094 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sj7pf\" (UniqueName: \"kubernetes.io/projected/b45f09ed-75c1-468b-a9ad-f1994301a61c-kube-api-access-sj7pf\") pod \"maas-api-revoke-sub-a5151c1f-7q6dl\" (UID: \"b45f09ed-75c1-468b-a9ad-f1994301a61c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5151c1f-7q6dl" Nov 02 06:41:29.632276 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:41:29.632231 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b45f09ed-75c1-468b-a9ad-f1994301a61c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a5151c1f-7q6dl\" (UID: \"b45f09ed-75c1-468b-a9ad-f1994301a61c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5151c1f-7q6dl" Nov 02 06:41:29.632345 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:41:29.632291 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b45f09ed-75c1-468b-a9ad-f1994301a61c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a5151c1f-7q6dl\" (UID: \"b45f09ed-75c1-468b-a9ad-f1994301a61c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5151c1f-7q6dl" Nov 02 06:41:29.661683 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:41:29.661658 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-68l4r\"" Nov 02 06:41:29.661959 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:41:29.661935 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 06:41:29.734122 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:41:29.734091 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sj7pf\" (UniqueName: \"kubernetes.io/projected/b45f09ed-75c1-468b-a9ad-f1994301a61c-kube-api-access-sj7pf\") pod \"maas-api-revoke-sub-a5151c1f-7q6dl\" (UID: \"b45f09ed-75c1-468b-a9ad-f1994301a61c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5151c1f-7q6dl" Nov 02 06:41:29.734319 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:41:29.734221 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b45f09ed-75c1-468b-a9ad-f1994301a61c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a5151c1f-7q6dl\" (UID: \"b45f09ed-75c1-468b-a9ad-f1994301a61c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5151c1f-7q6dl" Nov 02 06:41:29.734319 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:41:29.734282 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b45f09ed-75c1-468b-a9ad-f1994301a61c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a5151c1f-7q6dl\" (UID: \"b45f09ed-75c1-468b-a9ad-f1994301a61c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5151c1f-7q6dl" Nov 02 06:41:29.735133 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:41:29.735106 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b45f09ed-75c1-468b-a9ad-f1994301a61c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a5151c1f-7q6dl\" (UID: \"b45f09ed-75c1-468b-a9ad-f1994301a61c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5151c1f-7q6dl" Nov 02 06:41:29.736973 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:41:29.736950 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b45f09ed-75c1-468b-a9ad-f1994301a61c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a5151c1f-7q6dl\" (UID: \"b45f09ed-75c1-468b-a9ad-f1994301a61c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5151c1f-7q6dl" Nov 02 06:41:29.792017 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:41:29.791988 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sj7pf\" (UniqueName: \"kubernetes.io/projected/b45f09ed-75c1-468b-a9ad-f1994301a61c-kube-api-access-sj7pf\") pod \"maas-api-revoke-sub-a5151c1f-7q6dl\" (UID: \"b45f09ed-75c1-468b-a9ad-f1994301a61c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5151c1f-7q6dl" Nov 02 06:41:29.874606 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:41:29.874514 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5151c1f-7q6dl" Nov 02 06:41:30.297064 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:41:30.297033 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5151c1f-7q6dl"] Nov 02 06:41:30.297209 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:41:30.297126 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb45f09ed_75c1_468b_a9ad_f1994301a61c.slice/crio-6819a0a4e98c342f3c6a7d153af854aa148bce4d5523f96fe5eea8eaafde2e0e WatchSource:0}: Error finding container 6819a0a4e98c342f3c6a7d153af854aa148bce4d5523f96fe5eea8eaafde2e0e: Status 404 returned error can't find the container with id 6819a0a4e98c342f3c6a7d153af854aa148bce4d5523f96fe5eea8eaafde2e0e Nov 02 06:41:30.649080 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:41:30.649046 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5151c1f-7q6dl" event={"ID":"b45f09ed-75c1-468b-a9ad-f1994301a61c","Type":"ContainerStarted","Data":"6819a0a4e98c342f3c6a7d153af854aa148bce4d5523f96fe5eea8eaafde2e0e"} Nov 02 06:41:30.766603 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:41:30.766548 5916 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5151c1f-7q6dl" podStartSLOduration=1.7665353430000001 podStartE2EDuration="1.766535343s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:41:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:41:30.763104428 +0000 UTC m=+1479.034924820" watchObservedRunningTime="2026-11-02 06:41:30.766535343 +0000 UTC m=+1479.038355769" Nov 02 06:41:31.658534 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:41:31.658499 5916 generic.go:356] "Generic (PLEG): container finished" podID="b45f09ed-75c1-468b-a9ad-f1994301a61c" containerID="48be77aa50acf8f5a4105ec8d3925bea02af69fe3964db1df96ef6f0195a7656" exitCode=0 Nov 02 06:41:31.659013 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:41:31.658585 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5151c1f-7q6dl" event={"ID":"b45f09ed-75c1-468b-a9ad-f1994301a61c","Type":"ContainerDied","Data":"48be77aa50acf8f5a4105ec8d3925bea02af69fe3964db1df96ef6f0195a7656"} Nov 02 06:41:32.909211 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:41:32.909184 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5151c1f-7q6dl" Nov 02 06:41:32.985989 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:41:32.983764 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b45f09ed-75c1-468b-a9ad-f1994301a61c-kube-api-access-sj7pf\" (UniqueName: \"kubernetes.io/projected/b45f09ed-75c1-468b-a9ad-f1994301a61c-kube-api-access-sj7pf\") pod \"b45f09ed-75c1-468b-a9ad-f1994301a61c\" (UID: \"b45f09ed-75c1-468b-a9ad-f1994301a61c\") " Nov 02 06:41:32.985989 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:41:32.983840 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b45f09ed-75c1-468b-a9ad-f1994301a61c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b45f09ed-75c1-468b-a9ad-f1994301a61c-maas-api-xks-ca\") pod \"b45f09ed-75c1-468b-a9ad-f1994301a61c\" (UID: \"b45f09ed-75c1-468b-a9ad-f1994301a61c\") " Nov 02 06:41:32.985989 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:41:32.983995 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b45f09ed-75c1-468b-a9ad-f1994301a61c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b45f09ed-75c1-468b-a9ad-f1994301a61c-maas-api-openshift-ca\") pod \"b45f09ed-75c1-468b-a9ad-f1994301a61c\" (UID: \"b45f09ed-75c1-468b-a9ad-f1994301a61c\") " Nov 02 06:41:33.060648 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:41:33.060559 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b45f09ed-75c1-468b-a9ad-f1994301a61c-maas-api-openshift-ca" pod "b45f09ed-75c1-468b-a9ad-f1994301a61c" (UID: "b45f09ed-75c1-468b-a9ad-f1994301a61c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:41:33.099929 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:41:33.095782 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b45f09ed-75c1-468b-a9ad-f1994301a61c-maas-api-openshift-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:41:33.139929 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:41:33.122034 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b45f09ed-75c1-468b-a9ad-f1994301a61c-maas-api-xks-ca" pod "b45f09ed-75c1-468b-a9ad-f1994301a61c" (UID: "b45f09ed-75c1-468b-a9ad-f1994301a61c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:41:33.139929 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:41:33.139056 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b45f09ed-75c1-468b-a9ad-f1994301a61c-kube-api-access-sj7pf" pod "b45f09ed-75c1-468b-a9ad-f1994301a61c" (UID: "b45f09ed-75c1-468b-a9ad-f1994301a61c"). InnerVolumeSpecName "kube-api-access-sj7pf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:41:33.204844 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:41:33.204790 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sj7pf\" (UniqueName: \"kubernetes.io/projected/b45f09ed-75c1-468b-a9ad-f1994301a61c-kube-api-access-sj7pf\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:41:33.204844 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:41:33.204835 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b45f09ed-75c1-468b-a9ad-f1994301a61c-maas-api-xks-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:41:33.682163 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:41:33.682128 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5151c1f-7q6dl" event={"ID":"b45f09ed-75c1-468b-a9ad-f1994301a61c","Type":"ContainerDied","Data":"6819a0a4e98c342f3c6a7d153af854aa148bce4d5523f96fe5eea8eaafde2e0e"} Nov 02 06:41:33.682364 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:41:33.682207 5916 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6819a0a4e98c342f3c6a7d153af854aa148bce4d5523f96fe5eea8eaafde2e0e" Nov 02 06:41:33.682364 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:41:33.682142 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5151c1f-7q6dl" Nov 02 06:41:35.176945 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:41:35.176898 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3bf02ebf-59b5c46bd8-2d4n6"] Nov 02 06:41:35.177894 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:41:35.177856 5916 cpu_manager.go:414] "containerMap: removing container" podUID="b45f09ed-75c1-468b-a9ad-f1994301a61c" containerName="revoke-keys" Nov 02 06:41:35.177894 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:41:35.177882 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b45f09ed-75c1-468b-a9ad-f1994301a61c" containerName="revoke-keys" Nov 02 06:41:35.178113 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:41:35.178098 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="b45f09ed-75c1-468b-a9ad-f1994301a61c" containerName="revoke-keys" Nov 02 06:41:35.182385 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:41:35.182366 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3bf02ebf-59b5c46bd8-2d4n6" Nov 02 06:41:35.235166 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:41:35.235134 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-3bf02ebf-serving-cert\"" Nov 02 06:41:35.241376 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:41:35.241348 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3bf02ebf-59b5c46bd8-2d4n6"] Nov 02 06:41:35.336918 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:41:35.336871 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8df082e9-aecd-42e7-a7b0-a7cc929e269f-maas-api-tls\") pod \"maas-api-e2e-mt-3bf02ebf-59b5c46bd8-2d4n6\" (UID: \"8df082e9-aecd-42e7-a7b0-a7cc929e269f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3bf02ebf-59b5c46bd8-2d4n6" Nov 02 06:41:35.337115 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:41:35.337072 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcz9b\" (UniqueName: \"kubernetes.io/projected/8df082e9-aecd-42e7-a7b0-a7cc929e269f-kube-api-access-vcz9b\") pod \"maas-api-e2e-mt-3bf02ebf-59b5c46bd8-2d4n6\" (UID: \"8df082e9-aecd-42e7-a7b0-a7cc929e269f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3bf02ebf-59b5c46bd8-2d4n6" Nov 02 06:41:35.337115 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:41:35.337107 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8df082e9-aecd-42e7-a7b0-a7cc929e269f-openshift-service-ca\") pod \"maas-api-e2e-mt-3bf02ebf-59b5c46bd8-2d4n6\" (UID: \"8df082e9-aecd-42e7-a7b0-a7cc929e269f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3bf02ebf-59b5c46bd8-2d4n6" Nov 02 06:41:35.337288 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:41:35.337272 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8df082e9-aecd-42e7-a7b0-a7cc929e269f-metrics-certs\") pod \"maas-api-e2e-mt-3bf02ebf-59b5c46bd8-2d4n6\" (UID: \"8df082e9-aecd-42e7-a7b0-a7cc929e269f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3bf02ebf-59b5c46bd8-2d4n6" Nov 02 06:41:35.445977 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:41:35.444972 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8df082e9-aecd-42e7-a7b0-a7cc929e269f-maas-api-tls\") pod \"maas-api-e2e-mt-3bf02ebf-59b5c46bd8-2d4n6\" (UID: \"8df082e9-aecd-42e7-a7b0-a7cc929e269f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3bf02ebf-59b5c46bd8-2d4n6" Nov 02 06:41:35.445977 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:41:35.445114 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcz9b\" (UniqueName: \"kubernetes.io/projected/8df082e9-aecd-42e7-a7b0-a7cc929e269f-kube-api-access-vcz9b\") pod \"maas-api-e2e-mt-3bf02ebf-59b5c46bd8-2d4n6\" (UID: \"8df082e9-aecd-42e7-a7b0-a7cc929e269f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3bf02ebf-59b5c46bd8-2d4n6" Nov 02 06:41:35.445977 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:41:35.445146 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8df082e9-aecd-42e7-a7b0-a7cc929e269f-openshift-service-ca\") pod \"maas-api-e2e-mt-3bf02ebf-59b5c46bd8-2d4n6\" (UID: \"8df082e9-aecd-42e7-a7b0-a7cc929e269f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3bf02ebf-59b5c46bd8-2d4n6" Nov 02 06:41:35.445977 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:41:35.445262 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8df082e9-aecd-42e7-a7b0-a7cc929e269f-metrics-certs\") pod \"maas-api-e2e-mt-3bf02ebf-59b5c46bd8-2d4n6\" (UID: \"8df082e9-aecd-42e7-a7b0-a7cc929e269f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3bf02ebf-59b5c46bd8-2d4n6" Nov 02 06:41:35.451205 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:41:35.449336 5916 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-3bf02ebf-serving-cert: secret "maas-api-e2e-mt-3bf02ebf-serving-cert" not found Nov 02 06:41:35.451205 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:41:35.449418 5916 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8df082e9-aecd-42e7-a7b0-a7cc929e269f-maas-api-tls podName:8df082e9-aecd-42e7-a7b0-a7cc929e269f nodeName:}" failed. No retries permitted until 2026-11-02 06:41:35.949398637 +0000 UTC m=+1484.221219036 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8df082e9-aecd-42e7-a7b0-a7cc929e269f-maas-api-tls") pod "maas-api-e2e-mt-3bf02ebf-59b5c46bd8-2d4n6" (UID: "8df082e9-aecd-42e7-a7b0-a7cc929e269f") : secret "maas-api-e2e-mt-3bf02ebf-serving-cert" not found Nov 02 06:41:35.451205 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:41:35.449723 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8df082e9-aecd-42e7-a7b0-a7cc929e269f-metrics-certs\") pod \"maas-api-e2e-mt-3bf02ebf-59b5c46bd8-2d4n6\" (UID: \"8df082e9-aecd-42e7-a7b0-a7cc929e269f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3bf02ebf-59b5c46bd8-2d4n6" Nov 02 06:41:35.451205 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:41:35.451050 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8df082e9-aecd-42e7-a7b0-a7cc929e269f-openshift-service-ca\") pod \"maas-api-e2e-mt-3bf02ebf-59b5c46bd8-2d4n6\" (UID: \"8df082e9-aecd-42e7-a7b0-a7cc929e269f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3bf02ebf-59b5c46bd8-2d4n6" Nov 02 06:41:35.507214 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:41:35.507185 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcz9b\" (UniqueName: \"kubernetes.io/projected/8df082e9-aecd-42e7-a7b0-a7cc929e269f-kube-api-access-vcz9b\") pod \"maas-api-e2e-mt-3bf02ebf-59b5c46bd8-2d4n6\" (UID: \"8df082e9-aecd-42e7-a7b0-a7cc929e269f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3bf02ebf-59b5c46bd8-2d4n6" Nov 02 06:41:35.960219 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:41:35.960160 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8df082e9-aecd-42e7-a7b0-a7cc929e269f-maas-api-tls\") pod \"maas-api-e2e-mt-3bf02ebf-59b5c46bd8-2d4n6\" (UID: \"8df082e9-aecd-42e7-a7b0-a7cc929e269f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3bf02ebf-59b5c46bd8-2d4n6" Nov 02 06:41:35.964141 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:41:35.964117 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8df082e9-aecd-42e7-a7b0-a7cc929e269f-maas-api-tls\") pod \"maas-api-e2e-mt-3bf02ebf-59b5c46bd8-2d4n6\" (UID: \"8df082e9-aecd-42e7-a7b0-a7cc929e269f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3bf02ebf-59b5c46bd8-2d4n6" Nov 02 06:41:36.098160 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:41:36.098121 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3bf02ebf-59b5c46bd8-2d4n6" Nov 02 06:41:36.504048 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:41:36.503728 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3bf02ebf-59b5c46bd8-2d4n6"] Nov 02 06:41:36.746250 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:41:36.745983 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3bf02ebf-59b5c46bd8-2d4n6" event={"ID":"8df082e9-aecd-42e7-a7b0-a7cc929e269f","Type":"ContainerStarted","Data":"3290de6d3eba499ae9fa91f4a6e6182b4d2cd620179d8d5ff04d2b666dac3a80"} Nov 02 06:41:38.767989 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:41:38.767955 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3bf02ebf-59b5c46bd8-2d4n6" event={"ID":"8df082e9-aecd-42e7-a7b0-a7cc929e269f","Type":"ContainerStarted","Data":"df69e499b533701da6fdfcd3a2b60ef0d05ec1cea35de6c93b29f0c81c11bc29"} Nov 02 06:41:38.768395 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:41:38.768180 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3bf02ebf-59b5c46bd8-2d4n6" Nov 02 06:41:38.885115 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:41:38.885064 5916 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3bf02ebf-59b5c46bd8-2d4n6" podStartSLOduration=2.615945313 podStartE2EDuration="3.885048171s" totalImagesPullingTime="1.269102858s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:41:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:41:38.882603553 +0000 UTC m=+1487.154423938" watchObservedRunningTime="2026-11-02 06:41:38.885048171 +0000 UTC m=+1487.156868564" Nov 02 06:41:41.123823 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:41:41.123775 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27a288b1-m49qv"] Nov 02 06:41:41.131647 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:41:41.131613 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27a288b1-m49qv"] Nov 02 06:41:41.850255 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:41:41.850218 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b98be7f-f248-4ab5-9a80-f9d0f27cd6a4" path="/var/lib/kubelet/pods/6b98be7f-f248-4ab5-9a80-f9d0f27cd6a4/volumes" Nov 02 06:41:42.051895 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:41:42.051856 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03165c46-z897v"] Nov 02 06:41:42.104215 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:41:42.104135 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03165c46-z897v"] Nov 02 06:41:43.849546 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:41:43.849495 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1acf763f-268c-4bd3-ac4a-3f701c0e2dbd" path="/var/lib/kubelet/pods/1acf763f-268c-4bd3-ac4a-3f701c0e2dbd/volumes" Nov 02 06:41:44.783784 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:41:44.783753 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3bf02ebf-59b5c46bd8-2d4n6" Nov 02 06:41:47.050896 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:41:47.050861 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f9f9a772-sgd5d"] Nov 02 06:41:47.100571 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:41:47.100545 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f9f9a772-sgd5d"] Nov 02 06:41:47.916926 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:41:47.904670 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="253c37b4-7bd4-4310-b8be-b37c64f6b57a" path="/var/lib/kubelet/pods/253c37b4-7bd4-4310-b8be-b37c64f6b57a/volumes" Nov 02 06:41:49.896934 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:41:49.896887 5916 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p98cq" containerName="extractor" Nov 02 06:41:53.042092 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:41:53.042061 5916 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-csfhg_17dee72b-2f47-4016-8ffe-3a89934e66ee/console-operator/2.log" Nov 02 06:41:53.050229 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:41:53.050210 5916 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-csfhg_17dee72b-2f47-4016-8ffe-3a89934e66ee/console-operator/2.log" Nov 02 06:41:53.060093 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:41:53.060073 5916 kubelet.go:1689] "Image garbage collection succeeded" Nov 02 06:41:54.051006 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:41:54.050966 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8f6d2ba-bs5d7"] Nov 02 06:41:54.101089 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:41:54.101055 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8f6d2ba-bs5d7"] Nov 02 06:41:55.848955 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:41:55.848925 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88ea4c51-e049-495a-b69c-306b6b665085" path="/var/lib/kubelet/pods/88ea4c51-e049-495a-b69c-306b6b665085/volumes" Nov 02 06:41:59.051334 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:41:59.051299 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8bf36a9a-jgxp9"] Nov 02 06:41:59.100790 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:41:59.100760 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8bf36a9a-jgxp9"] Nov 02 06:41:59.845276 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:41:59.845247 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e7cefed-95d9-40d1-9c49-60394d0b4c84" path="/var/lib/kubelet/pods/3e7cefed-95d9-40d1-9c49-60394d0b4c84/volumes" Nov 02 06:42:04.052812 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:42:04.052774 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f42bb79-ddrpm"] Nov 02 06:42:04.102490 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:42:04.102458 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f42bb79-ddrpm"] Nov 02 06:42:05.846478 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:42:05.846446 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c80d239a-c8b0-4869-80e7-6e6f491f812e" path="/var/lib/kubelet/pods/c80d239a-c8b0-4869-80e7-6e6f491f812e/volumes" Nov 02 06:42:06.051356 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:42:06.051324 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06e94cdb-5sj4f"] Nov 02 06:42:06.100572 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:42:06.100504 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06e94cdb-5sj4f"] Nov 02 06:42:07.844387 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:42:07.844354 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a491cd95-8967-4f09-82f9-d154761e79ea" path="/var/lib/kubelet/pods/a491cd95-8967-4f09-82f9-d154761e79ea/volumes" Nov 02 06:42:12.068427 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:42:12.068391 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-262f26f2-p6vln"] Nov 02 06:42:12.105198 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:42:12.105152 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-262f26f2-p6vln"] Nov 02 06:42:13.852269 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:42:13.852230 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e2b0972-08eb-4028-9e16-1905c9855114" path="/var/lib/kubelet/pods/8e2b0972-08eb-4028-9e16-1905c9855114/volumes" Nov 02 06:42:17.051400 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:42:17.051366 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75a270a2-sj59m"] Nov 02 06:42:17.101668 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:42:17.101640 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6119a045-whwqc"] Nov 02 06:42:17.105739 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:42:17.105717 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75a270a2-sj59m"] Nov 02 06:42:17.109158 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:42:17.109138 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6119a045-whwqc"] Nov 02 06:42:17.728357 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:42:17.728314 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-29abe7-openshift-default-555ff8c7fc-6zdzz"] Nov 02 06:42:17.734535 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:42:17.733941 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-29abe7-openshift-default-555ff8c7fc-6zdzz" Nov 02 06:42:17.806979 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:42:17.804318 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-29abe7-openshift-default-dockercfg-nb2fc\"" Nov 02 06:42:17.828359 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:42:17.828321 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptcxt\" (UniqueName: \"kubernetes.io/projected/5f971e93-8d12-41cf-945c-c932fee7dc16-kube-api-access-ptcxt\") pod \"e2e-shared-b-w0-29abe7-openshift-default-555ff8c7fc-6zdzz\" (UID: \"5f971e93-8d12-41cf-945c-c932fee7dc16\") " pod="openshift-ingress/e2e-shared-b-w0-29abe7-openshift-default-555ff8c7fc-6zdzz" Nov 02 06:42:17.828636 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:42:17.828611 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f971e93-8d12-41cf-945c-c932fee7dc16-istio-token\") pod \"e2e-shared-b-w0-29abe7-openshift-default-555ff8c7fc-6zdzz\" (UID: \"5f971e93-8d12-41cf-945c-c932fee7dc16\") " pod="openshift-ingress/e2e-shared-b-w0-29abe7-openshift-default-555ff8c7fc-6zdzz" Nov 02 06:42:17.828979 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:42:17.828898 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f971e93-8d12-41cf-945c-c932fee7dc16-credential-socket\") pod \"e2e-shared-b-w0-29abe7-openshift-default-555ff8c7fc-6zdzz\" (UID: \"5f971e93-8d12-41cf-945c-c932fee7dc16\") " pod="openshift-ingress/e2e-shared-b-w0-29abe7-openshift-default-555ff8c7fc-6zdzz" Nov 02 06:42:17.829235 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:42:17.829204 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f971e93-8d12-41cf-945c-c932fee7dc16-workload-socket\") pod \"e2e-shared-b-w0-29abe7-openshift-default-555ff8c7fc-6zdzz\" (UID: \"5f971e93-8d12-41cf-945c-c932fee7dc16\") " pod="openshift-ingress/e2e-shared-b-w0-29abe7-openshift-default-555ff8c7fc-6zdzz" Nov 02 06:42:17.829417 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:42:17.829400 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f971e93-8d12-41cf-945c-c932fee7dc16-istiod-ca-cert\") pod \"e2e-shared-b-w0-29abe7-openshift-default-555ff8c7fc-6zdzz\" (UID: \"5f971e93-8d12-41cf-945c-c932fee7dc16\") " pod="openshift-ingress/e2e-shared-b-w0-29abe7-openshift-default-555ff8c7fc-6zdzz" Nov 02 06:42:17.829565 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:42:17.829549 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f971e93-8d12-41cf-945c-c932fee7dc16-istio-data\") pod \"e2e-shared-b-w0-29abe7-openshift-default-555ff8c7fc-6zdzz\" (UID: \"5f971e93-8d12-41cf-945c-c932fee7dc16\") " pod="openshift-ingress/e2e-shared-b-w0-29abe7-openshift-default-555ff8c7fc-6zdzz" Nov 02 06:42:17.829706 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:42:17.829689 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f971e93-8d12-41cf-945c-c932fee7dc16-workload-certs\") pod \"e2e-shared-b-w0-29abe7-openshift-default-555ff8c7fc-6zdzz\" (UID: \"5f971e93-8d12-41cf-945c-c932fee7dc16\") " pod="openshift-ingress/e2e-shared-b-w0-29abe7-openshift-default-555ff8c7fc-6zdzz" Nov 02 06:42:17.829865 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:42:17.829848 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f971e93-8d12-41cf-945c-c932fee7dc16-istio-podinfo\") pod \"e2e-shared-b-w0-29abe7-openshift-default-555ff8c7fc-6zdzz\" (UID: \"5f971e93-8d12-41cf-945c-c932fee7dc16\") " pod="openshift-ingress/e2e-shared-b-w0-29abe7-openshift-default-555ff8c7fc-6zdzz" Nov 02 06:42:17.830055 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:42:17.830036 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f971e93-8d12-41cf-945c-c932fee7dc16-istio-envoy\") pod \"e2e-shared-b-w0-29abe7-openshift-default-555ff8c7fc-6zdzz\" (UID: \"5f971e93-8d12-41cf-945c-c932fee7dc16\") " pod="openshift-ingress/e2e-shared-b-w0-29abe7-openshift-default-555ff8c7fc-6zdzz" Nov 02 06:42:17.905939 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:42:17.900221 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="106a8465-7e5e-4f8d-ae57-f8b00c6e5a09" path="/var/lib/kubelet/pods/106a8465-7e5e-4f8d-ae57-f8b00c6e5a09/volumes" Nov 02 06:42:17.905939 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:42:17.904258 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d220594f-4b71-4f58-9efd-d95525b5f8a0" path="/var/lib/kubelet/pods/d220594f-4b71-4f58-9efd-d95525b5f8a0/volumes" Nov 02 06:42:17.905939 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:42:17.904739 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-29abe7-openshift-default-555ff8c7fc-6zdzz"] Nov 02 06:42:17.940291 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:42:17.934932 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f971e93-8d12-41cf-945c-c932fee7dc16-credential-socket\") pod \"e2e-shared-b-w0-29abe7-openshift-default-555ff8c7fc-6zdzz\" (UID: \"5f971e93-8d12-41cf-945c-c932fee7dc16\") " pod="openshift-ingress/e2e-shared-b-w0-29abe7-openshift-default-555ff8c7fc-6zdzz" Nov 02 06:42:17.941397 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:42:17.941280 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f971e93-8d12-41cf-945c-c932fee7dc16-credential-socket\") pod \"e2e-shared-b-w0-29abe7-openshift-default-555ff8c7fc-6zdzz\" (UID: \"5f971e93-8d12-41cf-945c-c932fee7dc16\") " pod="openshift-ingress/e2e-shared-b-w0-29abe7-openshift-default-555ff8c7fc-6zdzz" Nov 02 06:42:17.941547 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:42:17.941369 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f971e93-8d12-41cf-945c-c932fee7dc16-workload-socket\") pod \"e2e-shared-b-w0-29abe7-openshift-default-555ff8c7fc-6zdzz\" (UID: \"5f971e93-8d12-41cf-945c-c932fee7dc16\") " pod="openshift-ingress/e2e-shared-b-w0-29abe7-openshift-default-555ff8c7fc-6zdzz" Nov 02 06:42:17.941547 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:42:17.941523 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f971e93-8d12-41cf-945c-c932fee7dc16-istiod-ca-cert\") pod \"e2e-shared-b-w0-29abe7-openshift-default-555ff8c7fc-6zdzz\" (UID: \"5f971e93-8d12-41cf-945c-c932fee7dc16\") " pod="openshift-ingress/e2e-shared-b-w0-29abe7-openshift-default-555ff8c7fc-6zdzz" Nov 02 06:42:17.941676 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:42:17.941618 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f971e93-8d12-41cf-945c-c932fee7dc16-istio-data\") pod \"e2e-shared-b-w0-29abe7-openshift-default-555ff8c7fc-6zdzz\" (UID: \"5f971e93-8d12-41cf-945c-c932fee7dc16\") " pod="openshift-ingress/e2e-shared-b-w0-29abe7-openshift-default-555ff8c7fc-6zdzz" Nov 02 06:42:17.941736 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:42:17.941685 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f971e93-8d12-41cf-945c-c932fee7dc16-workload-certs\") pod \"e2e-shared-b-w0-29abe7-openshift-default-555ff8c7fc-6zdzz\" (UID: \"5f971e93-8d12-41cf-945c-c932fee7dc16\") " pod="openshift-ingress/e2e-shared-b-w0-29abe7-openshift-default-555ff8c7fc-6zdzz" Nov 02 06:42:17.941791 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:42:17.941747 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f971e93-8d12-41cf-945c-c932fee7dc16-istio-podinfo\") pod \"e2e-shared-b-w0-29abe7-openshift-default-555ff8c7fc-6zdzz\" (UID: \"5f971e93-8d12-41cf-945c-c932fee7dc16\") " pod="openshift-ingress/e2e-shared-b-w0-29abe7-openshift-default-555ff8c7fc-6zdzz" Nov 02 06:42:17.941854 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:42:17.941828 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f971e93-8d12-41cf-945c-c932fee7dc16-istio-envoy\") pod \"e2e-shared-b-w0-29abe7-openshift-default-555ff8c7fc-6zdzz\" (UID: \"5f971e93-8d12-41cf-945c-c932fee7dc16\") " pod="openshift-ingress/e2e-shared-b-w0-29abe7-openshift-default-555ff8c7fc-6zdzz" Nov 02 06:42:17.941927 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:42:17.941900 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptcxt\" (UniqueName: \"kubernetes.io/projected/5f971e93-8d12-41cf-945c-c932fee7dc16-kube-api-access-ptcxt\") pod \"e2e-shared-b-w0-29abe7-openshift-default-555ff8c7fc-6zdzz\" (UID: \"5f971e93-8d12-41cf-945c-c932fee7dc16\") " pod="openshift-ingress/e2e-shared-b-w0-29abe7-openshift-default-555ff8c7fc-6zdzz" Nov 02 06:42:17.941990 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:42:17.941953 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f971e93-8d12-41cf-945c-c932fee7dc16-istio-token\") pod \"e2e-shared-b-w0-29abe7-openshift-default-555ff8c7fc-6zdzz\" (UID: \"5f971e93-8d12-41cf-945c-c932fee7dc16\") " pod="openshift-ingress/e2e-shared-b-w0-29abe7-openshift-default-555ff8c7fc-6zdzz" Nov 02 06:42:17.942344 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:42:17.942320 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f971e93-8d12-41cf-945c-c932fee7dc16-workload-certs\") pod \"e2e-shared-b-w0-29abe7-openshift-default-555ff8c7fc-6zdzz\" (UID: \"5f971e93-8d12-41cf-945c-c932fee7dc16\") " pod="openshift-ingress/e2e-shared-b-w0-29abe7-openshift-default-555ff8c7fc-6zdzz" Nov 02 06:42:17.943865 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:42:17.943845 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f971e93-8d12-41cf-945c-c932fee7dc16-istiod-ca-cert\") pod \"e2e-shared-b-w0-29abe7-openshift-default-555ff8c7fc-6zdzz\" (UID: \"5f971e93-8d12-41cf-945c-c932fee7dc16\") " pod="openshift-ingress/e2e-shared-b-w0-29abe7-openshift-default-555ff8c7fc-6zdzz" Nov 02 06:42:17.945337 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:42:17.944376 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f971e93-8d12-41cf-945c-c932fee7dc16-workload-socket\") pod \"e2e-shared-b-w0-29abe7-openshift-default-555ff8c7fc-6zdzz\" (UID: \"5f971e93-8d12-41cf-945c-c932fee7dc16\") " pod="openshift-ingress/e2e-shared-b-w0-29abe7-openshift-default-555ff8c7fc-6zdzz" Nov 02 06:42:17.945463 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:42:17.944806 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f971e93-8d12-41cf-945c-c932fee7dc16-istio-data\") pod \"e2e-shared-b-w0-29abe7-openshift-default-555ff8c7fc-6zdzz\" (UID: \"5f971e93-8d12-41cf-945c-c932fee7dc16\") " pod="openshift-ingress/e2e-shared-b-w0-29abe7-openshift-default-555ff8c7fc-6zdzz" Nov 02 06:42:17.975445 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:42:17.974566 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f971e93-8d12-41cf-945c-c932fee7dc16-istio-envoy\") pod \"e2e-shared-b-w0-29abe7-openshift-default-555ff8c7fc-6zdzz\" (UID: \"5f971e93-8d12-41cf-945c-c932fee7dc16\") " pod="openshift-ingress/e2e-shared-b-w0-29abe7-openshift-default-555ff8c7fc-6zdzz" Nov 02 06:42:17.989506 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:42:17.989419 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f971e93-8d12-41cf-945c-c932fee7dc16-istio-podinfo\") pod \"e2e-shared-b-w0-29abe7-openshift-default-555ff8c7fc-6zdzz\" (UID: \"5f971e93-8d12-41cf-945c-c932fee7dc16\") " pod="openshift-ingress/e2e-shared-b-w0-29abe7-openshift-default-555ff8c7fc-6zdzz" Nov 02 06:42:18.007571 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:42:18.006764 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptcxt\" (UniqueName: \"kubernetes.io/projected/5f971e93-8d12-41cf-945c-c932fee7dc16-kube-api-access-ptcxt\") pod \"e2e-shared-b-w0-29abe7-openshift-default-555ff8c7fc-6zdzz\" (UID: \"5f971e93-8d12-41cf-945c-c932fee7dc16\") " pod="openshift-ingress/e2e-shared-b-w0-29abe7-openshift-default-555ff8c7fc-6zdzz" Nov 02 06:42:18.022933 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:42:18.009789 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f971e93-8d12-41cf-945c-c932fee7dc16-istio-token\") pod \"e2e-shared-b-w0-29abe7-openshift-default-555ff8c7fc-6zdzz\" (UID: \"5f971e93-8d12-41cf-945c-c932fee7dc16\") " pod="openshift-ingress/e2e-shared-b-w0-29abe7-openshift-default-555ff8c7fc-6zdzz" Nov 02 06:42:18.127930 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:42:18.127873 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-29abe7-openshift-default-555ff8c7fc-6zdzz" Nov 02 06:42:18.686231 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:42:18.686201 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-29abe7-openshift-default-555ff8c7fc-6zdzz"] Nov 02 06:42:18.686735 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:42:18.686710 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f971e93_8d12_41cf_945c_c932fee7dc16.slice/crio-47d5ff6aa228002406ac330575aea9e32508f0ad724bedb6101a08773af04ef6 WatchSource:0}: Error finding container 47d5ff6aa228002406ac330575aea9e32508f0ad724bedb6101a08773af04ef6: Status 404 returned error can't find the container with id 47d5ff6aa228002406ac330575aea9e32508f0ad724bedb6101a08773af04ef6 Nov 02 06:42:18.689466 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:42:18.689435 5916 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 06:42:18.689568 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:42:18.689497 5916 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 06:42:18.689568 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:42:18.689528 5916 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 06:42:19.327752 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:42:19.327711 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-29abe7-openshift-default-555ff8c7fc-6zdzz" event={"ID":"5f971e93-8d12-41cf-945c-c932fee7dc16","Type":"ContainerStarted","Data":"35301f49b485c12f73f63e76bade89ff9a832ba49b644c6b94ad4b35f00b0610"} Nov 02 06:42:19.329288 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:42:19.329260 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-29abe7-openshift-default-555ff8c7fc-6zdzz" event={"ID":"5f971e93-8d12-41cf-945c-c932fee7dc16","Type":"ContainerStarted","Data":"47d5ff6aa228002406ac330575aea9e32508f0ad724bedb6101a08773af04ef6"} Nov 02 06:42:19.494485 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:42:19.494426 5916 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-29abe7-openshift-default-555ff8c7fc-6zdzz" podStartSLOduration=2.494409805 podStartE2EDuration="2.494409805s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:42:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:42:19.494043545 +0000 UTC m=+1527.765863966" watchObservedRunningTime="2026-11-02 06:42:19.494409805 +0000 UTC m=+1527.766230200" Nov 02 06:42:19.896652 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:42:19.896619 5916 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p98cq" containerName="extractor" Nov 02 06:42:20.128831 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:42:20.128800 5916 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-29abe7-openshift-default-555ff8c7fc-6zdzz" Nov 02 06:42:20.135540 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:42:20.135516 5916 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-29abe7-openshift-default-555ff8c7fc-6zdzz" Nov 02 06:42:20.340019 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:42:20.339900 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-29abe7-openshift-default-555ff8c7fc-6zdzz" Nov 02 06:42:20.341260 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:42:20.341239 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-29abe7-openshift-default-555ff8c7fc-6zdzz" Nov 02 06:42:22.049305 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:42:22.049272 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04d7ec61-hch7s"] Nov 02 06:42:22.098375 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:42:22.098347 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04d7ec61-hch7s"] Nov 02 06:42:23.847166 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:42:23.847129 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e338821a-dac8-4f6f-80fa-dd6e4a180258" path="/var/lib/kubelet/pods/e338821a-dac8-4f6f-80fa-dd6e4a180258/volumes" Nov 02 06:42:26.052229 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:42:26.052195 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f5fa4b3-9jtjn"] Nov 02 06:42:26.102006 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:42:26.101975 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f5fa4b3-9jtjn"] Nov 02 06:42:27.845208 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:42:27.845174 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f458ad6-b62f-4f8a-ad43-95f51f2bee4c" path="/var/lib/kubelet/pods/9f458ad6-b62f-4f8a-ad43-95f51f2bee4c/volumes" Nov 02 06:42:29.065936 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:42:29.064673 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d714765b-npkpv"] Nov 02 06:42:29.107095 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:42:29.107060 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d714765b-npkpv"] Nov 02 06:42:29.847015 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:42:29.846972 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88d99242-0923-4329-bfc5-9d3f2f94c8c7" path="/var/lib/kubelet/pods/88d99242-0923-4329-bfc5-9d3f2f94c8c7/volumes" Nov 02 06:42:36.053376 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:42:36.053296 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5636d7f-vvq8s"] Nov 02 06:42:36.101084 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:42:36.101052 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5636d7f-vvq8s"] Nov 02 06:42:37.845976 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:42:37.845933 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82b7b98c-5ebb-4517-97a1-e8f58413243c" path="/var/lib/kubelet/pods/82b7b98c-5ebb-4517-97a1-e8f58413243c/volumes" Nov 02 06:42:44.337108 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:42:44.337073 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e5502564-65855c7c98-q8rn7"] Nov 02 06:42:44.342671 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:42:44.342645 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e5502564-65855c7c98-q8rn7" Nov 02 06:42:44.442928 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:42:44.425185 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e5502564-65855c7c98-q8rn7"] Nov 02 06:42:44.442928 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:42:44.428564 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9edf52d8-d09b-40c8-84df-32020447afd2-metrics-certs\") pod \"maas-api-e2e-mt-e5502564-65855c7c98-q8rn7\" (UID: \"9edf52d8-d09b-40c8-84df-32020447afd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e5502564-65855c7c98-q8rn7" Nov 02 06:42:44.442928 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:42:44.440165 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9edf52d8-d09b-40c8-84df-32020447afd2-maas-api-tls\") pod \"maas-api-e2e-mt-e5502564-65855c7c98-q8rn7\" (UID: \"9edf52d8-d09b-40c8-84df-32020447afd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e5502564-65855c7c98-q8rn7" Nov 02 06:42:44.442928 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:42:44.440251 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkspc\" (UniqueName: \"kubernetes.io/projected/9edf52d8-d09b-40c8-84df-32020447afd2-kube-api-access-gkspc\") pod \"maas-api-e2e-mt-e5502564-65855c7c98-q8rn7\" (UID: \"9edf52d8-d09b-40c8-84df-32020447afd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e5502564-65855c7c98-q8rn7" Nov 02 06:42:44.442928 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:42:44.440392 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9edf52d8-d09b-40c8-84df-32020447afd2-openshift-service-ca\") pod \"maas-api-e2e-mt-e5502564-65855c7c98-q8rn7\" (UID: \"9edf52d8-d09b-40c8-84df-32020447afd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e5502564-65855c7c98-q8rn7" Nov 02 06:42:44.460926 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:42:44.456279 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-e5502564-serving-cert\"" Nov 02 06:42:44.547897 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:42:44.547853 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9edf52d8-d09b-40c8-84df-32020447afd2-metrics-certs\") pod \"maas-api-e2e-mt-e5502564-65855c7c98-q8rn7\" (UID: \"9edf52d8-d09b-40c8-84df-32020447afd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e5502564-65855c7c98-q8rn7" Nov 02 06:42:44.548237 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:42:44.548219 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9edf52d8-d09b-40c8-84df-32020447afd2-maas-api-tls\") pod \"maas-api-e2e-mt-e5502564-65855c7c98-q8rn7\" (UID: \"9edf52d8-d09b-40c8-84df-32020447afd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e5502564-65855c7c98-q8rn7" Nov 02 06:42:44.548379 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:42:44.548363 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkspc\" (UniqueName: \"kubernetes.io/projected/9edf52d8-d09b-40c8-84df-32020447afd2-kube-api-access-gkspc\") pod \"maas-api-e2e-mt-e5502564-65855c7c98-q8rn7\" (UID: \"9edf52d8-d09b-40c8-84df-32020447afd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e5502564-65855c7c98-q8rn7" Nov 02 06:42:44.555164 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:42:44.555131 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9edf52d8-d09b-40c8-84df-32020447afd2-openshift-service-ca\") pod \"maas-api-e2e-mt-e5502564-65855c7c98-q8rn7\" (UID: \"9edf52d8-d09b-40c8-84df-32020447afd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e5502564-65855c7c98-q8rn7" Nov 02 06:42:44.556073 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:42:44.556046 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9edf52d8-d09b-40c8-84df-32020447afd2-openshift-service-ca\") pod \"maas-api-e2e-mt-e5502564-65855c7c98-q8rn7\" (UID: \"9edf52d8-d09b-40c8-84df-32020447afd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e5502564-65855c7c98-q8rn7" Nov 02 06:42:44.556337 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:42:44.556320 5916 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-e5502564-serving-cert: secret "maas-api-e2e-mt-e5502564-serving-cert" not found Nov 02 06:42:44.556527 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:42:44.556515 5916 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9edf52d8-d09b-40c8-84df-32020447afd2-maas-api-tls podName:9edf52d8-d09b-40c8-84df-32020447afd2 nodeName:}" failed. No retries permitted until 2026-11-02 06:42:45.056485903 +0000 UTC m=+1553.328306281 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9edf52d8-d09b-40c8-84df-32020447afd2-maas-api-tls") pod "maas-api-e2e-mt-e5502564-65855c7c98-q8rn7" (UID: "9edf52d8-d09b-40c8-84df-32020447afd2") : secret "maas-api-e2e-mt-e5502564-serving-cert" not found Nov 02 06:42:44.576073 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:42:44.569488 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9edf52d8-d09b-40c8-84df-32020447afd2-metrics-certs\") pod \"maas-api-e2e-mt-e5502564-65855c7c98-q8rn7\" (UID: \"9edf52d8-d09b-40c8-84df-32020447afd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e5502564-65855c7c98-q8rn7" Nov 02 06:42:44.618568 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:42:44.618494 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkspc\" (UniqueName: \"kubernetes.io/projected/9edf52d8-d09b-40c8-84df-32020447afd2-kube-api-access-gkspc\") pod \"maas-api-e2e-mt-e5502564-65855c7c98-q8rn7\" (UID: \"9edf52d8-d09b-40c8-84df-32020447afd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e5502564-65855c7c98-q8rn7" Nov 02 06:42:45.074956 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:42:45.074922 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9edf52d8-d09b-40c8-84df-32020447afd2-maas-api-tls\") pod \"maas-api-e2e-mt-e5502564-65855c7c98-q8rn7\" (UID: \"9edf52d8-d09b-40c8-84df-32020447afd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e5502564-65855c7c98-q8rn7" Nov 02 06:42:45.077506 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:42:45.077481 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9edf52d8-d09b-40c8-84df-32020447afd2-maas-api-tls\") pod \"maas-api-e2e-mt-e5502564-65855c7c98-q8rn7\" (UID: \"9edf52d8-d09b-40c8-84df-32020447afd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e5502564-65855c7c98-q8rn7" Nov 02 06:42:45.299579 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:42:45.299544 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e5502564-65855c7c98-q8rn7" Nov 02 06:42:45.694446 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:42:45.694415 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e5502564-65855c7c98-q8rn7"] Nov 02 06:42:45.695253 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:42:45.695225 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9edf52d8_d09b_40c8_84df_32020447afd2.slice/crio-e2de3aa80d655d3da850431cddaef11eddd2f14d88317e32362829f29295aff5 WatchSource:0}: Error finding container e2de3aa80d655d3da850431cddaef11eddd2f14d88317e32362829f29295aff5: Status 404 returned error can't find the container with id e2de3aa80d655d3da850431cddaef11eddd2f14d88317e32362829f29295aff5 Nov 02 06:42:45.715510 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:42:45.715481 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e5502564-65855c7c98-q8rn7" event={"ID":"9edf52d8-d09b-40c8-84df-32020447afd2","Type":"ContainerStarted","Data":"e2de3aa80d655d3da850431cddaef11eddd2f14d88317e32362829f29295aff5"} Nov 02 06:42:46.049520 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:42:46.049430 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a40f9931-lfvwh"] Nov 02 06:42:46.099203 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:42:46.099174 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a40f9931-lfvwh"] Nov 02 06:42:47.736312 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:42:47.736276 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e5502564-65855c7c98-q8rn7" event={"ID":"9edf52d8-d09b-40c8-84df-32020447afd2","Type":"ContainerStarted","Data":"0afbcc0dd88b8237c95dd526254ab33d56af99d0e678228800ddff5a8579fd71"} Nov 02 06:42:47.736791 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:42:47.736365 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e5502564-65855c7c98-q8rn7" Nov 02 06:42:47.845866 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:42:47.845826 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d25304c-6148-45bc-8c2b-d0e8b894a02f" path="/var/lib/kubelet/pods/7d25304c-6148-45bc-8c2b-d0e8b894a02f/volumes" Nov 02 06:42:47.854889 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:42:47.854843 5916 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e5502564-65855c7c98-q8rn7" podStartSLOduration=2.883702887 podStartE2EDuration="3.854830731s" totalImagesPullingTime="971.127844ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:42:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:42:47.851758891 +0000 UTC m=+1556.123579284" watchObservedRunningTime="2026-11-02 06:42:47.854830731 +0000 UTC m=+1556.126651124" Nov 02 06:42:49.897204 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:42:49.897170 5916 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p98cq" containerName="extractor" Nov 02 06:42:53.750219 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:42:53.750192 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e5502564-65855c7c98-q8rn7" Nov 02 06:42:53.762237 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:42:53.762209 5916 scope.go:122] "RemoveContainer" containerID="71bfeb7ce9a65b585643e1c948c075d2f7674d61d3a33856a176061345d85164" Nov 02 06:42:53.775183 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:42:53.775157 5916 scope.go:122] "RemoveContainer" containerID="581a47bfcc87ad237583822cdb5786c083da41991a20596ec700f4c3e68fccc0" Nov 02 06:42:53.785192 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:42:53.785168 5916 scope.go:122] "RemoveContainer" containerID="28fa798fb1562c22d16eb5a93a232befd6a3e42dac3cdc5cfe144b58eff42ded" Nov 02 06:42:53.798405 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:42:53.798383 5916 scope.go:122] "RemoveContainer" containerID="4c494a6cb9eae74e1c14a0e9d6e5ae9e4c9571384b881727c76f9397b5c815af" Nov 02 06:42:53.810089 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:42:53.810060 5916 scope.go:122] "RemoveContainer" containerID="901080ed9502e689e4599eb9bc470b13f3e9df7944498af9251b998aa617cdd9" Nov 02 06:42:53.819796 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:42:53.819772 5916 scope.go:122] "RemoveContainer" containerID="22a25b082795494c52451bcf2b9604520fb23aea4f936d6e8a94236d2bcbb261" Nov 02 06:42:53.831687 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:42:53.831666 5916 scope.go:122] "RemoveContainer" containerID="497801801e4f758868dce5b928ae62a470511d8345a8d58fed4437426f964286" Nov 02 06:42:53.845233 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:42:53.845088 5916 scope.go:122] "RemoveContainer" containerID="48adf154072ce3abd5c2604be874752bcae88be693172a8d2528b6c30ff594d1" Nov 02 06:42:57.056884 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:42:57.056848 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8e54376-fmfs7"] Nov 02 06:42:57.104189 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:42:57.104155 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8e54376-fmfs7"] Nov 02 06:42:57.847509 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:42:57.847468 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="539c3a80-fe0d-43b4-8d33-6d53597edd64" path="/var/lib/kubelet/pods/539c3a80-fe0d-43b4-8d33-6d53597edd64/volumes" Nov 02 06:43:11.813680 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:43:11.813649 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-08bbf9-openshift-default-5479f6b7c-d2hk2"] Nov 02 06:43:11.818201 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:43:11.818174 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-08bbf9-openshift-default-5479f6b7c-d2hk2" Nov 02 06:43:11.877765 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:43:11.877239 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-08bbf9-openshift-default-5479f6b7c-d2hk2"] Nov 02 06:43:11.920011 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:43:11.919987 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-08bbf9-openshift-default-dockercfg-6mfl7\"" Nov 02 06:43:11.984198 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:43:11.984157 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b46fdb7-008c-45d7-92cb-8e392c54547f-istio-token\") pod \"e2e-shared-a-w2-08bbf9-openshift-default-5479f6b7c-d2hk2\" (UID: \"1b46fdb7-008c-45d7-92cb-8e392c54547f\") " pod="openshift-ingress/e2e-shared-a-w2-08bbf9-openshift-default-5479f6b7c-d2hk2" Nov 02 06:43:11.984399 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:43:11.984218 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhgzk\" (UniqueName: \"kubernetes.io/projected/1b46fdb7-008c-45d7-92cb-8e392c54547f-kube-api-access-mhgzk\") pod \"e2e-shared-a-w2-08bbf9-openshift-default-5479f6b7c-d2hk2\" (UID: \"1b46fdb7-008c-45d7-92cb-8e392c54547f\") " pod="openshift-ingress/e2e-shared-a-w2-08bbf9-openshift-default-5479f6b7c-d2hk2" Nov 02 06:43:11.984399 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:43:11.984374 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b46fdb7-008c-45d7-92cb-8e392c54547f-workload-socket\") pod \"e2e-shared-a-w2-08bbf9-openshift-default-5479f6b7c-d2hk2\" (UID: \"1b46fdb7-008c-45d7-92cb-8e392c54547f\") " pod="openshift-ingress/e2e-shared-a-w2-08bbf9-openshift-default-5479f6b7c-d2hk2" Nov 02 06:43:11.984530 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:43:11.984441 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b46fdb7-008c-45d7-92cb-8e392c54547f-istio-data\") pod \"e2e-shared-a-w2-08bbf9-openshift-default-5479f6b7c-d2hk2\" (UID: \"1b46fdb7-008c-45d7-92cb-8e392c54547f\") " pod="openshift-ingress/e2e-shared-a-w2-08bbf9-openshift-default-5479f6b7c-d2hk2" Nov 02 06:43:11.984585 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:43:11.984553 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b46fdb7-008c-45d7-92cb-8e392c54547f-credential-socket\") pod \"e2e-shared-a-w2-08bbf9-openshift-default-5479f6b7c-d2hk2\" (UID: \"1b46fdb7-008c-45d7-92cb-8e392c54547f\") " pod="openshift-ingress/e2e-shared-a-w2-08bbf9-openshift-default-5479f6b7c-d2hk2" Nov 02 06:43:11.984638 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:43:11.984598 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b46fdb7-008c-45d7-92cb-8e392c54547f-istiod-ca-cert\") pod \"e2e-shared-a-w2-08bbf9-openshift-default-5479f6b7c-d2hk2\" (UID: \"1b46fdb7-008c-45d7-92cb-8e392c54547f\") " pod="openshift-ingress/e2e-shared-a-w2-08bbf9-openshift-default-5479f6b7c-d2hk2" Nov 02 06:43:11.984735 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:43:11.984715 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b46fdb7-008c-45d7-92cb-8e392c54547f-istio-podinfo\") pod \"e2e-shared-a-w2-08bbf9-openshift-default-5479f6b7c-d2hk2\" (UID: \"1b46fdb7-008c-45d7-92cb-8e392c54547f\") " pod="openshift-ingress/e2e-shared-a-w2-08bbf9-openshift-default-5479f6b7c-d2hk2" Nov 02 06:43:11.984811 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:43:11.984751 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b46fdb7-008c-45d7-92cb-8e392c54547f-workload-certs\") pod \"e2e-shared-a-w2-08bbf9-openshift-default-5479f6b7c-d2hk2\" (UID: \"1b46fdb7-008c-45d7-92cb-8e392c54547f\") " pod="openshift-ingress/e2e-shared-a-w2-08bbf9-openshift-default-5479f6b7c-d2hk2" Nov 02 06:43:11.984891 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:43:11.984874 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b46fdb7-008c-45d7-92cb-8e392c54547f-istio-envoy\") pod \"e2e-shared-a-w2-08bbf9-openshift-default-5479f6b7c-d2hk2\" (UID: \"1b46fdb7-008c-45d7-92cb-8e392c54547f\") " pod="openshift-ingress/e2e-shared-a-w2-08bbf9-openshift-default-5479f6b7c-d2hk2" Nov 02 06:43:12.089030 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:43:12.087039 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b46fdb7-008c-45d7-92cb-8e392c54547f-istio-token\") pod \"e2e-shared-a-w2-08bbf9-openshift-default-5479f6b7c-d2hk2\" (UID: \"1b46fdb7-008c-45d7-92cb-8e392c54547f\") " pod="openshift-ingress/e2e-shared-a-w2-08bbf9-openshift-default-5479f6b7c-d2hk2" Nov 02 06:43:12.089030 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:43:12.087101 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhgzk\" (UniqueName: \"kubernetes.io/projected/1b46fdb7-008c-45d7-92cb-8e392c54547f-kube-api-access-mhgzk\") pod \"e2e-shared-a-w2-08bbf9-openshift-default-5479f6b7c-d2hk2\" (UID: \"1b46fdb7-008c-45d7-92cb-8e392c54547f\") " pod="openshift-ingress/e2e-shared-a-w2-08bbf9-openshift-default-5479f6b7c-d2hk2" Nov 02 06:43:12.089030 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:43:12.087197 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b46fdb7-008c-45d7-92cb-8e392c54547f-workload-socket\") pod \"e2e-shared-a-w2-08bbf9-openshift-default-5479f6b7c-d2hk2\" (UID: \"1b46fdb7-008c-45d7-92cb-8e392c54547f\") " pod="openshift-ingress/e2e-shared-a-w2-08bbf9-openshift-default-5479f6b7c-d2hk2" Nov 02 06:43:12.089030 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:43:12.087253 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b46fdb7-008c-45d7-92cb-8e392c54547f-istio-data\") pod \"e2e-shared-a-w2-08bbf9-openshift-default-5479f6b7c-d2hk2\" (UID: \"1b46fdb7-008c-45d7-92cb-8e392c54547f\") " pod="openshift-ingress/e2e-shared-a-w2-08bbf9-openshift-default-5479f6b7c-d2hk2" Nov 02 06:43:12.089030 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:43:12.087319 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b46fdb7-008c-45d7-92cb-8e392c54547f-credential-socket\") pod \"e2e-shared-a-w2-08bbf9-openshift-default-5479f6b7c-d2hk2\" (UID: \"1b46fdb7-008c-45d7-92cb-8e392c54547f\") " pod="openshift-ingress/e2e-shared-a-w2-08bbf9-openshift-default-5479f6b7c-d2hk2" Nov 02 06:43:12.089030 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:43:12.087362 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b46fdb7-008c-45d7-92cb-8e392c54547f-istiod-ca-cert\") pod \"e2e-shared-a-w2-08bbf9-openshift-default-5479f6b7c-d2hk2\" (UID: \"1b46fdb7-008c-45d7-92cb-8e392c54547f\") " pod="openshift-ingress/e2e-shared-a-w2-08bbf9-openshift-default-5479f6b7c-d2hk2" Nov 02 06:43:12.089030 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:43:12.087429 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b46fdb7-008c-45d7-92cb-8e392c54547f-istio-podinfo\") pod \"e2e-shared-a-w2-08bbf9-openshift-default-5479f6b7c-d2hk2\" (UID: \"1b46fdb7-008c-45d7-92cb-8e392c54547f\") " pod="openshift-ingress/e2e-shared-a-w2-08bbf9-openshift-default-5479f6b7c-d2hk2" Nov 02 06:43:12.089030 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:43:12.087458 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b46fdb7-008c-45d7-92cb-8e392c54547f-workload-certs\") pod \"e2e-shared-a-w2-08bbf9-openshift-default-5479f6b7c-d2hk2\" (UID: \"1b46fdb7-008c-45d7-92cb-8e392c54547f\") " pod="openshift-ingress/e2e-shared-a-w2-08bbf9-openshift-default-5479f6b7c-d2hk2" Nov 02 06:43:12.089030 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:43:12.087535 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b46fdb7-008c-45d7-92cb-8e392c54547f-istio-envoy\") pod \"e2e-shared-a-w2-08bbf9-openshift-default-5479f6b7c-d2hk2\" (UID: \"1b46fdb7-008c-45d7-92cb-8e392c54547f\") " pod="openshift-ingress/e2e-shared-a-w2-08bbf9-openshift-default-5479f6b7c-d2hk2" Nov 02 06:43:12.089030 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:43:12.088371 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b46fdb7-008c-45d7-92cb-8e392c54547f-istio-data\") pod \"e2e-shared-a-w2-08bbf9-openshift-default-5479f6b7c-d2hk2\" (UID: \"1b46fdb7-008c-45d7-92cb-8e392c54547f\") " pod="openshift-ingress/e2e-shared-a-w2-08bbf9-openshift-default-5479f6b7c-d2hk2" Nov 02 06:43:12.089671 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:43:12.089105 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b46fdb7-008c-45d7-92cb-8e392c54547f-workload-socket\") pod \"e2e-shared-a-w2-08bbf9-openshift-default-5479f6b7c-d2hk2\" (UID: \"1b46fdb7-008c-45d7-92cb-8e392c54547f\") " pod="openshift-ingress/e2e-shared-a-w2-08bbf9-openshift-default-5479f6b7c-d2hk2" Nov 02 06:43:12.089732 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:43:12.089715 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b46fdb7-008c-45d7-92cb-8e392c54547f-istiod-ca-cert\") pod \"e2e-shared-a-w2-08bbf9-openshift-default-5479f6b7c-d2hk2\" (UID: \"1b46fdb7-008c-45d7-92cb-8e392c54547f\") " pod="openshift-ingress/e2e-shared-a-w2-08bbf9-openshift-default-5479f6b7c-d2hk2" Nov 02 06:43:12.097681 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:43:12.094360 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b46fdb7-008c-45d7-92cb-8e392c54547f-istio-envoy\") pod \"e2e-shared-a-w2-08bbf9-openshift-default-5479f6b7c-d2hk2\" (UID: \"1b46fdb7-008c-45d7-92cb-8e392c54547f\") " pod="openshift-ingress/e2e-shared-a-w2-08bbf9-openshift-default-5479f6b7c-d2hk2" Nov 02 06:43:12.097681 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:43:12.095126 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b46fdb7-008c-45d7-92cb-8e392c54547f-workload-certs\") pod \"e2e-shared-a-w2-08bbf9-openshift-default-5479f6b7c-d2hk2\" (UID: \"1b46fdb7-008c-45d7-92cb-8e392c54547f\") " pod="openshift-ingress/e2e-shared-a-w2-08bbf9-openshift-default-5479f6b7c-d2hk2" Nov 02 06:43:12.097681 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:43:12.095841 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b46fdb7-008c-45d7-92cb-8e392c54547f-credential-socket\") pod \"e2e-shared-a-w2-08bbf9-openshift-default-5479f6b7c-d2hk2\" (UID: \"1b46fdb7-008c-45d7-92cb-8e392c54547f\") " pod="openshift-ingress/e2e-shared-a-w2-08bbf9-openshift-default-5479f6b7c-d2hk2" Nov 02 06:43:12.102182 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:43:12.102154 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b46fdb7-008c-45d7-92cb-8e392c54547f-istio-podinfo\") pod \"e2e-shared-a-w2-08bbf9-openshift-default-5479f6b7c-d2hk2\" (UID: \"1b46fdb7-008c-45d7-92cb-8e392c54547f\") " pod="openshift-ingress/e2e-shared-a-w2-08bbf9-openshift-default-5479f6b7c-d2hk2" Nov 02 06:43:12.159952 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:43:12.150358 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b46fdb7-008c-45d7-92cb-8e392c54547f-istio-token\") pod \"e2e-shared-a-w2-08bbf9-openshift-default-5479f6b7c-d2hk2\" (UID: \"1b46fdb7-008c-45d7-92cb-8e392c54547f\") " pod="openshift-ingress/e2e-shared-a-w2-08bbf9-openshift-default-5479f6b7c-d2hk2" Nov 02 06:43:12.159952 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:43:12.150840 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhgzk\" (UniqueName: \"kubernetes.io/projected/1b46fdb7-008c-45d7-92cb-8e392c54547f-kube-api-access-mhgzk\") pod \"e2e-shared-a-w2-08bbf9-openshift-default-5479f6b7c-d2hk2\" (UID: \"1b46fdb7-008c-45d7-92cb-8e392c54547f\") " pod="openshift-ingress/e2e-shared-a-w2-08bbf9-openshift-default-5479f6b7c-d2hk2" Nov 02 06:43:12.434987 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:43:12.434923 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-08bbf9-openshift-default-5479f6b7c-d2hk2" Nov 02 06:43:12.886689 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:43:12.886659 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-08bbf9-openshift-default-5479f6b7c-d2hk2"] Nov 02 06:43:12.888112 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:43:12.888082 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b46fdb7_008c_45d7_92cb_8e392c54547f.slice/crio-a3259bcc6ce1bd28e3ac14a8fed8ab0b4368c10b1f94ad750a5eb2339221ca61 WatchSource:0}: Error finding container a3259bcc6ce1bd28e3ac14a8fed8ab0b4368c10b1f94ad750a5eb2339221ca61: Status 404 returned error can't find the container with id a3259bcc6ce1bd28e3ac14a8fed8ab0b4368c10b1f94ad750a5eb2339221ca61 Nov 02 06:43:12.890685 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:43:12.890655 5916 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 06:43:12.890769 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:43:12.890717 5916 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 06:43:12.890769 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:43:12.890747 5916 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 06:43:13.037257 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:43:13.037225 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-08bbf9-openshift-default-5479f6b7c-d2hk2" event={"ID":"1b46fdb7-008c-45d7-92cb-8e392c54547f","Type":"ContainerStarted","Data":"ef905e16b3a5fd3ea80cf5a33b5275a7425d95738fb1e82d4afbd99a49491dfd"} Nov 02 06:43:13.037431 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:43:13.037274 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-08bbf9-openshift-default-5479f6b7c-d2hk2" event={"ID":"1b46fdb7-008c-45d7-92cb-8e392c54547f","Type":"ContainerStarted","Data":"a3259bcc6ce1bd28e3ac14a8fed8ab0b4368c10b1f94ad750a5eb2339221ca61"} Nov 02 06:43:13.062974 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:43:13.057346 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7de0cfea-dcgfp"] Nov 02 06:43:13.062974 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:43:13.061699 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7de0cfea-dcgfp"] Nov 02 06:43:13.164629 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:43:13.163560 5916 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-08bbf9-openshift-default-5479f6b7c-d2hk2" podStartSLOduration=2.163541865 podStartE2EDuration="2.163541865s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:43:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:43:13.155548433 +0000 UTC m=+1581.427368829" watchObservedRunningTime="2026-11-02 06:43:13.163541865 +0000 UTC m=+1581.435362261" Nov 02 06:43:13.435982 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:43:13.435824 5916 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-08bbf9-openshift-default-5479f6b7c-d2hk2" Nov 02 06:43:13.439925 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:43:13.437445 5916 patch_prober.go:29] interesting pod/e2e-shared-a-w2-08bbf9-openshift-default-5479f6b7c-d2hk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Nov 02 06:43:13.439925 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:43:13.437511 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-08bbf9-openshift-default-5479f6b7c-d2hk2" podUID="1b46fdb7-008c-45d7-92cb-8e392c54547f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Nov 02 06:43:13.846609 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:43:13.846520 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="851e0293-99c4-4ed8-a9ce-7bff1204e30e" path="/var/lib/kubelet/pods/851e0293-99c4-4ed8-a9ce-7bff1204e30e/volumes" Nov 02 06:43:14.441092 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:43:14.441055 5916 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-08bbf9-openshift-default-5479f6b7c-d2hk2" Nov 02 06:43:15.058969 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:43:15.058938 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-08bbf9-openshift-default-5479f6b7c-d2hk2" Nov 02 06:43:15.060461 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:43:15.060431 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-08bbf9-openshift-default-5479f6b7c-d2hk2" Nov 02 06:43:19.085013 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:43:19.084974 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98e3f2eb-9rds2"] Nov 02 06:43:19.199082 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:43:19.199045 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98e3f2eb-9rds2"] Nov 02 06:43:19.849398 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:43:19.849361 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0b8ef83-4677-405d-be44-4351cca0f47f" path="/var/lib/kubelet/pods/f0b8ef83-4677-405d-be44-4351cca0f47f/volumes" Nov 02 06:43:19.904477 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:43:19.904441 5916 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p98cq" containerName="extractor" Nov 02 06:43:49.685591 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:43:49.685519 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-br7cd"] Nov 02 06:43:49.691699 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:43:49.687146 5916 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-br7cd" podUID="2f9273e5-b930-47f0-b710-234974ad9025" containerName="istio-proxy" containerID="cri-o://d1262d9a4b500b35cab1feee8b0155366102b0d2bd27772791f5ee3824a45811" gracePeriod=30 Nov 02 06:43:49.898713 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:43:49.898655 5916 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p98cq" containerName="extractor" Nov 02 06:43:53.992928 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:43:53.989788 5916 scope.go:122] "RemoveContainer" containerID="b77e455e57f5140e7beb47f436ffafa57a91a986d08b5a46ce04a85ed38ac1eb" Nov 02 06:43:54.108130 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:43:54.107406 5916 scope.go:122] "RemoveContainer" containerID="134f5f98e3419367bf5bc641c64ad06b61742cd313094b4a3c379882fd222bfd" Nov 02 06:43:54.196286 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:43:54.190315 5916 scope.go:122] "RemoveContainer" containerID="4c99d2608b531894bed14e9738827afcfaae33740a6e2061412ca7108625f1f3" Nov 02 06:43:54.207149 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:43:54.207039 5916 scope.go:122] "RemoveContainer" containerID="1cec20cbab42866ab6af84b4ed85481b7f7237cb0698d5aee8e4ffa7e0935f10" Nov 02 06:43:54.222481 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:43:54.222459 5916 scope.go:122] "RemoveContainer" containerID="07acc4b8b08df889f973d4cd62a74b6069a3402a875c65052ae14d6763476a4e" Nov 02 06:43:54.248319 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:43:54.246646 5916 scope.go:122] "RemoveContainer" containerID="01c9211a8423c4ccdb6f02ad23124fc0d3f2b6f959cf6e4034055ee436191301" Nov 02 06:43:54.269713 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:43:54.268560 5916 scope.go:122] "RemoveContainer" containerID="99ac39e5536f07fb0b8085bd565fb219aedb0aa6b9be09a387fa736e0a56b852" Nov 02 06:43:54.281935 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:43:54.281686 5916 scope.go:122] "RemoveContainer" containerID="c6fe07e6465bfbbff5b5d2d9a7ebc012cfac082697c13934b3547d024e411925" Nov 02 06:43:54.292559 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:43:54.292539 5916 scope.go:122] "RemoveContainer" containerID="cf59f9897a09cc43f9bda887f20860ec32774e23ec3bfa1fd0e72405ad794d57" Nov 02 06:43:54.302076 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:43:54.302054 5916 scope.go:122] "RemoveContainer" containerID="14eccec5a556da217ed081f8a420c720be6b5bf11ba748ee7f60b10c192f98d6" Nov 02 06:43:54.313580 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:43:54.313563 5916 scope.go:122] "RemoveContainer" containerID="26361546f5c3528be1aba8913e67eafa96d258bcdff02495f89c16169f23628e" Nov 02 06:43:54.322900 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:43:54.322879 5916 scope.go:122] "RemoveContainer" containerID="1b626993ae088eed6c2d0befb2c49e83b403221684b69bc4f322f3f478cbd588" Nov 02 06:43:55.098901 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:43:55.098877 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-br7cd" Nov 02 06:43:55.176837 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:43:55.176804 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f9273e5-b930-47f0-b710-234974ad9025-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f9273e5-b930-47f0-b710-234974ad9025-workload-socket\") pod \"2f9273e5-b930-47f0-b710-234974ad9025\" (UID: \"2f9273e5-b930-47f0-b710-234974ad9025\") " Nov 02 06:43:55.177034 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:43:55.176861 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f9273e5-b930-47f0-b710-234974ad9025-istio-token\" (UniqueName: \"kubernetes.io/projected/2f9273e5-b930-47f0-b710-234974ad9025-istio-token\") pod \"2f9273e5-b930-47f0-b710-234974ad9025\" (UID: \"2f9273e5-b930-47f0-b710-234974ad9025\") " Nov 02 06:43:55.177034 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:43:55.176953 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f9273e5-b930-47f0-b710-234974ad9025-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f9273e5-b930-47f0-b710-234974ad9025-istio-data\") pod \"2f9273e5-b930-47f0-b710-234974ad9025\" (UID: \"2f9273e5-b930-47f0-b710-234974ad9025\") " Nov 02 06:43:55.177034 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:43:55.176993 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2f9273e5-b930-47f0-b710-234974ad9025-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f9273e5-b930-47f0-b710-234974ad9025-istio-podinfo\") pod \"2f9273e5-b930-47f0-b710-234974ad9025\" (UID: \"2f9273e5-b930-47f0-b710-234974ad9025\") " Nov 02 06:43:55.177206 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:43:55.177107 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f9273e5-b930-47f0-b710-234974ad9025-workload-socket" pod "2f9273e5-b930-47f0-b710-234974ad9025" (UID: "2f9273e5-b930-47f0-b710-234974ad9025"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:43:55.177262 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:43:55.177206 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f9273e5-b930-47f0-b710-234974ad9025-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f9273e5-b930-47f0-b710-234974ad9025-workload-certs\") pod \"2f9273e5-b930-47f0-b710-234974ad9025\" (UID: \"2f9273e5-b930-47f0-b710-234974ad9025\") " Nov 02 06:43:55.177325 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:43:55.177295 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f9273e5-b930-47f0-b710-234974ad9025-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f9273e5-b930-47f0-b710-234974ad9025-istio-envoy\") pod \"2f9273e5-b930-47f0-b710-234974ad9025\" (UID: \"2f9273e5-b930-47f0-b710-234974ad9025\") " Nov 02 06:43:55.177379 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:43:55.177352 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f9273e5-b930-47f0-b710-234974ad9025-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f9273e5-b930-47f0-b710-234974ad9025-istiod-ca-cert\") pod \"2f9273e5-b930-47f0-b710-234974ad9025\" (UID: \"2f9273e5-b930-47f0-b710-234974ad9025\") " Nov 02 06:43:55.177431 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:43:55.177386 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f9273e5-b930-47f0-b710-234974ad9025-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f9273e5-b930-47f0-b710-234974ad9025-credential-socket\") pod \"2f9273e5-b930-47f0-b710-234974ad9025\" (UID: \"2f9273e5-b930-47f0-b710-234974ad9025\") " Nov 02 06:43:55.177486 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:43:55.177414 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f9273e5-b930-47f0-b710-234974ad9025-kube-api-access-hcxvj\" (UniqueName: \"kubernetes.io/projected/2f9273e5-b930-47f0-b710-234974ad9025-kube-api-access-hcxvj\") pod \"2f9273e5-b930-47f0-b710-234974ad9025\" (UID: \"2f9273e5-b930-47f0-b710-234974ad9025\") " Nov 02 06:43:55.177715 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:43:55.177693 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f9273e5-b930-47f0-b710-234974ad9025-istiod-ca-cert" pod "2f9273e5-b930-47f0-b710-234974ad9025" (UID: "2f9273e5-b930-47f0-b710-234974ad9025"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:43:55.178008 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:43:55.177981 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f9273e5-b930-47f0-b710-234974ad9025-istio-data" pod "2f9273e5-b930-47f0-b710-234974ad9025" (UID: "2f9273e5-b930-47f0-b710-234974ad9025"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:43:55.178141 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:43:55.178080 5916 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f9273e5-b930-47f0-b710-234974ad9025-istiod-ca-cert\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:55.178141 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:43:55.178093 5916 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f9273e5-b930-47f0-b710-234974ad9025-workload-socket\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:55.178259 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:43:55.178147 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f9273e5-b930-47f0-b710-234974ad9025-workload-certs" pod "2f9273e5-b930-47f0-b710-234974ad9025" (UID: "2f9273e5-b930-47f0-b710-234974ad9025"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:43:55.178259 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:43:55.178189 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f9273e5-b930-47f0-b710-234974ad9025-credential-socket" pod "2f9273e5-b930-47f0-b710-234974ad9025" (UID: "2f9273e5-b930-47f0-b710-234974ad9025"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:43:55.179797 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:43:55.179776 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f9273e5-b930-47f0-b710-234974ad9025-istio-token" pod "2f9273e5-b930-47f0-b710-234974ad9025" (UID: "2f9273e5-b930-47f0-b710-234974ad9025"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:43:55.180067 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:43:55.180048 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2f9273e5-b930-47f0-b710-234974ad9025-istio-podinfo" pod "2f9273e5-b930-47f0-b710-234974ad9025" (UID: "2f9273e5-b930-47f0-b710-234974ad9025"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 06:43:55.180147 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:43:55.180097 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f9273e5-b930-47f0-b710-234974ad9025-kube-api-access-hcxvj" pod "2f9273e5-b930-47f0-b710-234974ad9025" (UID: "2f9273e5-b930-47f0-b710-234974ad9025"). InnerVolumeSpecName "kube-api-access-hcxvj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:43:55.180359 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:43:55.180342 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f9273e5-b930-47f0-b710-234974ad9025-istio-envoy" pod "2f9273e5-b930-47f0-b710-234974ad9025" (UID: "2f9273e5-b930-47f0-b710-234974ad9025"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:43:55.279846 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:43:55.279814 5916 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f9273e5-b930-47f0-b710-234974ad9025-istio-data\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:55.279846 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:43:55.279845 5916 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f9273e5-b930-47f0-b710-234974ad9025-istio-podinfo\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:55.279846 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:43:55.279854 5916 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f9273e5-b930-47f0-b710-234974ad9025-workload-certs\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:55.280157 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:43:55.279864 5916 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f9273e5-b930-47f0-b710-234974ad9025-istio-envoy\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:55.280157 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:43:55.279873 5916 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f9273e5-b930-47f0-b710-234974ad9025-credential-socket\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:55.280157 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:43:55.279882 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hcxvj\" (UniqueName: \"kubernetes.io/projected/2f9273e5-b930-47f0-b710-234974ad9025-kube-api-access-hcxvj\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:55.280157 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:43:55.279892 5916 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f9273e5-b930-47f0-b710-234974ad9025-istio-token\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:55.741518 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:43:55.741480 5916 generic.go:356] "Generic (PLEG): container finished" podID="2f9273e5-b930-47f0-b710-234974ad9025" containerID="d1262d9a4b500b35cab1feee8b0155366102b0d2bd27772791f5ee3824a45811" exitCode=0 Nov 02 06:43:55.741739 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:43:55.741541 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-br7cd" Nov 02 06:43:55.741739 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:43:55.741570 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-br7cd" event={"ID":"2f9273e5-b930-47f0-b710-234974ad9025","Type":"ContainerDied","Data":"d1262d9a4b500b35cab1feee8b0155366102b0d2bd27772791f5ee3824a45811"} Nov 02 06:43:55.741739 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:43:55.741627 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-br7cd" event={"ID":"2f9273e5-b930-47f0-b710-234974ad9025","Type":"ContainerDied","Data":"52835324304d8b359c2ffc8a55510eac0121d083d7ba878086125635846c2ee6"} Nov 02 06:43:55.741739 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:43:55.741656 5916 scope.go:122] "RemoveContainer" containerID="d1262d9a4b500b35cab1feee8b0155366102b0d2bd27772791f5ee3824a45811" Nov 02 06:43:55.756099 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:43:55.756079 5916 scope.go:122] "RemoveContainer" containerID="d1262d9a4b500b35cab1feee8b0155366102b0d2bd27772791f5ee3824a45811" Nov 02 06:43:55.756445 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:43:55.756414 5916 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1262d9a4b500b35cab1feee8b0155366102b0d2bd27772791f5ee3824a45811\": container with ID starting with d1262d9a4b500b35cab1feee8b0155366102b0d2bd27772791f5ee3824a45811 not found: ID does not exist" containerID="d1262d9a4b500b35cab1feee8b0155366102b0d2bd27772791f5ee3824a45811" Nov 02 06:43:55.756530 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:43:55.756453 5916 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1262d9a4b500b35cab1feee8b0155366102b0d2bd27772791f5ee3824a45811"} err="failed to get container status \"d1262d9a4b500b35cab1feee8b0155366102b0d2bd27772791f5ee3824a45811\": rpc error: code = NotFound desc = could not find container \"d1262d9a4b500b35cab1feee8b0155366102b0d2bd27772791f5ee3824a45811\": container with ID starting with d1262d9a4b500b35cab1feee8b0155366102b0d2bd27772791f5ee3824a45811 not found: ID does not exist" Nov 02 06:43:55.971957 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:43:55.971887 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-br7cd"] Nov 02 06:43:56.017454 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:43:56.017374 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-br7cd"] Nov 02 06:43:57.848476 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:43:57.848439 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f9273e5-b930-47f0-b710-234974ad9025" path="/var/lib/kubelet/pods/2f9273e5-b930-47f0-b710-234974ad9025/volumes" Nov 02 06:43:58.770784 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:43:58.770753 5916 generic.go:356] "Generic (PLEG): container finished" podID="5f971e93-8d12-41cf-945c-c932fee7dc16" containerID="35301f49b485c12f73f63e76bade89ff9a832ba49b644c6b94ad4b35f00b0610" exitCode=0 Nov 02 06:43:58.771027 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:43:58.770831 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-29abe7-openshift-default-555ff8c7fc-6zdzz" event={"ID":"5f971e93-8d12-41cf-945c-c932fee7dc16","Type":"ContainerDied","Data":"35301f49b485c12f73f63e76bade89ff9a832ba49b644c6b94ad4b35f00b0610"} Nov 02 06:43:58.771375 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:43:58.771360 5916 scope.go:122] "RemoveContainer" containerID="35301f49b485c12f73f63e76bade89ff9a832ba49b644c6b94ad4b35f00b0610" Nov 02 06:43:58.773442 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:43:58.773413 5916 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 06:43:58.773529 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:43:58.773501 5916 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 06:43:58.773573 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:43:58.773544 5916 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 06:43:59.781082 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:43:59.781052 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-29abe7-openshift-default-555ff8c7fc-6zdzz" event={"ID":"5f971e93-8d12-41cf-945c-c932fee7dc16","Type":"ContainerStarted","Data":"d7c62ec1ad79cf10ec72e4844e86837466e9d255c1d73289982959ef00790f23"} Nov 02 06:44:00.147940 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:44:00.145259 5916 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-29abe7-openshift-default-555ff8c7fc-6zdzz" Nov 02 06:44:00.147940 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:44:00.145418 5916 patch_prober.go:29] interesting pod/e2e-shared-b-w0-29abe7-openshift-default-555ff8c7fc-6zdzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body= Nov 02 06:44:00.147940 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:44:00.145461 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-29abe7-openshift-default-555ff8c7fc-6zdzz" podUID="5f971e93-8d12-41cf-945c-c932fee7dc16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" Nov 02 06:44:01.142980 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:44:01.142219 5916 patch_prober.go:29] interesting pod/e2e-shared-b-w0-29abe7-openshift-default-555ff8c7fc-6zdzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body= Nov 02 06:44:01.142980 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:44:01.142277 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-29abe7-openshift-default-555ff8c7fc-6zdzz" podUID="5f971e93-8d12-41cf-945c-c932fee7dc16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" Nov 02 06:44:02.164927 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:44:02.164067 5916 patch_prober.go:29] interesting pod/e2e-shared-b-w0-29abe7-openshift-default-555ff8c7fc-6zdzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body= Nov 02 06:44:02.164927 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:44:02.164157 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-29abe7-openshift-default-555ff8c7fc-6zdzz" podUID="5f971e93-8d12-41cf-945c-c932fee7dc16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" Nov 02 06:44:03.134930 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:44:03.129241 5916 patch_prober.go:29] interesting pod/e2e-shared-b-w0-29abe7-openshift-default-555ff8c7fc-6zdzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body= Nov 02 06:44:03.134930 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:44:03.129309 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-29abe7-openshift-default-555ff8c7fc-6zdzz" podUID="5f971e93-8d12-41cf-945c-c932fee7dc16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" Nov 02 06:44:04.130353 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:44:04.130318 5916 patch_prober.go:29] interesting pod/e2e-shared-b-w0-29abe7-openshift-default-555ff8c7fc-6zdzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body= Nov 02 06:44:04.130993 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:44:04.130961 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-29abe7-openshift-default-555ff8c7fc-6zdzz" podUID="5f971e93-8d12-41cf-945c-c932fee7dc16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" Nov 02 06:44:05.130932 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:44:05.129260 5916 patch_prober.go:29] interesting pod/e2e-shared-b-w0-29abe7-openshift-default-555ff8c7fc-6zdzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body= Nov 02 06:44:05.130932 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:44:05.129323 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-29abe7-openshift-default-555ff8c7fc-6zdzz" podUID="5f971e93-8d12-41cf-945c-c932fee7dc16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" Nov 02 06:44:05.340312 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:44:05.340279 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-29abe7-openshift-default-555ff8c7fc-6zdzz" Nov 02 06:44:06.133602 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:44:06.133567 5916 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-29abe7-openshift-default-555ff8c7fc-6zdzz" Nov 02 06:44:06.134585 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:44:06.134561 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-29abe7-openshift-default-555ff8c7fc-6zdzz" Nov 02 06:44:19.616614 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:44:19.616576 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-jr76m"] Nov 02 06:44:19.617109 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:44:19.616762 5916 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-jr76m" podUID="20a4dcea-b779-47d9-a38b-5ede52dc0076" containerName="istio-proxy" containerID="cri-o://2c69bf27890cf6e7f0f1459a17e5aa476b117401b7d27701f3e343659e42eeed" gracePeriod=30 Nov 02 06:44:19.670465 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:44:19.670427 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hbwqc"] Nov 02 06:44:19.671231 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:44:19.671178 5916 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hbwqc" podUID="2871e54e-8f8f-47d6-a16b-72d986b3835e" containerName="istio-proxy" containerID="cri-o://97aff99737a873d789e9cf6c6b6d78eb202c7fd93c2df833cafd1dc7497e7005" gracePeriod=30 Nov 02 06:44:19.901471 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:44:19.901438 5916 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p98cq" containerName="extractor" Nov 02 06:44:23.942332 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:44:23.942295 5916 patch_prober.go:29] interesting pod/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-jr76m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 06:44:23.942829 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:44:23.942800 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-jr76m" podUID="20a4dcea-b779-47d9-a38b-5ede52dc0076" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 06:44:24.997063 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:44:24.997037 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-jr76m" Nov 02 06:44:25.021294 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:44:25.021263 5916 generic.go:356] "Generic (PLEG): container finished" podID="20a4dcea-b779-47d9-a38b-5ede52dc0076" containerID="2c69bf27890cf6e7f0f1459a17e5aa476b117401b7d27701f3e343659e42eeed" exitCode=0 Nov 02 06:44:25.021433 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:44:25.021298 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-jr76m" event={"ID":"20a4dcea-b779-47d9-a38b-5ede52dc0076","Type":"ContainerDied","Data":"2c69bf27890cf6e7f0f1459a17e5aa476b117401b7d27701f3e343659e42eeed"} Nov 02 06:44:25.021433 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:44:25.021315 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-jr76m" Nov 02 06:44:25.021433 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:44:25.021377 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-jr76m" event={"ID":"20a4dcea-b779-47d9-a38b-5ede52dc0076","Type":"ContainerDied","Data":"754948647f69d2d90c4453f2afd5676ad923a9379476c4aea74e4de6c19c052c"} Nov 02 06:44:25.021433 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:44:25.021394 5916 scope.go:122] "RemoveContainer" containerID="2c69bf27890cf6e7f0f1459a17e5aa476b117401b7d27701f3e343659e42eeed" Nov 02 06:44:25.023025 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:44:25.022998 5916 generic.go:356] "Generic (PLEG): container finished" podID="2871e54e-8f8f-47d6-a16b-72d986b3835e" containerID="97aff99737a873d789e9cf6c6b6d78eb202c7fd93c2df833cafd1dc7497e7005" exitCode=0 Nov 02 06:44:25.023126 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:44:25.023048 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hbwqc" event={"ID":"2871e54e-8f8f-47d6-a16b-72d986b3835e","Type":"ContainerDied","Data":"97aff99737a873d789e9cf6c6b6d78eb202c7fd93c2df833cafd1dc7497e7005"} Nov 02 06:44:25.023126 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:44:25.023096 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hbwqc" event={"ID":"2871e54e-8f8f-47d6-a16b-72d986b3835e","Type":"ContainerDied","Data":"6606399d4fcd046a8a4a6cc17e36192072421560ec31d6b1a675e108c709c540"} Nov 02 06:44:25.023126 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:44:25.023119 5916 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6606399d4fcd046a8a4a6cc17e36192072421560ec31d6b1a675e108c709c540" Nov 02 06:44:25.028099 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:44:25.028076 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hbwqc" Nov 02 06:44:25.032670 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:44:25.032644 5916 scope.go:122] "RemoveContainer" containerID="2c69bf27890cf6e7f0f1459a17e5aa476b117401b7d27701f3e343659e42eeed" Nov 02 06:44:25.032968 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:44:25.032949 5916 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c69bf27890cf6e7f0f1459a17e5aa476b117401b7d27701f3e343659e42eeed\": container with ID starting with 2c69bf27890cf6e7f0f1459a17e5aa476b117401b7d27701f3e343659e42eeed not found: ID does not exist" containerID="2c69bf27890cf6e7f0f1459a17e5aa476b117401b7d27701f3e343659e42eeed" Nov 02 06:44:25.033093 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:44:25.032980 5916 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c69bf27890cf6e7f0f1459a17e5aa476b117401b7d27701f3e343659e42eeed"} err="failed to get container status \"2c69bf27890cf6e7f0f1459a17e5aa476b117401b7d27701f3e343659e42eeed\": rpc error: code = NotFound desc = could not find container \"2c69bf27890cf6e7f0f1459a17e5aa476b117401b7d27701f3e343659e42eeed\": container with ID starting with 2c69bf27890cf6e7f0f1459a17e5aa476b117401b7d27701f3e343659e42eeed not found: ID does not exist" Nov 02 06:44:25.036052 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:44:25.036029 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/20a4dcea-b779-47d9-a38b-5ede52dc0076-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20a4dcea-b779-47d9-a38b-5ede52dc0076-istio-podinfo\") pod \"20a4dcea-b779-47d9-a38b-5ede52dc0076\" (UID: \"20a4dcea-b779-47d9-a38b-5ede52dc0076\") " Nov 02 06:44:25.036166 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:44:25.036073 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2871e54e-8f8f-47d6-a16b-72d986b3835e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2871e54e-8f8f-47d6-a16b-72d986b3835e-workload-certs\") pod \"2871e54e-8f8f-47d6-a16b-72d986b3835e\" (UID: \"2871e54e-8f8f-47d6-a16b-72d986b3835e\") " Nov 02 06:44:25.036166 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:44:25.036151 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2871e54e-8f8f-47d6-a16b-72d986b3835e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2871e54e-8f8f-47d6-a16b-72d986b3835e-istio-data\") pod \"2871e54e-8f8f-47d6-a16b-72d986b3835e\" (UID: \"2871e54e-8f8f-47d6-a16b-72d986b3835e\") " Nov 02 06:44:25.036277 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:44:25.036191 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20a4dcea-b779-47d9-a38b-5ede52dc0076-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20a4dcea-b779-47d9-a38b-5ede52dc0076-workload-socket\") pod \"20a4dcea-b779-47d9-a38b-5ede52dc0076\" (UID: \"20a4dcea-b779-47d9-a38b-5ede52dc0076\") " Nov 02 06:44:25.036277 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:44:25.036218 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20a4dcea-b779-47d9-a38b-5ede52dc0076-kube-api-access-5d22b\" (UniqueName: \"kubernetes.io/projected/20a4dcea-b779-47d9-a38b-5ede52dc0076-kube-api-access-5d22b\") pod \"20a4dcea-b779-47d9-a38b-5ede52dc0076\" (UID: \"20a4dcea-b779-47d9-a38b-5ede52dc0076\") " Nov 02 06:44:25.036277 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:44:25.036254 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20a4dcea-b779-47d9-a38b-5ede52dc0076-istio-token\" (UniqueName: \"kubernetes.io/projected/20a4dcea-b779-47d9-a38b-5ede52dc0076-istio-token\") pod \"20a4dcea-b779-47d9-a38b-5ede52dc0076\" (UID: \"20a4dcea-b779-47d9-a38b-5ede52dc0076\") " Nov 02 06:44:25.036431 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:44:25.036298 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20a4dcea-b779-47d9-a38b-5ede52dc0076-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20a4dcea-b779-47d9-a38b-5ede52dc0076-istio-envoy\") pod \"20a4dcea-b779-47d9-a38b-5ede52dc0076\" (UID: \"20a4dcea-b779-47d9-a38b-5ede52dc0076\") " Nov 02 06:44:25.036431 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:44:25.036353 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20a4dcea-b779-47d9-a38b-5ede52dc0076-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20a4dcea-b779-47d9-a38b-5ede52dc0076-istiod-ca-cert\") pod \"20a4dcea-b779-47d9-a38b-5ede52dc0076\" (UID: \"20a4dcea-b779-47d9-a38b-5ede52dc0076\") " Nov 02 06:44:25.036431 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:44:25.036379 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2871e54e-8f8f-47d6-a16b-72d986b3835e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2871e54e-8f8f-47d6-a16b-72d986b3835e-workload-socket\") pod \"2871e54e-8f8f-47d6-a16b-72d986b3835e\" (UID: \"2871e54e-8f8f-47d6-a16b-72d986b3835e\") " Nov 02 06:44:25.036431 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:44:25.036414 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2871e54e-8f8f-47d6-a16b-72d986b3835e-istio-token\" (UniqueName: \"kubernetes.io/projected/2871e54e-8f8f-47d6-a16b-72d986b3835e-istio-token\") pod \"2871e54e-8f8f-47d6-a16b-72d986b3835e\" (UID: \"2871e54e-8f8f-47d6-a16b-72d986b3835e\") " Nov 02 06:44:25.036634 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:44:25.036476 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20a4dcea-b779-47d9-a38b-5ede52dc0076-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20a4dcea-b779-47d9-a38b-5ede52dc0076-workload-certs\") pod \"20a4dcea-b779-47d9-a38b-5ede52dc0076\" (UID: \"20a4dcea-b779-47d9-a38b-5ede52dc0076\") " Nov 02 06:44:25.036634 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:44:25.036514 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20a4dcea-b779-47d9-a38b-5ede52dc0076-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20a4dcea-b779-47d9-a38b-5ede52dc0076-istio-data\") pod \"20a4dcea-b779-47d9-a38b-5ede52dc0076\" (UID: \"20a4dcea-b779-47d9-a38b-5ede52dc0076\") " Nov 02 06:44:25.036634 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:44:25.036549 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2871e54e-8f8f-47d6-a16b-72d986b3835e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2871e54e-8f8f-47d6-a16b-72d986b3835e-istio-podinfo\") pod \"2871e54e-8f8f-47d6-a16b-72d986b3835e\" (UID: \"2871e54e-8f8f-47d6-a16b-72d986b3835e\") " Nov 02 06:44:25.036634 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:44:25.036574 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2871e54e-8f8f-47d6-a16b-72d986b3835e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2871e54e-8f8f-47d6-a16b-72d986b3835e-istio-envoy\") pod \"2871e54e-8f8f-47d6-a16b-72d986b3835e\" (UID: \"2871e54e-8f8f-47d6-a16b-72d986b3835e\") " Nov 02 06:44:25.036634 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:44:25.036602 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20a4dcea-b779-47d9-a38b-5ede52dc0076-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20a4dcea-b779-47d9-a38b-5ede52dc0076-credential-socket\") pod \"20a4dcea-b779-47d9-a38b-5ede52dc0076\" (UID: \"20a4dcea-b779-47d9-a38b-5ede52dc0076\") " Nov 02 06:44:25.036878 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:44:25.036713 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2871e54e-8f8f-47d6-a16b-72d986b3835e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2871e54e-8f8f-47d6-a16b-72d986b3835e-istiod-ca-cert\") pod \"2871e54e-8f8f-47d6-a16b-72d986b3835e\" (UID: \"2871e54e-8f8f-47d6-a16b-72d986b3835e\") " Nov 02 06:44:25.036878 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:44:25.036827 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2871e54e-8f8f-47d6-a16b-72d986b3835e-istio-data" pod "2871e54e-8f8f-47d6-a16b-72d986b3835e" (UID: "2871e54e-8f8f-47d6-a16b-72d986b3835e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:44:25.036878 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:44:25.036852 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2871e54e-8f8f-47d6-a16b-72d986b3835e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2871e54e-8f8f-47d6-a16b-72d986b3835e-credential-socket\") pod \"2871e54e-8f8f-47d6-a16b-72d986b3835e\" (UID: \"2871e54e-8f8f-47d6-a16b-72d986b3835e\") " Nov 02 06:44:25.037048 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:44:25.036890 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2871e54e-8f8f-47d6-a16b-72d986b3835e-kube-api-access-cjp57\" (UniqueName: \"kubernetes.io/projected/2871e54e-8f8f-47d6-a16b-72d986b3835e-kube-api-access-cjp57\") pod \"2871e54e-8f8f-47d6-a16b-72d986b3835e\" (UID: \"2871e54e-8f8f-47d6-a16b-72d986b3835e\") " Nov 02 06:44:25.037048 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:44:25.036946 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20a4dcea-b779-47d9-a38b-5ede52dc0076-istiod-ca-cert" pod "20a4dcea-b779-47d9-a38b-5ede52dc0076" (UID: "20a4dcea-b779-47d9-a38b-5ede52dc0076"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:44:25.037152 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:44:25.037122 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2871e54e-8f8f-47d6-a16b-72d986b3835e-workload-certs" pod "2871e54e-8f8f-47d6-a16b-72d986b3835e" (UID: "2871e54e-8f8f-47d6-a16b-72d986b3835e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:44:25.037604 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:44:25.037378 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20a4dcea-b779-47d9-a38b-5ede52dc0076-workload-socket" pod "20a4dcea-b779-47d9-a38b-5ede52dc0076" (UID: "20a4dcea-b779-47d9-a38b-5ede52dc0076"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:44:25.037604 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:44:25.037583 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20a4dcea-b779-47d9-a38b-5ede52dc0076-credential-socket" pod "20a4dcea-b779-47d9-a38b-5ede52dc0076" (UID: "20a4dcea-b779-47d9-a38b-5ede52dc0076"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:44:25.038209 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:44:25.037955 5916 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20a4dcea-b779-47d9-a38b-5ede52dc0076-credential-socket\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:44:25.038209 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:44:25.037981 5916 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2871e54e-8f8f-47d6-a16b-72d986b3835e-workload-certs\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:44:25.038209 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:44:25.037998 5916 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2871e54e-8f8f-47d6-a16b-72d986b3835e-istio-data\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:44:25.038209 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:44:25.038012 5916 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20a4dcea-b779-47d9-a38b-5ede52dc0076-workload-socket\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:44:25.038209 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:44:25.038027 5916 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20a4dcea-b779-47d9-a38b-5ede52dc0076-istiod-ca-cert\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:44:25.038209 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:44:25.038121 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20a4dcea-b779-47d9-a38b-5ede52dc0076-workload-certs" pod "20a4dcea-b779-47d9-a38b-5ede52dc0076" (UID: "20a4dcea-b779-47d9-a38b-5ede52dc0076"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:44:25.038575 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:44:25.038306 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20a4dcea-b779-47d9-a38b-5ede52dc0076-istio-data" pod "20a4dcea-b779-47d9-a38b-5ede52dc0076" (UID: "20a4dcea-b779-47d9-a38b-5ede52dc0076"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:44:25.038575 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:44:25.038486 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2871e54e-8f8f-47d6-a16b-72d986b3835e-credential-socket" pod "2871e54e-8f8f-47d6-a16b-72d986b3835e" (UID: "2871e54e-8f8f-47d6-a16b-72d986b3835e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:44:25.038707 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:44:25.038570 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2871e54e-8f8f-47d6-a16b-72d986b3835e-istiod-ca-cert" pod "2871e54e-8f8f-47d6-a16b-72d986b3835e" (UID: "2871e54e-8f8f-47d6-a16b-72d986b3835e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:44:25.038707 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:44:25.038580 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/20a4dcea-b779-47d9-a38b-5ede52dc0076-istio-podinfo" pod "20a4dcea-b779-47d9-a38b-5ede52dc0076" (UID: "20a4dcea-b779-47d9-a38b-5ede52dc0076"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 06:44:25.038893 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:44:25.038862 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2871e54e-8f8f-47d6-a16b-72d986b3835e-workload-socket" pod "2871e54e-8f8f-47d6-a16b-72d986b3835e" (UID: "2871e54e-8f8f-47d6-a16b-72d986b3835e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:44:25.041498 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:44:25.041476 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2871e54e-8f8f-47d6-a16b-72d986b3835e-istio-envoy" pod "2871e54e-8f8f-47d6-a16b-72d986b3835e" (UID: "2871e54e-8f8f-47d6-a16b-72d986b3835e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:44:25.041941 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:44:25.041894 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2871e54e-8f8f-47d6-a16b-72d986b3835e-kube-api-access-cjp57" pod "2871e54e-8f8f-47d6-a16b-72d986b3835e" (UID: "2871e54e-8f8f-47d6-a16b-72d986b3835e"). InnerVolumeSpecName "kube-api-access-cjp57". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:44:25.042822 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:44:25.042128 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2871e54e-8f8f-47d6-a16b-72d986b3835e-istio-podinfo" pod "2871e54e-8f8f-47d6-a16b-72d986b3835e" (UID: "2871e54e-8f8f-47d6-a16b-72d986b3835e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 06:44:25.042822 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:44:25.042163 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20a4dcea-b779-47d9-a38b-5ede52dc0076-istio-envoy" pod "20a4dcea-b779-47d9-a38b-5ede52dc0076" (UID: "20a4dcea-b779-47d9-a38b-5ede52dc0076"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:44:25.042822 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:44:25.042192 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20a4dcea-b779-47d9-a38b-5ede52dc0076-kube-api-access-5d22b" pod "20a4dcea-b779-47d9-a38b-5ede52dc0076" (UID: "20a4dcea-b779-47d9-a38b-5ede52dc0076"). InnerVolumeSpecName "kube-api-access-5d22b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:44:25.042822 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:44:25.042223 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20a4dcea-b779-47d9-a38b-5ede52dc0076-istio-token" pod "20a4dcea-b779-47d9-a38b-5ede52dc0076" (UID: "20a4dcea-b779-47d9-a38b-5ede52dc0076"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:44:25.044300 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:44:25.044274 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2871e54e-8f8f-47d6-a16b-72d986b3835e-istio-token" pod "2871e54e-8f8f-47d6-a16b-72d986b3835e" (UID: "2871e54e-8f8f-47d6-a16b-72d986b3835e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:44:25.139714 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:44:25.139680 5916 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20a4dcea-b779-47d9-a38b-5ede52dc0076-istio-podinfo\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:44:25.139714 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:44:25.139709 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5d22b\" (UniqueName: \"kubernetes.io/projected/20a4dcea-b779-47d9-a38b-5ede52dc0076-kube-api-access-5d22b\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:44:25.139894 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:44:25.139725 5916 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20a4dcea-b779-47d9-a38b-5ede52dc0076-istio-token\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:44:25.139894 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:44:25.139738 5916 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20a4dcea-b779-47d9-a38b-5ede52dc0076-istio-envoy\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:44:25.139894 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:44:25.139751 5916 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2871e54e-8f8f-47d6-a16b-72d986b3835e-workload-socket\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:44:25.139894 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:44:25.139763 5916 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2871e54e-8f8f-47d6-a16b-72d986b3835e-istio-token\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:44:25.139894 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:44:25.139774 5916 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20a4dcea-b779-47d9-a38b-5ede52dc0076-workload-certs\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:44:25.139894 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:44:25.139787 5916 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20a4dcea-b779-47d9-a38b-5ede52dc0076-istio-data\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:44:25.139894 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:44:25.139800 5916 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2871e54e-8f8f-47d6-a16b-72d986b3835e-istio-podinfo\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:44:25.139894 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:44:25.139812 5916 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2871e54e-8f8f-47d6-a16b-72d986b3835e-istio-envoy\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:44:25.139894 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:44:25.139825 5916 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2871e54e-8f8f-47d6-a16b-72d986b3835e-istiod-ca-cert\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:44:25.139894 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:44:25.139837 5916 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2871e54e-8f8f-47d6-a16b-72d986b3835e-credential-socket\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:44:25.139894 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:44:25.139849 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjp57\" (UniqueName: \"kubernetes.io/projected/2871e54e-8f8f-47d6-a16b-72d986b3835e-kube-api-access-cjp57\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:44:25.461481 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:44:25.461447 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-jr76m"] Nov 02 06:44:25.467158 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:44:25.467136 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-jr76m"] Nov 02 06:44:25.851503 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:44:25.851469 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20a4dcea-b779-47d9-a38b-5ede52dc0076" path="/var/lib/kubelet/pods/20a4dcea-b779-47d9-a38b-5ede52dc0076/volumes" Nov 02 06:44:26.032294 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:44:26.032222 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hbwqc" Nov 02 06:44:26.199895 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:44:26.199851 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hbwqc"] Nov 02 06:44:26.250334 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:44:26.250301 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hbwqc"] Nov 02 06:44:27.843701 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:44:27.843665 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2871e54e-8f8f-47d6-a16b-72d986b3835e" path="/var/lib/kubelet/pods/2871e54e-8f8f-47d6-a16b-72d986b3835e/volumes" Nov 02 06:44:49.895591 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:44:49.895552 5916 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p98cq" containerName="extractor" Nov 02 06:44:54.457216 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:44:54.457178 5916 scope.go:122] "RemoveContainer" containerID="97aff99737a873d789e9cf6c6b6d78eb202c7fd93c2df833cafd1dc7497e7005" Nov 02 06:44:54.471610 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:44:54.471590 5916 scope.go:122] "RemoveContainer" containerID="20fd1e3d7d3d4c7d1c853fddb05b01f9fb89cb809daf8c3f4c617cb6a7778e08" Nov 02 06:44:54.482600 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:44:54.482568 5916 scope.go:122] "RemoveContainer" containerID="26b089f54ed089640fbc8fb80ffc3cb88195f912df29c6c21273e310c8f4dec8" Nov 02 06:45:00.052305 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:00.052254 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24af19b3-lp25w"] Nov 02 06:45:00.099732 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:00.099694 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24af19b3-lp25w"] Nov 02 06:45:00.164763 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:00.164724 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-581608-29893365-t2rws"] Nov 02 06:45:00.165488 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:00.165466 5916 cpu_manager.go:414] "containerMap: removing container" podUID="2f9273e5-b930-47f0-b710-234974ad9025" containerName="istio-proxy" Nov 02 06:45:00.165488 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:00.165488 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f9273e5-b930-47f0-b710-234974ad9025" containerName="istio-proxy" Nov 02 06:45:00.165664 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:00.165506 5916 cpu_manager.go:414] "containerMap: removing container" podUID="20a4dcea-b779-47d9-a38b-5ede52dc0076" containerName="istio-proxy" Nov 02 06:45:00.165664 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:00.165523 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20a4dcea-b779-47d9-a38b-5ede52dc0076" containerName="istio-proxy" Nov 02 06:45:00.165664 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:00.165566 5916 cpu_manager.go:414] "containerMap: removing container" podUID="2871e54e-8f8f-47d6-a16b-72d986b3835e" containerName="istio-proxy" Nov 02 06:45:00.165664 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:00.165575 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2871e54e-8f8f-47d6-a16b-72d986b3835e" containerName="istio-proxy" Nov 02 06:45:00.165862 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:00.165806 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="2871e54e-8f8f-47d6-a16b-72d986b3835e" containerName="istio-proxy" Nov 02 06:45:00.165862 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:00.165838 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f9273e5-b930-47f0-b710-234974ad9025" containerName="istio-proxy" Nov 02 06:45:00.165862 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:00.165849 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="20a4dcea-b779-47d9-a38b-5ede52dc0076" containerName="istio-proxy" Nov 02 06:45:00.171477 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:00.171452 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fb1398-29893365-pqgct"] Nov 02 06:45:00.171623 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:00.171572 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-581608-29893365-t2rws" Nov 02 06:45:00.176576 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:00.176556 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fb1398-29893365-pqgct" Nov 02 06:45:00.177492 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:00.177463 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-680b7b-29893365-5wmwz"] Nov 02 06:45:00.182720 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:00.182696 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-680b7b-29893365-5wmwz" Nov 02 06:45:00.186431 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:00.186407 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-70898f-29893365-c4vxq"] Nov 02 06:45:00.190604 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:00.190572 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-70898f-29893365-c4vxq" Nov 02 06:45:00.194143 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:00.194125 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-8ecb23-29893365-kb456"] Nov 02 06:45:00.198254 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:00.198237 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-8ecb23-29893365-kb456" Nov 02 06:45:00.216105 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:00.216083 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-bdcd80-29893365-4l8lg"] Nov 02 06:45:00.220693 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:00.220675 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-bdcd80-29893365-4l8lg" Nov 02 06:45:00.221984 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:00.221965 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-3e96be-29893365-c9n69"] Nov 02 06:45:00.226758 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:00.226738 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-3e96be-29893365-c9n69" Nov 02 06:45:00.227809 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:00.227791 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-581608-29893365-t2rws"] Nov 02 06:45:00.233406 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:00.233384 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fb1398-29893365-pqgct"] Nov 02 06:45:00.259365 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:00.259330 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7494bbe-ebe5-490e-9861-9d573c45f068-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-bdcd80-29893365-4l8lg\" (UID: \"d7494bbe-ebe5-490e-9861-9d573c45f068\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-bdcd80-29893365-4l8lg" Nov 02 06:45:00.259527 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:00.259451 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6hvq\" (UniqueName: \"kubernetes.io/projected/d7494bbe-ebe5-490e-9861-9d573c45f068-kube-api-access-f6hvq\") pod \"maas-api-key-cleanup-e2e-worker-w0-bdcd80-29893365-4l8lg\" (UID: \"d7494bbe-ebe5-490e-9861-9d573c45f068\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-bdcd80-29893365-4l8lg" Nov 02 06:45:00.268927 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:00.267599 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-70898f-29893365-c4vxq"] Nov 02 06:45:00.275517 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:00.275484 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-8ecb23-29893365-kb456"] Nov 02 06:45:00.321995 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:00.316998 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-bdcd80-29893365-4l8lg"] Nov 02 06:45:00.321995 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:00.319413 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-3e96be-29893365-c9n69"] Nov 02 06:45:00.321995 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:00.321256 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-68l4r\"" Nov 02 06:45:00.354833 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:00.354802 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-680b7b-29893365-5wmwz"] Nov 02 06:45:00.364166 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:00.364132 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcebf78f-ba62-491d-a987-b784158c1c70-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-fb1398-29893365-pqgct\" (UID: \"fcebf78f-ba62-491d-a987-b784158c1c70\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fb1398-29893365-pqgct" Nov 02 06:45:00.364337 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:00.364219 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2cac8cd-4ce7-4c70-b380-b9aec98431b8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-3e96be-29893365-c9n69\" (UID: \"f2cac8cd-4ce7-4c70-b380-b9aec98431b8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-3e96be-29893365-c9n69" Nov 02 06:45:00.364337 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:00.364301 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6hvq\" (UniqueName: \"kubernetes.io/projected/d7494bbe-ebe5-490e-9861-9d573c45f068-kube-api-access-f6hvq\") pod \"maas-api-key-cleanup-e2e-worker-w0-bdcd80-29893365-4l8lg\" (UID: \"d7494bbe-ebe5-490e-9861-9d573c45f068\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-bdcd80-29893365-4l8lg" Nov 02 06:45:00.364455 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:00.364392 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63a2970a-3a1d-4646-8ae6-9432dc18ab15-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-8ecb23-29893365-kb456\" (UID: \"63a2970a-3a1d-4646-8ae6-9432dc18ab15\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-8ecb23-29893365-kb456" Nov 02 06:45:00.364455 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:00.364434 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghjgn\" (UniqueName: \"kubernetes.io/projected/f2cac8cd-4ce7-4c70-b380-b9aec98431b8-kube-api-access-ghjgn\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-3e96be-29893365-c9n69\" (UID: \"f2cac8cd-4ce7-4c70-b380-b9aec98431b8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-3e96be-29893365-c9n69" Nov 02 06:45:00.364589 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:00.364515 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fr9w6\" (UniqueName: \"kubernetes.io/projected/fcebf78f-ba62-491d-a987-b784158c1c70-kube-api-access-fr9w6\") pod \"maas-api-key-cleanup-e2e-worker-w3-fb1398-29893365-pqgct\" (UID: \"fcebf78f-ba62-491d-a987-b784158c1c70\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fb1398-29893365-pqgct" Nov 02 06:45:00.364589 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:00.364556 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4whc\" (UniqueName: \"kubernetes.io/projected/4d559915-7623-421a-9631-d308bf5a8c4e-kube-api-access-p4whc\") pod \"maas-api-key-cleanup-e2e-worker-w4-680b7b-29893365-5wmwz\" (UID: \"4d559915-7623-421a-9631-d308bf5a8c4e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-680b7b-29893365-5wmwz" Nov 02 06:45:00.364700 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:00.364637 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rszq8\" (UniqueName: \"kubernetes.io/projected/63a2970a-3a1d-4646-8ae6-9432dc18ab15-kube-api-access-rszq8\") pod \"maas-api-key-cleanup-e2e-api-a-8ecb23-29893365-kb456\" (UID: \"63a2970a-3a1d-4646-8ae6-9432dc18ab15\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-8ecb23-29893365-kb456" Nov 02 06:45:00.364760 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:00.364695 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmlvx\" (UniqueName: \"kubernetes.io/projected/b0419130-0781-47f1-af14-a8eb8917b1eb-kube-api-access-gmlvx\") pod \"maas-api-key-cleanup-e2e-api-a-70898f-29893365-c4vxq\" (UID: \"b0419130-0781-47f1-af14-a8eb8917b1eb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-70898f-29893365-c4vxq" Nov 02 06:45:00.364760 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:00.364732 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0419130-0781-47f1-af14-a8eb8917b1eb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-70898f-29893365-c4vxq\" (UID: \"b0419130-0781-47f1-af14-a8eb8917b1eb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-70898f-29893365-c4vxq" Nov 02 06:45:00.364863 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:00.364757 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d559915-7623-421a-9631-d308bf5a8c4e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-680b7b-29893365-5wmwz\" (UID: \"4d559915-7623-421a-9631-d308bf5a8c4e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-680b7b-29893365-5wmwz" Nov 02 06:45:00.364863 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:00.364806 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xbsd\" (UniqueName: \"kubernetes.io/projected/3de4bd72-539c-4fd8-a0a1-c08383426e53-kube-api-access-6xbsd\") pod \"maas-api-key-cleanup-e2e-worker-w2-581608-29893365-t2rws\" (UID: \"3de4bd72-539c-4fd8-a0a1-c08383426e53\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-581608-29893365-t2rws" Nov 02 06:45:00.364863 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:00.364840 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7494bbe-ebe5-490e-9861-9d573c45f068-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-bdcd80-29893365-4l8lg\" (UID: \"d7494bbe-ebe5-490e-9861-9d573c45f068\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-bdcd80-29893365-4l8lg" Nov 02 06:45:00.365046 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:00.364879 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3de4bd72-539c-4fd8-a0a1-c08383426e53-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-581608-29893365-t2rws\" (UID: \"3de4bd72-539c-4fd8-a0a1-c08383426e53\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-581608-29893365-t2rws" Nov 02 06:45:00.366009 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:00.365985 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7494bbe-ebe5-490e-9861-9d573c45f068-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-bdcd80-29893365-4l8lg\" (UID: \"d7494bbe-ebe5-490e-9861-9d573c45f068\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-bdcd80-29893365-4l8lg" Nov 02 06:45:00.421869 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:00.421834 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6hvq\" (UniqueName: \"kubernetes.io/projected/d7494bbe-ebe5-490e-9861-9d573c45f068-kube-api-access-f6hvq\") pod \"maas-api-key-cleanup-e2e-worker-w0-bdcd80-29893365-4l8lg\" (UID: \"d7494bbe-ebe5-490e-9861-9d573c45f068\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-bdcd80-29893365-4l8lg" Nov 02 06:45:00.466273 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:00.466226 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xbsd\" (UniqueName: \"kubernetes.io/projected/3de4bd72-539c-4fd8-a0a1-c08383426e53-kube-api-access-6xbsd\") pod \"maas-api-key-cleanup-e2e-worker-w2-581608-29893365-t2rws\" (UID: \"3de4bd72-539c-4fd8-a0a1-c08383426e53\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-581608-29893365-t2rws" Nov 02 06:45:00.466445 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:00.466295 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3de4bd72-539c-4fd8-a0a1-c08383426e53-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-581608-29893365-t2rws\" (UID: \"3de4bd72-539c-4fd8-a0a1-c08383426e53\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-581608-29893365-t2rws" Nov 02 06:45:00.466445 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:00.466329 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcebf78f-ba62-491d-a987-b784158c1c70-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-fb1398-29893365-pqgct\" (UID: \"fcebf78f-ba62-491d-a987-b784158c1c70\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fb1398-29893365-pqgct" Nov 02 06:45:00.466445 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:00.466396 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2cac8cd-4ce7-4c70-b380-b9aec98431b8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-3e96be-29893365-c9n69\" (UID: \"f2cac8cd-4ce7-4c70-b380-b9aec98431b8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-3e96be-29893365-c9n69" Nov 02 06:45:00.467087 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:00.467062 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcebf78f-ba62-491d-a987-b784158c1c70-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-fb1398-29893365-pqgct\" (UID: \"fcebf78f-ba62-491d-a987-b784158c1c70\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fb1398-29893365-pqgct" Nov 02 06:45:00.467210 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:00.467143 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3de4bd72-539c-4fd8-a0a1-c08383426e53-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-581608-29893365-t2rws\" (UID: \"3de4bd72-539c-4fd8-a0a1-c08383426e53\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-581608-29893365-t2rws" Nov 02 06:45:00.467261 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:00.467231 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2cac8cd-4ce7-4c70-b380-b9aec98431b8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-3e96be-29893365-c9n69\" (UID: \"f2cac8cd-4ce7-4c70-b380-b9aec98431b8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-3e96be-29893365-c9n69" Nov 02 06:45:00.467409 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:00.467390 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63a2970a-3a1d-4646-8ae6-9432dc18ab15-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-8ecb23-29893365-kb456\" (UID: \"63a2970a-3a1d-4646-8ae6-9432dc18ab15\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-8ecb23-29893365-kb456" Nov 02 06:45:00.467461 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:00.467438 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghjgn\" (UniqueName: \"kubernetes.io/projected/f2cac8cd-4ce7-4c70-b380-b9aec98431b8-kube-api-access-ghjgn\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-3e96be-29893365-c9n69\" (UID: \"f2cac8cd-4ce7-4c70-b380-b9aec98431b8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-3e96be-29893365-c9n69" Nov 02 06:45:00.467527 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:00.467513 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fr9w6\" (UniqueName: \"kubernetes.io/projected/fcebf78f-ba62-491d-a987-b784158c1c70-kube-api-access-fr9w6\") pod \"maas-api-key-cleanup-e2e-worker-w3-fb1398-29893365-pqgct\" (UID: \"fcebf78f-ba62-491d-a987-b784158c1c70\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fb1398-29893365-pqgct" Nov 02 06:45:00.467596 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:00.467551 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4whc\" (UniqueName: \"kubernetes.io/projected/4d559915-7623-421a-9631-d308bf5a8c4e-kube-api-access-p4whc\") pod \"maas-api-key-cleanup-e2e-worker-w4-680b7b-29893365-5wmwz\" (UID: \"4d559915-7623-421a-9631-d308bf5a8c4e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-680b7b-29893365-5wmwz" Nov 02 06:45:00.467652 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:00.467624 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rszq8\" (UniqueName: \"kubernetes.io/projected/63a2970a-3a1d-4646-8ae6-9432dc18ab15-kube-api-access-rszq8\") pod \"maas-api-key-cleanup-e2e-api-a-8ecb23-29893365-kb456\" (UID: \"63a2970a-3a1d-4646-8ae6-9432dc18ab15\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-8ecb23-29893365-kb456" Nov 02 06:45:00.467708 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:00.467679 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmlvx\" (UniqueName: \"kubernetes.io/projected/b0419130-0781-47f1-af14-a8eb8917b1eb-kube-api-access-gmlvx\") pod \"maas-api-key-cleanup-e2e-api-a-70898f-29893365-c4vxq\" (UID: \"b0419130-0781-47f1-af14-a8eb8917b1eb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-70898f-29893365-c4vxq" Nov 02 06:45:00.467763 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:00.467724 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0419130-0781-47f1-af14-a8eb8917b1eb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-70898f-29893365-c4vxq\" (UID: \"b0419130-0781-47f1-af14-a8eb8917b1eb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-70898f-29893365-c4vxq" Nov 02 06:45:00.467763 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:00.467749 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d559915-7623-421a-9631-d308bf5a8c4e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-680b7b-29893365-5wmwz\" (UID: \"4d559915-7623-421a-9631-d308bf5a8c4e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-680b7b-29893365-5wmwz" Nov 02 06:45:00.468095 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:00.468067 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63a2970a-3a1d-4646-8ae6-9432dc18ab15-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-8ecb23-29893365-kb456\" (UID: \"63a2970a-3a1d-4646-8ae6-9432dc18ab15\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-8ecb23-29893365-kb456" Nov 02 06:45:00.468446 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:00.468426 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0419130-0781-47f1-af14-a8eb8917b1eb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-70898f-29893365-c4vxq\" (UID: \"b0419130-0781-47f1-af14-a8eb8917b1eb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-70898f-29893365-c4vxq" Nov 02 06:45:00.468546 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:00.468524 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d559915-7623-421a-9631-d308bf5a8c4e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-680b7b-29893365-5wmwz\" (UID: \"4d559915-7623-421a-9631-d308bf5a8c4e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-680b7b-29893365-5wmwz" Nov 02 06:45:00.533878 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:00.532683 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghjgn\" (UniqueName: \"kubernetes.io/projected/f2cac8cd-4ce7-4c70-b380-b9aec98431b8-kube-api-access-ghjgn\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-3e96be-29893365-c9n69\" (UID: \"f2cac8cd-4ce7-4c70-b380-b9aec98431b8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-3e96be-29893365-c9n69" Nov 02 06:45:00.536050 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:00.534724 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4whc\" (UniqueName: \"kubernetes.io/projected/4d559915-7623-421a-9631-d308bf5a8c4e-kube-api-access-p4whc\") pod \"maas-api-key-cleanup-e2e-worker-w4-680b7b-29893365-5wmwz\" (UID: \"4d559915-7623-421a-9631-d308bf5a8c4e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-680b7b-29893365-5wmwz" Nov 02 06:45:00.536050 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:00.535657 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fr9w6\" (UniqueName: \"kubernetes.io/projected/fcebf78f-ba62-491d-a987-b784158c1c70-kube-api-access-fr9w6\") pod \"maas-api-key-cleanup-e2e-worker-w3-fb1398-29893365-pqgct\" (UID: \"fcebf78f-ba62-491d-a987-b784158c1c70\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fb1398-29893365-pqgct" Nov 02 06:45:00.536190 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:00.536081 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rszq8\" (UniqueName: \"kubernetes.io/projected/63a2970a-3a1d-4646-8ae6-9432dc18ab15-kube-api-access-rszq8\") pod \"maas-api-key-cleanup-e2e-api-a-8ecb23-29893365-kb456\" (UID: \"63a2970a-3a1d-4646-8ae6-9432dc18ab15\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-8ecb23-29893365-kb456" Nov 02 06:45:00.536775 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:00.536728 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xbsd\" (UniqueName: \"kubernetes.io/projected/3de4bd72-539c-4fd8-a0a1-c08383426e53-kube-api-access-6xbsd\") pod \"maas-api-key-cleanup-e2e-worker-w2-581608-29893365-t2rws\" (UID: \"3de4bd72-539c-4fd8-a0a1-c08383426e53\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-581608-29893365-t2rws" Nov 02 06:45:00.537285 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:00.537263 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmlvx\" (UniqueName: \"kubernetes.io/projected/b0419130-0781-47f1-af14-a8eb8917b1eb-kube-api-access-gmlvx\") pod \"maas-api-key-cleanup-e2e-api-a-70898f-29893365-c4vxq\" (UID: \"b0419130-0781-47f1-af14-a8eb8917b1eb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-70898f-29893365-c4vxq" Nov 02 06:45:00.538981 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:00.538957 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-bdcd80-29893365-4l8lg" Nov 02 06:45:00.548148 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:00.548126 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-3e96be-29893365-c9n69" Nov 02 06:45:00.789859 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:00.789821 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-581608-29893365-t2rws" Nov 02 06:45:00.800230 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:00.800204 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fb1398-29893365-pqgct" Nov 02 06:45:00.808393 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:00.808368 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-680b7b-29893365-5wmwz" Nov 02 06:45:00.818337 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:00.818137 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-70898f-29893365-c4vxq" Nov 02 06:45:00.825736 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:00.825656 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-8ecb23-29893365-kb456" Nov 02 06:45:00.965063 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:00.964456 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-bdcd80-29893365-4l8lg"] Nov 02 06:45:01.051725 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:01.051692 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-3e96be-29893365-c9n69"] Nov 02 06:45:01.401511 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:01.401481 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-8ecb23-29893365-kb456"] Nov 02 06:45:01.402643 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:45:01.402616 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63a2970a_3a1d_4646_8ae6_9432dc18ab15.slice/crio-1813cc5503456da1f2412c7ecdd370dccc31352f24c0f8af496d9b702fc3bf8a WatchSource:0}: Error finding container 1813cc5503456da1f2412c7ecdd370dccc31352f24c0f8af496d9b702fc3bf8a: Status 404 returned error can't find the container with id 1813cc5503456da1f2412c7ecdd370dccc31352f24c0f8af496d9b702fc3bf8a Nov 02 06:45:01.452803 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:45:01.452774 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0419130_0781_47f1_af14_a8eb8917b1eb.slice/crio-b00381abec244fabd0d37eef09e8b03a23c5b3f509b6de61d13ef2f80e913d46 WatchSource:0}: Error finding container b00381abec244fabd0d37eef09e8b03a23c5b3f509b6de61d13ef2f80e913d46: Status 404 returned error can't find the container with id b00381abec244fabd0d37eef09e8b03a23c5b3f509b6de61d13ef2f80e913d46 Nov 02 06:45:01.452979 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:01.452879 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-70898f-29893365-c4vxq"] Nov 02 06:45:01.453622 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:45:01.453598 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3de4bd72_539c_4fd8_a0a1_c08383426e53.slice/crio-09d1fa8fb4abba3921e64ee0d56d69a017469d97903df005d0d87ae3254f9fb9 WatchSource:0}: Error finding container 09d1fa8fb4abba3921e64ee0d56d69a017469d97903df005d0d87ae3254f9fb9: Status 404 returned error can't find the container with id 09d1fa8fb4abba3921e64ee0d56d69a017469d97903df005d0d87ae3254f9fb9 Nov 02 06:45:01.459434 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:01.459329 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-581608-29893365-t2rws"] Nov 02 06:45:01.529034 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:01.528999 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fb1398-29893365-pqgct"] Nov 02 06:45:01.531644 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:45:01.531614 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcebf78f_ba62_491d_a987_b784158c1c70.slice/crio-22839b2c5f894512076d86808d849c4e46a893e70b2d11aa14481146a39189c2 WatchSource:0}: Error finding container 22839b2c5f894512076d86808d849c4e46a893e70b2d11aa14481146a39189c2: Status 404 returned error can't find the container with id 22839b2c5f894512076d86808d849c4e46a893e70b2d11aa14481146a39189c2 Nov 02 06:45:01.559105 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:01.559079 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-680b7b-29893365-5wmwz"] Nov 02 06:45:01.562231 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:45:01.562206 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d559915_7623_421a_9631_d308bf5a8c4e.slice/crio-e0e0de2d0f8ef04111735781be3a7c4e927b5697a6cd45166f5a06f2447279df WatchSource:0}: Error finding container e0e0de2d0f8ef04111735781be3a7c4e927b5697a6cd45166f5a06f2447279df: Status 404 returned error can't find the container with id e0e0de2d0f8ef04111735781be3a7c4e927b5697a6cd45166f5a06f2447279df Nov 02 06:45:01.597303 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:01.597218 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-680b7b-29893365-5wmwz" event={"ID":"4d559915-7623-421a-9631-d308bf5a8c4e","Type":"ContainerStarted","Data":"e0e0de2d0f8ef04111735781be3a7c4e927b5697a6cd45166f5a06f2447279df"} Nov 02 06:45:01.598479 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:01.598455 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-bdcd80-29893365-4l8lg" event={"ID":"d7494bbe-ebe5-490e-9861-9d573c45f068","Type":"ContainerStarted","Data":"07cef68d7a08d31b16f00462263070e9429d8eb065d3ef0b9a7f4d31cec5d074"} Nov 02 06:45:01.599623 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:01.599600 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-581608-29893365-t2rws" event={"ID":"3de4bd72-539c-4fd8-a0a1-c08383426e53","Type":"ContainerStarted","Data":"09d1fa8fb4abba3921e64ee0d56d69a017469d97903df005d0d87ae3254f9fb9"} Nov 02 06:45:01.600840 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:01.600817 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fb1398-29893365-pqgct" event={"ID":"fcebf78f-ba62-491d-a987-b784158c1c70","Type":"ContainerStarted","Data":"22839b2c5f894512076d86808d849c4e46a893e70b2d11aa14481146a39189c2"} Nov 02 06:45:01.602200 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:01.602178 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-70898f-29893365-c4vxq" event={"ID":"b0419130-0781-47f1-af14-a8eb8917b1eb","Type":"ContainerStarted","Data":"b00381abec244fabd0d37eef09e8b03a23c5b3f509b6de61d13ef2f80e913d46"} Nov 02 06:45:01.603593 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:01.603544 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-8ecb23-29893365-kb456" event={"ID":"63a2970a-3a1d-4646-8ae6-9432dc18ab15","Type":"ContainerStarted","Data":"1813cc5503456da1f2412c7ecdd370dccc31352f24c0f8af496d9b702fc3bf8a"} Nov 02 06:45:01.605163 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:01.605140 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-3e96be-29893365-c9n69" event={"ID":"f2cac8cd-4ce7-4c70-b380-b9aec98431b8","Type":"ContainerStarted","Data":"b44b12ed0ea5ef9a1f8b4965d31e432029e869c4df89f02c117019f21f155a84"} Nov 02 06:45:01.605255 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:01.605178 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-3e96be-29893365-c9n69" event={"ID":"f2cac8cd-4ce7-4c70-b380-b9aec98431b8","Type":"ContainerStarted","Data":"6ee1440163f477f802993a2ecaf0d1ffe3a809bbcdabd0710aeac5e1f8e549d6"} Nov 02 06:45:01.720481 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:01.720412 5916 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-3e96be-29893365-c9n69" podStartSLOduration=1.720393922 podStartE2EDuration="1.720393922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:45:01.71811998 +0000 UTC m=+1689.989940374" watchObservedRunningTime="2026-11-02 06:45:01.720393922 +0000 UTC m=+1689.992214317" Nov 02 06:45:01.845680 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:01.845656 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2816f619-278a-4693-a977-1261c31ea5a4" path="/var/lib/kubelet/pods/2816f619-278a-4693-a977-1261c31ea5a4/volumes" Nov 02 06:45:02.616432 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:02.616325 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-680b7b-29893365-5wmwz" event={"ID":"4d559915-7623-421a-9631-d308bf5a8c4e","Type":"ContainerStarted","Data":"a0d3b4c635ea2b362bebcbba3b6b7649b6f40d09ded30d36a2e664a5c7b5b4cd"} Nov 02 06:45:02.617916 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:02.617877 5916 generic.go:356] "Generic (PLEG): container finished" podID="d7494bbe-ebe5-490e-9861-9d573c45f068" containerID="91eb3cf91c522c3a1270ce9c70cb7cc5fbafc898c4f7dd17e09c42b729069532" exitCode=60 Nov 02 06:45:02.618026 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:02.617931 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-bdcd80-29893365-4l8lg" event={"ID":"d7494bbe-ebe5-490e-9861-9d573c45f068","Type":"ContainerDied","Data":"91eb3cf91c522c3a1270ce9c70cb7cc5fbafc898c4f7dd17e09c42b729069532"} Nov 02 06:45:02.618181 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:02.618165 5916 scope.go:122] "RemoveContainer" containerID="91eb3cf91c522c3a1270ce9c70cb7cc5fbafc898c4f7dd17e09c42b729069532" Nov 02 06:45:02.619725 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:02.619705 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-581608-29893365-t2rws" event={"ID":"3de4bd72-539c-4fd8-a0a1-c08383426e53","Type":"ContainerStarted","Data":"becc164bd9578ac3eff8113f82db0047b6b19ab07e3f818513a5c9b98a18a400"} Nov 02 06:45:02.621493 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:02.621472 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fb1398-29893365-pqgct" event={"ID":"fcebf78f-ba62-491d-a987-b784158c1c70","Type":"ContainerStarted","Data":"7ffe0c89c0bb980fee7c36cbf42cc01924cd27b49189faab96403bfb60edac17"} Nov 02 06:45:02.623205 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:02.623161 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-70898f-29893365-c4vxq" event={"ID":"b0419130-0781-47f1-af14-a8eb8917b1eb","Type":"ContainerStarted","Data":"2253f44b1b4ceb0a81b15a1ace41a405b8441c0ed0f6b71b37b87f64129452f3"} Nov 02 06:45:02.624725 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:02.624701 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-8ecb23-29893365-kb456" event={"ID":"63a2970a-3a1d-4646-8ae6-9432dc18ab15","Type":"ContainerStarted","Data":"7b2bf62dc2b31f9300d157f9971148915ebbf80f5ef4bcbc6457ac2b27944377"} Nov 02 06:45:02.626674 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:02.626653 5916 generic.go:356] "Generic (PLEG): container finished" podID="f2cac8cd-4ce7-4c70-b380-b9aec98431b8" containerID="b44b12ed0ea5ef9a1f8b4965d31e432029e869c4df89f02c117019f21f155a84" exitCode=60 Nov 02 06:45:02.627250 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:02.627114 5916 scope.go:122] "RemoveContainer" containerID="b44b12ed0ea5ef9a1f8b4965d31e432029e869c4df89f02c117019f21f155a84" Nov 02 06:45:02.627629 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:02.627129 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-3e96be-29893365-c9n69" event={"ID":"f2cac8cd-4ce7-4c70-b380-b9aec98431b8","Type":"ContainerDied","Data":"b44b12ed0ea5ef9a1f8b4965d31e432029e869c4df89f02c117019f21f155a84"} Nov 02 06:45:02.740804 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:02.740745 5916 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-680b7b-29893365-5wmwz" podStartSLOduration=2.7407200830000003 podStartE2EDuration="2.740720083s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:45:02.735038169 +0000 UTC m=+1691.006858563" watchObservedRunningTime="2026-11-02 06:45:02.740720083 +0000 UTC m=+1691.012540473" Nov 02 06:45:02.866269 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:02.866225 5916 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-70898f-29893365-c4vxq" podStartSLOduration=2.866211985 podStartE2EDuration="2.866211985s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:45:02.860279898 +0000 UTC m=+1691.132100293" watchObservedRunningTime="2026-11-02 06:45:02.866211985 +0000 UTC m=+1691.138032401" Nov 02 06:45:02.974698 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:02.974636 5916 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-581608-29893365-t2rws" podStartSLOduration=2.974623038 podStartE2EDuration="2.974623038s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:45:02.973451022 +0000 UTC m=+1691.245271417" watchObservedRunningTime="2026-11-02 06:45:02.974623038 +0000 UTC m=+1691.246443432" Nov 02 06:45:03.087534 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:03.087477 5916 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fb1398-29893365-pqgct" podStartSLOduration=3.087459995 podStartE2EDuration="3.087459995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:45:03.086891427 +0000 UTC m=+1691.358711820" watchObservedRunningTime="2026-11-02 06:45:03.087459995 +0000 UTC m=+1691.359280389" Nov 02 06:45:03.429657 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:03.429604 5916 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-8ecb23-29893365-kb456" podStartSLOduration=3.429591836 podStartE2EDuration="3.429591836s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:45:03.428267328 +0000 UTC m=+1691.700087722" watchObservedRunningTime="2026-11-02 06:45:03.429591836 +0000 UTC m=+1691.701412229" Nov 02 06:45:03.645526 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:03.645492 5916 generic.go:356] "Generic (PLEG): container finished" podID="b0419130-0781-47f1-af14-a8eb8917b1eb" containerID="2253f44b1b4ceb0a81b15a1ace41a405b8441c0ed0f6b71b37b87f64129452f3" exitCode=60 Nov 02 06:45:03.646041 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:03.645600 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-70898f-29893365-c4vxq" event={"ID":"b0419130-0781-47f1-af14-a8eb8917b1eb","Type":"ContainerDied","Data":"2253f44b1b4ceb0a81b15a1ace41a405b8441c0ed0f6b71b37b87f64129452f3"} Nov 02 06:45:03.646041 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:03.645951 5916 scope.go:122] "RemoveContainer" containerID="2253f44b1b4ceb0a81b15a1ace41a405b8441c0ed0f6b71b37b87f64129452f3" Nov 02 06:45:03.647941 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:03.647880 5916 generic.go:356] "Generic (PLEG): container finished" podID="63a2970a-3a1d-4646-8ae6-9432dc18ab15" containerID="7b2bf62dc2b31f9300d157f9971148915ebbf80f5ef4bcbc6457ac2b27944377" exitCode=60 Nov 02 06:45:03.648387 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:03.648368 5916 scope.go:122] "RemoveContainer" containerID="7b2bf62dc2b31f9300d157f9971148915ebbf80f5ef4bcbc6457ac2b27944377" Nov 02 06:45:03.648715 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:03.648692 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-8ecb23-29893365-kb456" event={"ID":"63a2970a-3a1d-4646-8ae6-9432dc18ab15","Type":"ContainerDied","Data":"7b2bf62dc2b31f9300d157f9971148915ebbf80f5ef4bcbc6457ac2b27944377"} Nov 02 06:45:03.655999 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:03.655948 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-3e96be-29893365-c9n69" event={"ID":"f2cac8cd-4ce7-4c70-b380-b9aec98431b8","Type":"ContainerStarted","Data":"a2742a61c2faf2dbad556065f751c74e9ae1f2584d455c8e41d41ca5179d1905"} Nov 02 06:45:03.658662 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:03.658637 5916 generic.go:356] "Generic (PLEG): container finished" podID="4d559915-7623-421a-9631-d308bf5a8c4e" containerID="a0d3b4c635ea2b362bebcbba3b6b7649b6f40d09ded30d36a2e664a5c7b5b4cd" exitCode=60 Nov 02 06:45:03.658760 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:03.658732 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-680b7b-29893365-5wmwz" event={"ID":"4d559915-7623-421a-9631-d308bf5a8c4e","Type":"ContainerDied","Data":"a0d3b4c635ea2b362bebcbba3b6b7649b6f40d09ded30d36a2e664a5c7b5b4cd"} Nov 02 06:45:03.658961 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:03.658897 5916 scope.go:122] "RemoveContainer" containerID="a0d3b4c635ea2b362bebcbba3b6b7649b6f40d09ded30d36a2e664a5c7b5b4cd" Nov 02 06:45:03.663915 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:03.663863 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-bdcd80-29893365-4l8lg" event={"ID":"d7494bbe-ebe5-490e-9861-9d573c45f068","Type":"ContainerStarted","Data":"6ef841088abc3ded98176f238b23fe394af7eeebb1d392a325a1c04115987d9d"} Nov 02 06:45:03.665872 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:03.665848 5916 generic.go:356] "Generic (PLEG): container finished" podID="3de4bd72-539c-4fd8-a0a1-c08383426e53" containerID="becc164bd9578ac3eff8113f82db0047b6b19ab07e3f818513a5c9b98a18a400" exitCode=60 Nov 02 06:45:03.666120 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:03.665958 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-581608-29893365-t2rws" event={"ID":"3de4bd72-539c-4fd8-a0a1-c08383426e53","Type":"ContainerDied","Data":"becc164bd9578ac3eff8113f82db0047b6b19ab07e3f818513a5c9b98a18a400"} Nov 02 06:45:03.666447 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:03.666431 5916 scope.go:122] "RemoveContainer" containerID="becc164bd9578ac3eff8113f82db0047b6b19ab07e3f818513a5c9b98a18a400" Nov 02 06:45:03.667881 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:03.667856 5916 generic.go:356] "Generic (PLEG): container finished" podID="fcebf78f-ba62-491d-a987-b784158c1c70" containerID="7ffe0c89c0bb980fee7c36cbf42cc01924cd27b49189faab96403bfb60edac17" exitCode=60 Nov 02 06:45:03.667999 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:03.667956 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fb1398-29893365-pqgct" event={"ID":"fcebf78f-ba62-491d-a987-b784158c1c70","Type":"ContainerDied","Data":"7ffe0c89c0bb980fee7c36cbf42cc01924cd27b49189faab96403bfb60edac17"} Nov 02 06:45:03.668133 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:03.668117 5916 scope.go:122] "RemoveContainer" containerID="7ffe0c89c0bb980fee7c36cbf42cc01924cd27b49189faab96403bfb60edac17" Nov 02 06:45:04.231346 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:04.231298 5916 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-bdcd80-29893365-4l8lg" podStartSLOduration=4.231283178 podStartE2EDuration="4.231283178s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:45:04.2224437 +0000 UTC m=+1692.494264093" watchObservedRunningTime="2026-11-02 06:45:04.231283178 +0000 UTC m=+1692.503103571" Nov 02 06:45:04.681224 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:04.681178 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-680b7b-29893365-5wmwz" event={"ID":"4d559915-7623-421a-9631-d308bf5a8c4e","Type":"ContainerStarted","Data":"8eb4d09cd62501a9ac05d17c810fa48558ae1ecd665f5d19d9c44bf9ae3ae706"} Nov 02 06:45:04.683190 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:04.683070 5916 generic.go:356] "Generic (PLEG): container finished" podID="d7494bbe-ebe5-490e-9861-9d573c45f068" containerID="6ef841088abc3ded98176f238b23fe394af7eeebb1d392a325a1c04115987d9d" exitCode=60 Nov 02 06:45:04.683313 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:04.683189 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-bdcd80-29893365-4l8lg" event={"ID":"d7494bbe-ebe5-490e-9861-9d573c45f068","Type":"ContainerDied","Data":"6ef841088abc3ded98176f238b23fe394af7eeebb1d392a325a1c04115987d9d"} Nov 02 06:45:04.683313 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:04.683249 5916 scope.go:122] "RemoveContainer" containerID="91eb3cf91c522c3a1270ce9c70cb7cc5fbafc898c4f7dd17e09c42b729069532" Nov 02 06:45:04.683431 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:04.683409 5916 scope.go:122] "RemoveContainer" containerID="6ef841088abc3ded98176f238b23fe394af7eeebb1d392a325a1c04115987d9d" Nov 02 06:45:04.683758 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:45:04.683734 5916 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-bdcd80-29893365-4l8lg_odh-ai-gateway-infra(d7494bbe-ebe5-490e-9861-9d573c45f068)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-bdcd80-29893365-4l8lg" podUID="d7494bbe-ebe5-490e-9861-9d573c45f068" Nov 02 06:45:04.685146 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:04.685122 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-581608-29893365-t2rws" event={"ID":"3de4bd72-539c-4fd8-a0a1-c08383426e53","Type":"ContainerStarted","Data":"f12bae89648ace6f4cba382899bfbc69f93b5a14ac1a35625f83e7dc646f5373"} Nov 02 06:45:04.687079 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:04.687059 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fb1398-29893365-pqgct" event={"ID":"fcebf78f-ba62-491d-a987-b784158c1c70","Type":"ContainerStarted","Data":"a67a92896cac5330ffcf79b579ab4c06e9744685fe2c6108c6d6d78c359ff6eb"} Nov 02 06:45:04.689034 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:04.689014 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-70898f-29893365-c4vxq" event={"ID":"b0419130-0781-47f1-af14-a8eb8917b1eb","Type":"ContainerStarted","Data":"a1200f5c1567c2570e318824f8fad47bb5fe6843743b9e802b64b0bc8efa19ad"} Nov 02 06:45:04.690956 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:04.690930 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-8ecb23-29893365-kb456" event={"ID":"63a2970a-3a1d-4646-8ae6-9432dc18ab15","Type":"ContainerStarted","Data":"e08157c504b01ac71705f38d6e0f1dbf1d014f567440ac33d1823e617d543518"} Nov 02 06:45:04.692844 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:04.692821 5916 generic.go:356] "Generic (PLEG): container finished" podID="f2cac8cd-4ce7-4c70-b380-b9aec98431b8" containerID="a2742a61c2faf2dbad556065f751c74e9ae1f2584d455c8e41d41ca5179d1905" exitCode=60 Nov 02 06:45:04.692942 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:04.692895 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-3e96be-29893365-c9n69" event={"ID":"f2cac8cd-4ce7-4c70-b380-b9aec98431b8","Type":"ContainerDied","Data":"a2742a61c2faf2dbad556065f751c74e9ae1f2584d455c8e41d41ca5179d1905"} Nov 02 06:45:04.693122 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:04.693108 5916 scope.go:122] "RemoveContainer" containerID="a2742a61c2faf2dbad556065f751c74e9ae1f2584d455c8e41d41ca5179d1905" Nov 02 06:45:04.693338 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:45:04.693321 5916 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-3e96be-29893365-c9n69_odh-ai-gateway-infra(f2cac8cd-4ce7-4c70-b380-b9aec98431b8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-3e96be-29893365-c9n69" podUID="f2cac8cd-4ce7-4c70-b380-b9aec98431b8" Nov 02 06:45:04.697777 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:04.697748 5916 scope.go:122] "RemoveContainer" containerID="b44b12ed0ea5ef9a1f8b4965d31e432029e869c4df89f02c117019f21f155a84" Nov 02 06:45:05.703940 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:05.703869 5916 generic.go:356] "Generic (PLEG): container finished" podID="b0419130-0781-47f1-af14-a8eb8917b1eb" containerID="a1200f5c1567c2570e318824f8fad47bb5fe6843743b9e802b64b0bc8efa19ad" exitCode=60 Nov 02 06:45:05.704413 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:05.703977 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-70898f-29893365-c4vxq" event={"ID":"b0419130-0781-47f1-af14-a8eb8917b1eb","Type":"ContainerDied","Data":"a1200f5c1567c2570e318824f8fad47bb5fe6843743b9e802b64b0bc8efa19ad"} Nov 02 06:45:05.704413 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:05.704046 5916 scope.go:122] "RemoveContainer" containerID="2253f44b1b4ceb0a81b15a1ace41a405b8441c0ed0f6b71b37b87f64129452f3" Nov 02 06:45:05.704541 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:05.704443 5916 scope.go:122] "RemoveContainer" containerID="a1200f5c1567c2570e318824f8fad47bb5fe6843743b9e802b64b0bc8efa19ad" Nov 02 06:45:05.704806 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:45:05.704781 5916 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-70898f-29893365-c4vxq_odh-ai-gateway-infra(b0419130-0781-47f1-af14-a8eb8917b1eb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-70898f-29893365-c4vxq" podUID="b0419130-0781-47f1-af14-a8eb8917b1eb" Nov 02 06:45:05.707541 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:05.707512 5916 generic.go:356] "Generic (PLEG): container finished" podID="63a2970a-3a1d-4646-8ae6-9432dc18ab15" containerID="e08157c504b01ac71705f38d6e0f1dbf1d014f567440ac33d1823e617d543518" exitCode=60 Nov 02 06:45:05.707693 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:05.707592 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-8ecb23-29893365-kb456" event={"ID":"63a2970a-3a1d-4646-8ae6-9432dc18ab15","Type":"ContainerDied","Data":"e08157c504b01ac71705f38d6e0f1dbf1d014f567440ac33d1823e617d543518"} Nov 02 06:45:05.708126 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:05.708104 5916 scope.go:122] "RemoveContainer" containerID="e08157c504b01ac71705f38d6e0f1dbf1d014f567440ac33d1823e617d543518" Nov 02 06:45:05.708474 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:45:05.708440 5916 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-8ecb23-29893365-kb456_odh-ai-gateway-infra(63a2970a-3a1d-4646-8ae6-9432dc18ab15)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-8ecb23-29893365-kb456" podUID="63a2970a-3a1d-4646-8ae6-9432dc18ab15" Nov 02 06:45:05.709786 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:05.709759 5916 scope.go:122] "RemoveContainer" containerID="a2742a61c2faf2dbad556065f751c74e9ae1f2584d455c8e41d41ca5179d1905" Nov 02 06:45:05.710108 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:45:05.710085 5916 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-3e96be-29893365-c9n69_odh-ai-gateway-infra(f2cac8cd-4ce7-4c70-b380-b9aec98431b8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-3e96be-29893365-c9n69" podUID="f2cac8cd-4ce7-4c70-b380-b9aec98431b8" Nov 02 06:45:05.711659 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:05.711636 5916 generic.go:356] "Generic (PLEG): container finished" podID="4d559915-7623-421a-9631-d308bf5a8c4e" containerID="8eb4d09cd62501a9ac05d17c810fa48558ae1ecd665f5d19d9c44bf9ae3ae706" exitCode=60 Nov 02 06:45:05.711769 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:05.711716 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-680b7b-29893365-5wmwz" event={"ID":"4d559915-7623-421a-9631-d308bf5a8c4e","Type":"ContainerDied","Data":"8eb4d09cd62501a9ac05d17c810fa48558ae1ecd665f5d19d9c44bf9ae3ae706"} Nov 02 06:45:05.711944 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:05.711928 5916 scope.go:122] "RemoveContainer" containerID="8eb4d09cd62501a9ac05d17c810fa48558ae1ecd665f5d19d9c44bf9ae3ae706" Nov 02 06:45:05.712208 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:45:05.712189 5916 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-680b7b-29893365-5wmwz_odh-ai-gateway-infra(4d559915-7623-421a-9631-d308bf5a8c4e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-680b7b-29893365-5wmwz" podUID="4d559915-7623-421a-9631-d308bf5a8c4e" Nov 02 06:45:05.713714 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:05.713697 5916 scope.go:122] "RemoveContainer" containerID="6ef841088abc3ded98176f238b23fe394af7eeebb1d392a325a1c04115987d9d" Nov 02 06:45:05.713993 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:45:05.713969 5916 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-bdcd80-29893365-4l8lg_odh-ai-gateway-infra(d7494bbe-ebe5-490e-9861-9d573c45f068)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-bdcd80-29893365-4l8lg" podUID="d7494bbe-ebe5-490e-9861-9d573c45f068" Nov 02 06:45:05.715481 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:05.715458 5916 generic.go:356] "Generic (PLEG): container finished" podID="3de4bd72-539c-4fd8-a0a1-c08383426e53" containerID="f12bae89648ace6f4cba382899bfbc69f93b5a14ac1a35625f83e7dc646f5373" exitCode=60 Nov 02 06:45:05.715586 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:05.715539 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-581608-29893365-t2rws" event={"ID":"3de4bd72-539c-4fd8-a0a1-c08383426e53","Type":"ContainerDied","Data":"f12bae89648ace6f4cba382899bfbc69f93b5a14ac1a35625f83e7dc646f5373"} Nov 02 06:45:05.715730 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:05.715715 5916 scope.go:122] "RemoveContainer" containerID="f12bae89648ace6f4cba382899bfbc69f93b5a14ac1a35625f83e7dc646f5373" Nov 02 06:45:05.715994 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:45:05.715974 5916 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-581608-29893365-t2rws_odh-ai-gateway-infra(3de4bd72-539c-4fd8-a0a1-c08383426e53)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-581608-29893365-t2rws" podUID="3de4bd72-539c-4fd8-a0a1-c08383426e53" Nov 02 06:45:05.717427 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:05.717397 5916 generic.go:356] "Generic (PLEG): container finished" podID="fcebf78f-ba62-491d-a987-b784158c1c70" containerID="a67a92896cac5330ffcf79b579ab4c06e9744685fe2c6108c6d6d78c359ff6eb" exitCode=60 Nov 02 06:45:05.717505 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:05.717434 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fb1398-29893365-pqgct" event={"ID":"fcebf78f-ba62-491d-a987-b784158c1c70","Type":"ContainerDied","Data":"a67a92896cac5330ffcf79b579ab4c06e9744685fe2c6108c6d6d78c359ff6eb"} Nov 02 06:45:05.717762 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:05.717745 5916 scope.go:122] "RemoveContainer" containerID="a67a92896cac5330ffcf79b579ab4c06e9744685fe2c6108c6d6d78c359ff6eb" Nov 02 06:45:05.718087 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:45:05.718067 5916 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-fb1398-29893365-pqgct_odh-ai-gateway-infra(fcebf78f-ba62-491d-a987-b784158c1c70)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fb1398-29893365-pqgct" podUID="fcebf78f-ba62-491d-a987-b784158c1c70" Nov 02 06:45:05.720778 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:05.720760 5916 scope.go:122] "RemoveContainer" containerID="7b2bf62dc2b31f9300d157f9971148915ebbf80f5ef4bcbc6457ac2b27944377" Nov 02 06:45:05.735993 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:05.735977 5916 scope.go:122] "RemoveContainer" containerID="a0d3b4c635ea2b362bebcbba3b6b7649b6f40d09ded30d36a2e664a5c7b5b4cd" Nov 02 06:45:05.761493 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:05.761470 5916 scope.go:122] "RemoveContainer" containerID="becc164bd9578ac3eff8113f82db0047b6b19ab07e3f818513a5c9b98a18a400" Nov 02 06:45:05.777664 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:05.777645 5916 scope.go:122] "RemoveContainer" containerID="7ffe0c89c0bb980fee7c36cbf42cc01924cd27b49189faab96403bfb60edac17" Nov 02 06:45:06.729390 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:06.729358 5916 scope.go:122] "RemoveContainer" containerID="8eb4d09cd62501a9ac05d17c810fa48558ae1ecd665f5d19d9c44bf9ae3ae706" Nov 02 06:45:06.729888 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:45:06.729786 5916 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-680b7b-29893365-5wmwz_odh-ai-gateway-infra(4d559915-7623-421a-9631-d308bf5a8c4e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-680b7b-29893365-5wmwz" podUID="4d559915-7623-421a-9631-d308bf5a8c4e" Nov 02 06:45:06.731369 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:06.731340 5916 scope.go:122] "RemoveContainer" containerID="f12bae89648ace6f4cba382899bfbc69f93b5a14ac1a35625f83e7dc646f5373" Nov 02 06:45:06.731786 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:45:06.731756 5916 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-581608-29893365-t2rws_odh-ai-gateway-infra(3de4bd72-539c-4fd8-a0a1-c08383426e53)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-581608-29893365-t2rws" podUID="3de4bd72-539c-4fd8-a0a1-c08383426e53" Nov 02 06:45:06.733288 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:06.733268 5916 scope.go:122] "RemoveContainer" containerID="a67a92896cac5330ffcf79b579ab4c06e9744685fe2c6108c6d6d78c359ff6eb" Nov 02 06:45:06.733610 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:45:06.733580 5916 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-fb1398-29893365-pqgct_odh-ai-gateway-infra(fcebf78f-ba62-491d-a987-b784158c1c70)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fb1398-29893365-pqgct" podUID="fcebf78f-ba62-491d-a987-b784158c1c70" Nov 02 06:45:06.735232 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:06.735210 5916 scope.go:122] "RemoveContainer" containerID="a1200f5c1567c2570e318824f8fad47bb5fe6843743b9e802b64b0bc8efa19ad" Nov 02 06:45:06.735572 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:45:06.735548 5916 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-70898f-29893365-c4vxq_odh-ai-gateway-infra(b0419130-0781-47f1-af14-a8eb8917b1eb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-70898f-29893365-c4vxq" podUID="b0419130-0781-47f1-af14-a8eb8917b1eb" Nov 02 06:45:06.737199 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:06.737167 5916 scope.go:122] "RemoveContainer" containerID="e08157c504b01ac71705f38d6e0f1dbf1d014f567440ac33d1823e617d543518" Nov 02 06:45:06.737504 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:45:06.737480 5916 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-8ecb23-29893365-kb456_odh-ai-gateway-infra(63a2970a-3a1d-4646-8ae6-9432dc18ab15)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-8ecb23-29893365-kb456" podUID="63a2970a-3a1d-4646-8ae6-9432dc18ab15" Nov 02 06:45:15.835753 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:15.835205 5916 scope.go:122] "RemoveContainer" containerID="a1200f5c1567c2570e318824f8fad47bb5fe6843743b9e802b64b0bc8efa19ad" Nov 02 06:45:15.835753 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:15.835354 5916 scope.go:122] "RemoveContainer" containerID="a2742a61c2faf2dbad556065f751c74e9ae1f2584d455c8e41d41ca5179d1905" Nov 02 06:45:15.835753 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:15.835414 5916 scope.go:122] "RemoveContainer" containerID="8eb4d09cd62501a9ac05d17c810fa48558ae1ecd665f5d19d9c44bf9ae3ae706" Nov 02 06:45:15.836376 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:15.836088 5916 scope.go:122] "RemoveContainer" containerID="6ef841088abc3ded98176f238b23fe394af7eeebb1d392a325a1c04115987d9d" Nov 02 06:45:16.101132 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:16.101039 5916 patch_prober.go:29] interesting pod/e2e-shared-a-w2-08bbf9-openshift-default-5479f6b7c-d2hk2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 06:45:16.101132 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:16.101112 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-08bbf9-openshift-default-5479f6b7c-d2hk2" podUID="1b46fdb7-008c-45d7-92cb-8e392c54547f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 06:45:16.835760 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:16.835675 5916 scope.go:122] "RemoveContainer" containerID="a67a92896cac5330ffcf79b579ab4c06e9744685fe2c6108c6d6d78c359ff6eb" Nov 02 06:45:16.842890 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:16.842853 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-70898f-29893365-c4vxq" event={"ID":"b0419130-0781-47f1-af14-a8eb8917b1eb","Type":"ContainerStarted","Data":"40f75155763fdc2f97b1e82ff65867830c5b65714dc840665d23c1847517c5f9"} Nov 02 06:45:16.846127 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:16.846101 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-3e96be-29893365-c9n69" event={"ID":"f2cac8cd-4ce7-4c70-b380-b9aec98431b8","Type":"ContainerStarted","Data":"6ea9c577466772ce7b2fc0344c088ef0291b786d5992ed891bffc7e3c9772e00"} Nov 02 06:45:16.848761 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:16.848741 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-680b7b-29893365-5wmwz" event={"ID":"4d559915-7623-421a-9631-d308bf5a8c4e","Type":"ContainerStarted","Data":"0280699a2cfd582ba672c8967b2f60814346ab561da7d60168041900b00daa82"} Nov 02 06:45:16.852462 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:16.852436 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-bdcd80-29893365-4l8lg" event={"ID":"d7494bbe-ebe5-490e-9861-9d573c45f068","Type":"ContainerStarted","Data":"dcc0849b4c53fb1f70027b16d2a9806197f02afd7992ed38a848e3a81244684a"} Nov 02 06:45:17.835106 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:17.835067 5916 scope.go:122] "RemoveContainer" containerID="e08157c504b01ac71705f38d6e0f1dbf1d014f567440ac33d1823e617d543518" Nov 02 06:45:17.867815 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:17.867778 5916 generic.go:356] "Generic (PLEG): container finished" podID="b0419130-0781-47f1-af14-a8eb8917b1eb" containerID="40f75155763fdc2f97b1e82ff65867830c5b65714dc840665d23c1847517c5f9" exitCode=60 Nov 02 06:45:17.868308 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:17.867939 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-70898f-29893365-c4vxq" event={"ID":"b0419130-0781-47f1-af14-a8eb8917b1eb","Type":"ContainerDied","Data":"40f75155763fdc2f97b1e82ff65867830c5b65714dc840665d23c1847517c5f9"} Nov 02 06:45:17.868308 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:17.867996 5916 scope.go:122] "RemoveContainer" containerID="a1200f5c1567c2570e318824f8fad47bb5fe6843743b9e802b64b0bc8efa19ad" Nov 02 06:45:17.868481 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:17.868459 5916 scope.go:122] "RemoveContainer" containerID="40f75155763fdc2f97b1e82ff65867830c5b65714dc840665d23c1847517c5f9" Nov 02 06:45:17.869072 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:45:17.868812 5916 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-70898f-29893365-c4vxq_odh-ai-gateway-infra(b0419130-0781-47f1-af14-a8eb8917b1eb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-70898f-29893365-c4vxq" podUID="b0419130-0781-47f1-af14-a8eb8917b1eb" Nov 02 06:45:17.870645 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:17.870620 5916 generic.go:356] "Generic (PLEG): container finished" podID="f2cac8cd-4ce7-4c70-b380-b9aec98431b8" containerID="6ea9c577466772ce7b2fc0344c088ef0291b786d5992ed891bffc7e3c9772e00" exitCode=60 Nov 02 06:45:17.870804 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:17.870783 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-3e96be-29893365-c9n69" event={"ID":"f2cac8cd-4ce7-4c70-b380-b9aec98431b8","Type":"ContainerDied","Data":"6ea9c577466772ce7b2fc0344c088ef0291b786d5992ed891bffc7e3c9772e00"} Nov 02 06:45:17.871175 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:17.871148 5916 scope.go:122] "RemoveContainer" containerID="6ea9c577466772ce7b2fc0344c088ef0291b786d5992ed891bffc7e3c9772e00" Nov 02 06:45:17.871660 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:45:17.871444 5916 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-3e96be-29893365-c9n69_odh-ai-gateway-infra(f2cac8cd-4ce7-4c70-b380-b9aec98431b8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-3e96be-29893365-c9n69" podUID="f2cac8cd-4ce7-4c70-b380-b9aec98431b8" Nov 02 06:45:17.876401 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:17.873413 5916 generic.go:356] "Generic (PLEG): container finished" podID="4d559915-7623-421a-9631-d308bf5a8c4e" containerID="0280699a2cfd582ba672c8967b2f60814346ab561da7d60168041900b00daa82" exitCode=60 Nov 02 06:45:17.876401 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:17.873499 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-680b7b-29893365-5wmwz" event={"ID":"4d559915-7623-421a-9631-d308bf5a8c4e","Type":"ContainerDied","Data":"0280699a2cfd582ba672c8967b2f60814346ab561da7d60168041900b00daa82"} Nov 02 06:45:17.876401 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:17.874011 5916 scope.go:122] "RemoveContainer" containerID="0280699a2cfd582ba672c8967b2f60814346ab561da7d60168041900b00daa82" Nov 02 06:45:17.876401 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:45:17.874316 5916 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-680b7b-29893365-5wmwz_odh-ai-gateway-infra(4d559915-7623-421a-9631-d308bf5a8c4e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-680b7b-29893365-5wmwz" podUID="4d559915-7623-421a-9631-d308bf5a8c4e" Nov 02 06:45:17.877546 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:17.877524 5916 generic.go:356] "Generic (PLEG): container finished" podID="d7494bbe-ebe5-490e-9861-9d573c45f068" containerID="dcc0849b4c53fb1f70027b16d2a9806197f02afd7992ed38a848e3a81244684a" exitCode=60 Nov 02 06:45:17.877717 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:17.877701 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-bdcd80-29893365-4l8lg" event={"ID":"d7494bbe-ebe5-490e-9861-9d573c45f068","Type":"ContainerDied","Data":"dcc0849b4c53fb1f70027b16d2a9806197f02afd7992ed38a848e3a81244684a"} Nov 02 06:45:17.878042 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:17.878025 5916 scope.go:122] "RemoveContainer" containerID="dcc0849b4c53fb1f70027b16d2a9806197f02afd7992ed38a848e3a81244684a" Nov 02 06:45:17.878341 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:45:17.878318 5916 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-bdcd80-29893365-4l8lg_odh-ai-gateway-infra(d7494bbe-ebe5-490e-9861-9d573c45f068)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-bdcd80-29893365-4l8lg" podUID="d7494bbe-ebe5-490e-9861-9d573c45f068" Nov 02 06:45:17.880711 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:17.880690 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fb1398-29893365-pqgct" event={"ID":"fcebf78f-ba62-491d-a987-b784158c1c70","Type":"ContainerStarted","Data":"3ffd518dc39cdbdaa4e261b16876c566fb767a04ec44d8a4911049a12a47e840"} Nov 02 06:45:17.939242 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:17.939221 5916 scope.go:122] "RemoveContainer" containerID="a2742a61c2faf2dbad556065f751c74e9ae1f2584d455c8e41d41ca5179d1905" Nov 02 06:45:17.972935 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:17.972865 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-70898f-29893365-c4vxq"] Nov 02 06:45:17.984438 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:17.984413 5916 scope.go:122] "RemoveContainer" containerID="8eb4d09cd62501a9ac05d17c810fa48558ae1ecd665f5d19d9c44bf9ae3ae706" Nov 02 06:45:18.036412 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:18.036385 5916 scope.go:122] "RemoveContainer" containerID="6ef841088abc3ded98176f238b23fe394af7eeebb1d392a325a1c04115987d9d" Nov 02 06:45:18.093110 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:18.093079 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-680b7b-29893365-5wmwz"] Nov 02 06:45:18.223413 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:18.223384 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-bdcd80-29893365-4l8lg"] Nov 02 06:45:18.331850 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:18.331817 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-3e96be-29893365-c9n69"] Nov 02 06:45:18.835286 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:18.835252 5916 scope.go:122] "RemoveContainer" containerID="f12bae89648ace6f4cba382899bfbc69f93b5a14ac1a35625f83e7dc646f5373" Nov 02 06:45:18.893361 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:18.893327 5916 generic.go:356] "Generic (PLEG): container finished" podID="fcebf78f-ba62-491d-a987-b784158c1c70" containerID="3ffd518dc39cdbdaa4e261b16876c566fb767a04ec44d8a4911049a12a47e840" exitCode=60 Nov 02 06:45:18.893744 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:18.893393 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fb1398-29893365-pqgct" event={"ID":"fcebf78f-ba62-491d-a987-b784158c1c70","Type":"ContainerDied","Data":"3ffd518dc39cdbdaa4e261b16876c566fb767a04ec44d8a4911049a12a47e840"} Nov 02 06:45:18.893744 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:18.893454 5916 scope.go:122] "RemoveContainer" containerID="a67a92896cac5330ffcf79b579ab4c06e9744685fe2c6108c6d6d78c359ff6eb" Nov 02 06:45:18.893895 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:18.893876 5916 scope.go:122] "RemoveContainer" containerID="3ffd518dc39cdbdaa4e261b16876c566fb767a04ec44d8a4911049a12a47e840" Nov 02 06:45:18.894325 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:45:18.894290 5916 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-fb1398-29893365-pqgct_odh-ai-gateway-infra(fcebf78f-ba62-491d-a987-b784158c1c70)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fb1398-29893365-pqgct" podUID="fcebf78f-ba62-491d-a987-b784158c1c70" Nov 02 06:45:18.897575 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:18.897548 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-8ecb23-29893365-kb456" event={"ID":"63a2970a-3a1d-4646-8ae6-9432dc18ab15","Type":"ContainerStarted","Data":"678b5d4ae3c447d07a5c1240ead5560dd6af8c3fd87a454aa599870cf13cefee"} Nov 02 06:45:18.901796 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:18.901774 5916 generic.go:356] "Generic (PLEG): container finished" podID="1b46fdb7-008c-45d7-92cb-8e392c54547f" containerID="ef905e16b3a5fd3ea80cf5a33b5275a7425d95738fb1e82d4afbd99a49491dfd" exitCode=0 Nov 02 06:45:18.901898 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:18.901847 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-08bbf9-openshift-default-5479f6b7c-d2hk2" event={"ID":"1b46fdb7-008c-45d7-92cb-8e392c54547f","Type":"ContainerDied","Data":"ef905e16b3a5fd3ea80cf5a33b5275a7425d95738fb1e82d4afbd99a49491dfd"} Nov 02 06:45:18.902285 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:18.902266 5916 scope.go:122] "RemoveContainer" containerID="ef905e16b3a5fd3ea80cf5a33b5275a7425d95738fb1e82d4afbd99a49491dfd" Nov 02 06:45:18.904022 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:18.903995 5916 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 06:45:18.904129 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:18.904074 5916 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 06:45:18.904129 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:18.904121 5916 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 06:45:19.119078 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:19.119051 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fb1398-29893365-pqgct"] Nov 02 06:45:19.307114 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:19.307072 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-bdcd80-29893365-4l8lg" Nov 02 06:45:19.314558 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:19.314430 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-3e96be-29893365-c9n69" Nov 02 06:45:19.326328 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:19.325674 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-70898f-29893365-c4vxq" Nov 02 06:45:19.327838 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:19.327815 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-680b7b-29893365-5wmwz" Nov 02 06:45:19.463008 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:19.462562 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0419130-0781-47f1-af14-a8eb8917b1eb-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0419130-0781-47f1-af14-a8eb8917b1eb-maas-api-service-ca\") pod \"b0419130-0781-47f1-af14-a8eb8917b1eb\" (UID: \"b0419130-0781-47f1-af14-a8eb8917b1eb\") " Nov 02 06:45:19.463334 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:19.463314 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2cac8cd-4ce7-4c70-b380-b9aec98431b8-kube-api-access-ghjgn\" (UniqueName: \"kubernetes.io/projected/f2cac8cd-4ce7-4c70-b380-b9aec98431b8-kube-api-access-ghjgn\") pod \"f2cac8cd-4ce7-4c70-b380-b9aec98431b8\" (UID: \"f2cac8cd-4ce7-4c70-b380-b9aec98431b8\") " Nov 02 06:45:19.463957 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:19.462959 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0419130-0781-47f1-af14-a8eb8917b1eb-maas-api-service-ca" pod "b0419130-0781-47f1-af14-a8eb8917b1eb" (UID: "b0419130-0781-47f1-af14-a8eb8917b1eb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:45:19.464119 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:19.464103 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2cac8cd-4ce7-4c70-b380-b9aec98431b8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2cac8cd-4ce7-4c70-b380-b9aec98431b8-maas-api-service-ca\") pod \"f2cac8cd-4ce7-4c70-b380-b9aec98431b8\" (UID: \"f2cac8cd-4ce7-4c70-b380-b9aec98431b8\") " Nov 02 06:45:19.464381 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:19.464356 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2cac8cd-4ce7-4c70-b380-b9aec98431b8-maas-api-service-ca" pod "f2cac8cd-4ce7-4c70-b380-b9aec98431b8" (UID: "f2cac8cd-4ce7-4c70-b380-b9aec98431b8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:45:19.464523 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:19.464506 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d559915-7623-421a-9631-d308bf5a8c4e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d559915-7623-421a-9631-d308bf5a8c4e-maas-api-service-ca\") pod \"4d559915-7623-421a-9631-d308bf5a8c4e\" (UID: \"4d559915-7623-421a-9631-d308bf5a8c4e\") " Nov 02 06:45:19.464983 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:19.464966 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0419130-0781-47f1-af14-a8eb8917b1eb-kube-api-access-gmlvx\" (UniqueName: \"kubernetes.io/projected/b0419130-0781-47f1-af14-a8eb8917b1eb-kube-api-access-gmlvx\") pod \"b0419130-0781-47f1-af14-a8eb8917b1eb\" (UID: \"b0419130-0781-47f1-af14-a8eb8917b1eb\") " Nov 02 06:45:19.465486 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:19.465468 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7494bbe-ebe5-490e-9861-9d573c45f068-kube-api-access-f6hvq\" (UniqueName: \"kubernetes.io/projected/d7494bbe-ebe5-490e-9861-9d573c45f068-kube-api-access-f6hvq\") pod \"d7494bbe-ebe5-490e-9861-9d573c45f068\" (UID: \"d7494bbe-ebe5-490e-9861-9d573c45f068\") " Nov 02 06:45:19.466123 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:19.464877 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d559915-7623-421a-9631-d308bf5a8c4e-maas-api-service-ca" pod "4d559915-7623-421a-9631-d308bf5a8c4e" (UID: "4d559915-7623-421a-9631-d308bf5a8c4e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:45:19.466310 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:19.466285 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7494bbe-ebe5-490e-9861-9d573c45f068-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7494bbe-ebe5-490e-9861-9d573c45f068-maas-api-service-ca\") pod \"d7494bbe-ebe5-490e-9861-9d573c45f068\" (UID: \"d7494bbe-ebe5-490e-9861-9d573c45f068\") " Nov 02 06:45:19.466954 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:19.466936 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d559915-7623-421a-9631-d308bf5a8c4e-kube-api-access-p4whc\" (UniqueName: \"kubernetes.io/projected/4d559915-7623-421a-9631-d308bf5a8c4e-kube-api-access-p4whc\") pod \"4d559915-7623-421a-9631-d308bf5a8c4e\" (UID: \"4d559915-7623-421a-9631-d308bf5a8c4e\") " Nov 02 06:45:19.467518 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:19.466868 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7494bbe-ebe5-490e-9861-9d573c45f068-maas-api-service-ca" pod "d7494bbe-ebe5-490e-9861-9d573c45f068" (UID: "d7494bbe-ebe5-490e-9861-9d573c45f068"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:45:19.467791 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:19.467762 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2cac8cd-4ce7-4c70-b380-b9aec98431b8-kube-api-access-ghjgn" pod "f2cac8cd-4ce7-4c70-b380-b9aec98431b8" (UID: "f2cac8cd-4ce7-4c70-b380-b9aec98431b8"). InnerVolumeSpecName "kube-api-access-ghjgn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:45:19.468601 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:19.468581 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0419130-0781-47f1-af14-a8eb8917b1eb-maas-api-service-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:45:19.468710 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:19.468696 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghjgn\" (UniqueName: \"kubernetes.io/projected/f2cac8cd-4ce7-4c70-b380-b9aec98431b8-kube-api-access-ghjgn\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:45:19.468810 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:19.468799 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2cac8cd-4ce7-4c70-b380-b9aec98431b8-maas-api-service-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:45:19.468885 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:19.468874 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d559915-7623-421a-9631-d308bf5a8c4e-maas-api-service-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:45:19.469012 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:19.469000 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7494bbe-ebe5-490e-9861-9d573c45f068-maas-api-service-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:45:19.471411 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:19.471376 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7494bbe-ebe5-490e-9861-9d573c45f068-kube-api-access-f6hvq" pod "d7494bbe-ebe5-490e-9861-9d573c45f068" (UID: "d7494bbe-ebe5-490e-9861-9d573c45f068"). InnerVolumeSpecName "kube-api-access-f6hvq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:45:19.471679 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:19.471646 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0419130-0781-47f1-af14-a8eb8917b1eb-kube-api-access-gmlvx" pod "b0419130-0781-47f1-af14-a8eb8917b1eb" (UID: "b0419130-0781-47f1-af14-a8eb8917b1eb"). InnerVolumeSpecName "kube-api-access-gmlvx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:45:19.473272 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:19.473245 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d559915-7623-421a-9631-d308bf5a8c4e-kube-api-access-p4whc" pod "4d559915-7623-421a-9631-d308bf5a8c4e" (UID: "4d559915-7623-421a-9631-d308bf5a8c4e"). InnerVolumeSpecName "kube-api-access-p4whc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:45:19.572840 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:19.572803 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmlvx\" (UniqueName: \"kubernetes.io/projected/b0419130-0781-47f1-af14-a8eb8917b1eb-kube-api-access-gmlvx\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:45:19.573029 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:19.572839 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6hvq\" (UniqueName: \"kubernetes.io/projected/d7494bbe-ebe5-490e-9861-9d573c45f068-kube-api-access-f6hvq\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:45:19.573029 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:19.572884 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4whc\" (UniqueName: \"kubernetes.io/projected/4d559915-7623-421a-9631-d308bf5a8c4e-kube-api-access-p4whc\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:45:19.922039 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:19.922003 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-70898f-29893365-c4vxq" event={"ID":"b0419130-0781-47f1-af14-a8eb8917b1eb","Type":"ContainerDied","Data":"b00381abec244fabd0d37eef09e8b03a23c5b3f509b6de61d13ef2f80e913d46"} Nov 02 06:45:19.922547 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:19.922070 5916 scope.go:122] "RemoveContainer" containerID="40f75155763fdc2f97b1e82ff65867830c5b65714dc840665d23c1847517c5f9" Nov 02 06:45:19.922547 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:19.922208 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-70898f-29893365-c4vxq" Nov 02 06:45:19.925574 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:45:19.924306 5916 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p98cq" containerName="extractor" Nov 02 06:45:19.947213 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:19.946675 5916 generic.go:356] "Generic (PLEG): container finished" podID="63a2970a-3a1d-4646-8ae6-9432dc18ab15" containerID="678b5d4ae3c447d07a5c1240ead5560dd6af8c3fd87a454aa599870cf13cefee" exitCode=60 Nov 02 06:45:19.947213 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:19.946782 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-8ecb23-29893365-kb456" event={"ID":"63a2970a-3a1d-4646-8ae6-9432dc18ab15","Type":"ContainerDied","Data":"678b5d4ae3c447d07a5c1240ead5560dd6af8c3fd87a454aa599870cf13cefee"} Nov 02 06:45:19.947213 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:19.946838 5916 scope.go:122] "RemoveContainer" containerID="e08157c504b01ac71705f38d6e0f1dbf1d014f567440ac33d1823e617d543518" Nov 02 06:45:19.950044 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:19.947890 5916 scope.go:122] "RemoveContainer" containerID="678b5d4ae3c447d07a5c1240ead5560dd6af8c3fd87a454aa599870cf13cefee" Nov 02 06:45:19.950044 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:45:19.948399 5916 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-8ecb23-29893365-kb456_odh-ai-gateway-infra(63a2970a-3a1d-4646-8ae6-9432dc18ab15)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-8ecb23-29893365-kb456" podUID="63a2970a-3a1d-4646-8ae6-9432dc18ab15" Nov 02 06:45:19.964004 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:19.963181 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-3e96be-29893365-c9n69" event={"ID":"f2cac8cd-4ce7-4c70-b380-b9aec98431b8","Type":"ContainerDied","Data":"6ee1440163f477f802993a2ecaf0d1ffe3a809bbcdabd0710aeac5e1f8e549d6"} Nov 02 06:45:19.964004 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:19.963304 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-3e96be-29893365-c9n69" Nov 02 06:45:19.971443 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:19.971412 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-08bbf9-openshift-default-5479f6b7c-d2hk2" event={"ID":"1b46fdb7-008c-45d7-92cb-8e392c54547f","Type":"ContainerStarted","Data":"99f1370b5029fee6fc8ee76ecc8c970f413ecd3fd0cda3c3accce9ca2f81f183"} Nov 02 06:45:19.974578 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:19.973866 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-680b7b-29893365-5wmwz" event={"ID":"4d559915-7623-421a-9631-d308bf5a8c4e","Type":"ContainerDied","Data":"e0e0de2d0f8ef04111735781be3a7c4e927b5697a6cd45166f5a06f2447279df"} Nov 02 06:45:19.974578 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:19.973967 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-680b7b-29893365-5wmwz" Nov 02 06:45:19.986759 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:19.983200 5916 scope.go:122] "RemoveContainer" containerID="6ea9c577466772ce7b2fc0344c088ef0291b786d5992ed891bffc7e3c9772e00" Nov 02 06:45:19.990114 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:19.990093 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-bdcd80-29893365-4l8lg" event={"ID":"d7494bbe-ebe5-490e-9861-9d573c45f068","Type":"ContainerDied","Data":"07cef68d7a08d31b16f00462263070e9429d8eb065d3ef0b9a7f4d31cec5d074"} Nov 02 06:45:19.990229 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:19.990158 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-bdcd80-29893365-4l8lg" Nov 02 06:45:19.996358 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:19.995498 5916 generic.go:356] "Generic (PLEG): container finished" podID="3de4bd72-539c-4fd8-a0a1-c08383426e53" containerID="a54bfb40f340825c64983a09982e66c389a57b54356e39250c512ec2855359ee" exitCode=60 Nov 02 06:45:19.996358 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:19.995588 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-581608-29893365-t2rws" event={"ID":"3de4bd72-539c-4fd8-a0a1-c08383426e53","Type":"ContainerDied","Data":"a54bfb40f340825c64983a09982e66c389a57b54356e39250c512ec2855359ee"} Nov 02 06:45:19.996358 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:19.995867 5916 scope.go:122] "RemoveContainer" containerID="a54bfb40f340825c64983a09982e66c389a57b54356e39250c512ec2855359ee" Nov 02 06:45:19.996358 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:45:19.996106 5916 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-581608-29893365-t2rws_odh-ai-gateway-infra(3de4bd72-539c-4fd8-a0a1-c08383426e53)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-581608-29893365-t2rws" podUID="3de4bd72-539c-4fd8-a0a1-c08383426e53" Nov 02 06:45:20.002204 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:20.002184 5916 scope.go:122] "RemoveContainer" containerID="0280699a2cfd582ba672c8967b2f60814346ab561da7d60168041900b00daa82" Nov 02 06:45:20.015558 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:20.015540 5916 scope.go:122] "RemoveContainer" containerID="dcc0849b4c53fb1f70027b16d2a9806197f02afd7992ed38a848e3a81244684a" Nov 02 06:45:20.060050 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:20.059679 5916 scope.go:122] "RemoveContainer" containerID="f12bae89648ace6f4cba382899bfbc69f93b5a14ac1a35625f83e7dc646f5373" Nov 02 06:45:20.277540 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:20.277503 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-8ecb23-29893365-kb456"] Nov 02 06:45:20.374706 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:20.374648 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fb1398-29893365-pqgct" Nov 02 06:45:20.456932 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:20.444257 5916 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-08bbf9-openshift-default-5479f6b7c-d2hk2" Nov 02 06:45:20.456932 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:20.444377 5916 patch_prober.go:29] interesting pod/e2e-shared-a-w2-08bbf9-openshift-default-5479f6b7c-d2hk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Nov 02 06:45:20.456932 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:20.444419 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-08bbf9-openshift-default-5479f6b7c-d2hk2" podUID="1b46fdb7-008c-45d7-92cb-8e392c54547f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Nov 02 06:45:20.467097 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:20.467051 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-70898f-29893365-c4vxq"] Nov 02 06:45:20.471418 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:20.471390 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-70898f-29893365-c4vxq"] Nov 02 06:45:20.487457 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:20.487428 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcebf78f-ba62-491d-a987-b784158c1c70-kube-api-access-fr9w6\" (UniqueName: \"kubernetes.io/projected/fcebf78f-ba62-491d-a987-b784158c1c70-kube-api-access-fr9w6\") pod \"fcebf78f-ba62-491d-a987-b784158c1c70\" (UID: \"fcebf78f-ba62-491d-a987-b784158c1c70\") " Nov 02 06:45:20.487596 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:20.487531 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fcebf78f-ba62-491d-a987-b784158c1c70-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcebf78f-ba62-491d-a987-b784158c1c70-maas-api-service-ca\") pod \"fcebf78f-ba62-491d-a987-b784158c1c70\" (UID: \"fcebf78f-ba62-491d-a987-b784158c1c70\") " Nov 02 06:45:20.487896 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:20.487869 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcebf78f-ba62-491d-a987-b784158c1c70-maas-api-service-ca" pod "fcebf78f-ba62-491d-a987-b784158c1c70" (UID: "fcebf78f-ba62-491d-a987-b784158c1c70"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:45:20.488077 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:20.488063 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcebf78f-ba62-491d-a987-b784158c1c70-maas-api-service-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:45:20.489603 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:20.489579 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcebf78f-ba62-491d-a987-b784158c1c70-kube-api-access-fr9w6" pod "fcebf78f-ba62-491d-a987-b784158c1c70" (UID: "fcebf78f-ba62-491d-a987-b784158c1c70"). InnerVolumeSpecName "kube-api-access-fr9w6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:45:20.590090 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:20.590051 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fr9w6\" (UniqueName: \"kubernetes.io/projected/fcebf78f-ba62-491d-a987-b784158c1c70-kube-api-access-fr9w6\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:45:20.974688 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:20.974652 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-680b7b-29893365-5wmwz"] Nov 02 06:45:20.980765 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:20.980731 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-680b7b-29893365-5wmwz"] Nov 02 06:45:21.017423 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:21.017396 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fb1398-29893365-pqgct" Nov 02 06:45:21.017583 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:21.017439 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fb1398-29893365-pqgct" event={"ID":"fcebf78f-ba62-491d-a987-b784158c1c70","Type":"ContainerDied","Data":"22839b2c5f894512076d86808d849c4e46a893e70b2d11aa14481146a39189c2"} Nov 02 06:45:21.017583 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:21.017495 5916 scope.go:122] "RemoveContainer" containerID="3ffd518dc39cdbdaa4e261b16876c566fb767a04ec44d8a4911049a12a47e840" Nov 02 06:45:21.084491 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:21.084459 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-bdcd80-29893365-4l8lg"] Nov 02 06:45:21.096069 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:21.096033 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-bdcd80-29893365-4l8lg"] Nov 02 06:45:21.212899 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:21.212862 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-3e96be-29893365-c9n69"] Nov 02 06:45:21.219585 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:21.219558 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-3e96be-29893365-c9n69"] Nov 02 06:45:21.300437 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:21.300411 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-8ecb23-29893365-kb456" Nov 02 06:45:21.400811 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:21.400764 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63a2970a-3a1d-4646-8ae6-9432dc18ab15-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63a2970a-3a1d-4646-8ae6-9432dc18ab15-maas-api-service-ca\") pod \"63a2970a-3a1d-4646-8ae6-9432dc18ab15\" (UID: \"63a2970a-3a1d-4646-8ae6-9432dc18ab15\") " Nov 02 06:45:21.401031 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:21.400820 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63a2970a-3a1d-4646-8ae6-9432dc18ab15-kube-api-access-rszq8\" (UniqueName: \"kubernetes.io/projected/63a2970a-3a1d-4646-8ae6-9432dc18ab15-kube-api-access-rszq8\") pod \"63a2970a-3a1d-4646-8ae6-9432dc18ab15\" (UID: \"63a2970a-3a1d-4646-8ae6-9432dc18ab15\") " Nov 02 06:45:21.401154 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:21.401128 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63a2970a-3a1d-4646-8ae6-9432dc18ab15-maas-api-service-ca" pod "63a2970a-3a1d-4646-8ae6-9432dc18ab15" (UID: "63a2970a-3a1d-4646-8ae6-9432dc18ab15"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:45:21.401453 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:21.401434 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63a2970a-3a1d-4646-8ae6-9432dc18ab15-maas-api-service-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:45:21.402967 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:21.402943 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63a2970a-3a1d-4646-8ae6-9432dc18ab15-kube-api-access-rszq8" pod "63a2970a-3a1d-4646-8ae6-9432dc18ab15" (UID: "63a2970a-3a1d-4646-8ae6-9432dc18ab15"). InnerVolumeSpecName "kube-api-access-rszq8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:45:21.455929 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:21.455641 5916 patch_prober.go:29] interesting pod/e2e-shared-a-w2-08bbf9-openshift-default-5479f6b7c-d2hk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Nov 02 06:45:21.455929 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:21.455703 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-08bbf9-openshift-default-5479f6b7c-d2hk2" podUID="1b46fdb7-008c-45d7-92cb-8e392c54547f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Nov 02 06:45:21.503691 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:21.503651 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rszq8\" (UniqueName: \"kubernetes.io/projected/63a2970a-3a1d-4646-8ae6-9432dc18ab15-kube-api-access-rszq8\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:45:21.520035 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:21.519955 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fb1398-29893365-pqgct"] Nov 02 06:45:21.567965 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:21.563402 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fb1398-29893365-pqgct"] Nov 02 06:45:21.592520 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:21.592487 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-581608-29893365-t2rws"] Nov 02 06:45:21.882450 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:21.882409 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d559915-7623-421a-9631-d308bf5a8c4e" path="/var/lib/kubelet/pods/4d559915-7623-421a-9631-d308bf5a8c4e/volumes" Nov 02 06:45:21.883037 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:21.883016 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0419130-0781-47f1-af14-a8eb8917b1eb" path="/var/lib/kubelet/pods/b0419130-0781-47f1-af14-a8eb8917b1eb/volumes" Nov 02 06:45:21.883502 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:21.883483 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7494bbe-ebe5-490e-9861-9d573c45f068" path="/var/lib/kubelet/pods/d7494bbe-ebe5-490e-9861-9d573c45f068/volumes" Nov 02 06:45:21.884281 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:21.884249 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2cac8cd-4ce7-4c70-b380-b9aec98431b8" path="/var/lib/kubelet/pods/f2cac8cd-4ce7-4c70-b380-b9aec98431b8/volumes" Nov 02 06:45:21.884696 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:21.884675 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fcebf78f-ba62-491d-a987-b784158c1c70" path="/var/lib/kubelet/pods/fcebf78f-ba62-491d-a987-b784158c1c70/volumes" Nov 02 06:45:21.909107 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:21.909088 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-581608-29893365-t2rws" Nov 02 06:45:22.011261 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:22.011228 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3de4bd72-539c-4fd8-a0a1-c08383426e53-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3de4bd72-539c-4fd8-a0a1-c08383426e53-maas-api-service-ca\") pod \"3de4bd72-539c-4fd8-a0a1-c08383426e53\" (UID: \"3de4bd72-539c-4fd8-a0a1-c08383426e53\") " Nov 02 06:45:22.011650 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:22.011389 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3de4bd72-539c-4fd8-a0a1-c08383426e53-kube-api-access-6xbsd\" (UniqueName: \"kubernetes.io/projected/3de4bd72-539c-4fd8-a0a1-c08383426e53-kube-api-access-6xbsd\") pod \"3de4bd72-539c-4fd8-a0a1-c08383426e53\" (UID: \"3de4bd72-539c-4fd8-a0a1-c08383426e53\") " Nov 02 06:45:22.011650 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:22.011623 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3de4bd72-539c-4fd8-a0a1-c08383426e53-maas-api-service-ca" pod "3de4bd72-539c-4fd8-a0a1-c08383426e53" (UID: "3de4bd72-539c-4fd8-a0a1-c08383426e53"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:45:22.012245 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:22.012224 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3de4bd72-539c-4fd8-a0a1-c08383426e53-maas-api-service-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:45:22.013503 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:22.013483 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3de4bd72-539c-4fd8-a0a1-c08383426e53-kube-api-access-6xbsd" pod "3de4bd72-539c-4fd8-a0a1-c08383426e53" (UID: "3de4bd72-539c-4fd8-a0a1-c08383426e53"). InnerVolumeSpecName "kube-api-access-6xbsd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:45:22.069930 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:22.061776 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-581608-29893365-t2rws" event={"ID":"3de4bd72-539c-4fd8-a0a1-c08383426e53","Type":"ContainerDied","Data":"09d1fa8fb4abba3921e64ee0d56d69a017469d97903df005d0d87ae3254f9fb9"} Nov 02 06:45:22.069930 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:22.061847 5916 scope.go:122] "RemoveContainer" containerID="a54bfb40f340825c64983a09982e66c389a57b54356e39250c512ec2855359ee" Nov 02 06:45:22.069930 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:22.062004 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-581608-29893365-t2rws" Nov 02 06:45:22.082950 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:22.082797 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-8ecb23-29893365-kb456" event={"ID":"63a2970a-3a1d-4646-8ae6-9432dc18ab15","Type":"ContainerDied","Data":"1813cc5503456da1f2412c7ecdd370dccc31352f24c0f8af496d9b702fc3bf8a"} Nov 02 06:45:22.082950 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:22.082947 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-8ecb23-29893365-kb456" Nov 02 06:45:22.091310 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:22.091289 5916 scope.go:122] "RemoveContainer" containerID="678b5d4ae3c447d07a5c1240ead5560dd6af8c3fd87a454aa599870cf13cefee" Nov 02 06:45:22.114023 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:22.113994 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xbsd\" (UniqueName: \"kubernetes.io/projected/3de4bd72-539c-4fd8-a0a1-c08383426e53-kube-api-access-6xbsd\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:45:22.272091 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:22.272008 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-8ecb23-29893365-kb456"] Nov 02 06:45:22.287099 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:22.287000 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-8ecb23-29893365-kb456"] Nov 02 06:45:22.383130 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:22.383097 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-581608-29893365-t2rws"] Nov 02 06:45:22.388738 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:22.388714 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-581608-29893365-t2rws"] Nov 02 06:45:22.439926 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:22.436232 5916 patch_prober.go:29] interesting pod/e2e-shared-a-w2-08bbf9-openshift-default-5479f6b7c-d2hk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Nov 02 06:45:22.439926 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:22.436291 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-08bbf9-openshift-default-5479f6b7c-d2hk2" podUID="1b46fdb7-008c-45d7-92cb-8e392c54547f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Nov 02 06:45:23.444930 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:23.443003 5916 patch_prober.go:29] interesting pod/e2e-shared-a-w2-08bbf9-openshift-default-5479f6b7c-d2hk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Nov 02 06:45:23.444930 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:23.443066 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-08bbf9-openshift-default-5479f6b7c-d2hk2" podUID="1b46fdb7-008c-45d7-92cb-8e392c54547f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Nov 02 06:45:23.480424 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:23.480388 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3bf02ebf-whd2b"] Nov 02 06:45:23.481042 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:23.481028 5916 cpu_manager.go:414] "containerMap: removing container" podUID="fcebf78f-ba62-491d-a987-b784158c1c70" containerName="cleanup" Nov 02 06:45:23.481042 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:23.481043 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcebf78f-ba62-491d-a987-b784158c1c70" containerName="cleanup" Nov 02 06:45:23.481149 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:23.481053 5916 cpu_manager.go:414] "containerMap: removing container" podUID="b0419130-0781-47f1-af14-a8eb8917b1eb" containerName="cleanup" Nov 02 06:45:23.481149 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:23.481058 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0419130-0781-47f1-af14-a8eb8917b1eb" containerName="cleanup" Nov 02 06:45:23.481149 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:23.481070 5916 cpu_manager.go:414] "containerMap: removing container" podUID="fcebf78f-ba62-491d-a987-b784158c1c70" containerName="cleanup" Nov 02 06:45:23.481149 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:23.481075 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcebf78f-ba62-491d-a987-b784158c1c70" containerName="cleanup" Nov 02 06:45:23.481149 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:23.481082 5916 cpu_manager.go:414] "containerMap: removing container" podUID="3de4bd72-539c-4fd8-a0a1-c08383426e53" containerName="cleanup" Nov 02 06:45:23.481149 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:23.481087 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3de4bd72-539c-4fd8-a0a1-c08383426e53" containerName="cleanup" Nov 02 06:45:23.481149 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:23.481096 5916 cpu_manager.go:414] "containerMap: removing container" podUID="f2cac8cd-4ce7-4c70-b380-b9aec98431b8" containerName="cleanup" Nov 02 06:45:23.481149 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:23.481100 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2cac8cd-4ce7-4c70-b380-b9aec98431b8" containerName="cleanup" Nov 02 06:45:23.481149 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:23.481126 5916 cpu_manager.go:414] "containerMap: removing container" podUID="d7494bbe-ebe5-490e-9861-9d573c45f068" containerName="cleanup" Nov 02 06:45:23.481149 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:23.481132 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7494bbe-ebe5-490e-9861-9d573c45f068" containerName="cleanup" Nov 02 06:45:23.481149 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:23.481139 5916 cpu_manager.go:414] "containerMap: removing container" podUID="4d559915-7623-421a-9631-d308bf5a8c4e" containerName="cleanup" Nov 02 06:45:23.481149 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:23.481143 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d559915-7623-421a-9631-d308bf5a8c4e" containerName="cleanup" Nov 02 06:45:23.481149 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:23.481149 5916 cpu_manager.go:414] "containerMap: removing container" podUID="63a2970a-3a1d-4646-8ae6-9432dc18ab15" containerName="cleanup" Nov 02 06:45:23.481149 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:23.481153 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63a2970a-3a1d-4646-8ae6-9432dc18ab15" containerName="cleanup" Nov 02 06:45:23.481546 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:23.481158 5916 cpu_manager.go:414] "containerMap: removing container" podUID="b0419130-0781-47f1-af14-a8eb8917b1eb" containerName="cleanup" Nov 02 06:45:23.481546 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:23.481162 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0419130-0781-47f1-af14-a8eb8917b1eb" containerName="cleanup" Nov 02 06:45:23.481546 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:23.481168 5916 cpu_manager.go:414] "containerMap: removing container" podUID="63a2970a-3a1d-4646-8ae6-9432dc18ab15" containerName="cleanup" Nov 02 06:45:23.481546 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:23.481172 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63a2970a-3a1d-4646-8ae6-9432dc18ab15" containerName="cleanup" Nov 02 06:45:23.481546 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:23.481183 5916 cpu_manager.go:414] "containerMap: removing container" podUID="d7494bbe-ebe5-490e-9861-9d573c45f068" containerName="cleanup" Nov 02 06:45:23.481546 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:23.481187 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7494bbe-ebe5-490e-9861-9d573c45f068" containerName="cleanup" Nov 02 06:45:23.481546 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:23.481195 5916 cpu_manager.go:414] "containerMap: removing container" podUID="d7494bbe-ebe5-490e-9861-9d573c45f068" containerName="cleanup" Nov 02 06:45:23.481546 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:23.481199 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7494bbe-ebe5-490e-9861-9d573c45f068" containerName="cleanup" Nov 02 06:45:23.481546 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:23.481208 5916 cpu_manager.go:414] "containerMap: removing container" podUID="f2cac8cd-4ce7-4c70-b380-b9aec98431b8" containerName="cleanup" Nov 02 06:45:23.481546 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:23.481212 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2cac8cd-4ce7-4c70-b380-b9aec98431b8" containerName="cleanup" Nov 02 06:45:23.481546 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:23.481220 5916 cpu_manager.go:414] "containerMap: removing container" podUID="4d559915-7623-421a-9631-d308bf5a8c4e" containerName="cleanup" Nov 02 06:45:23.481546 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:23.481224 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d559915-7623-421a-9631-d308bf5a8c4e" containerName="cleanup" Nov 02 06:45:23.481546 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:23.481240 5916 cpu_manager.go:414] "containerMap: removing container" podUID="f2cac8cd-4ce7-4c70-b380-b9aec98431b8" containerName="cleanup" Nov 02 06:45:23.481546 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:23.481244 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2cac8cd-4ce7-4c70-b380-b9aec98431b8" containerName="cleanup" Nov 02 06:45:23.481546 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:23.481250 5916 cpu_manager.go:414] "containerMap: removing container" podUID="3de4bd72-539c-4fd8-a0a1-c08383426e53" containerName="cleanup" Nov 02 06:45:23.481546 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:23.481255 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3de4bd72-539c-4fd8-a0a1-c08383426e53" containerName="cleanup" Nov 02 06:45:23.481546 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:23.481371 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="3de4bd72-539c-4fd8-a0a1-c08383426e53" containerName="cleanup" Nov 02 06:45:23.481546 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:23.481381 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="fcebf78f-ba62-491d-a987-b784158c1c70" containerName="cleanup" Nov 02 06:45:23.481546 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:23.481388 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0419130-0781-47f1-af14-a8eb8917b1eb" containerName="cleanup" Nov 02 06:45:23.481546 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:23.481394 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="fcebf78f-ba62-491d-a987-b784158c1c70" containerName="cleanup" Nov 02 06:45:23.481546 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:23.481399 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="3de4bd72-539c-4fd8-a0a1-c08383426e53" containerName="cleanup" Nov 02 06:45:23.481546 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:23.481405 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2cac8cd-4ce7-4c70-b380-b9aec98431b8" containerName="cleanup" Nov 02 06:45:23.481546 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:23.481414 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7494bbe-ebe5-490e-9861-9d573c45f068" containerName="cleanup" Nov 02 06:45:23.481546 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:23.481419 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d559915-7623-421a-9631-d308bf5a8c4e" containerName="cleanup" Nov 02 06:45:23.481546 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:23.481425 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="63a2970a-3a1d-4646-8ae6-9432dc18ab15" containerName="cleanup" Nov 02 06:45:23.481546 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:23.481429 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0419130-0781-47f1-af14-a8eb8917b1eb" containerName="cleanup" Nov 02 06:45:23.481546 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:23.481434 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="63a2970a-3a1d-4646-8ae6-9432dc18ab15" containerName="cleanup" Nov 02 06:45:23.481546 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:23.481440 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7494bbe-ebe5-490e-9861-9d573c45f068" containerName="cleanup" Nov 02 06:45:23.481546 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:23.481447 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d559915-7623-421a-9631-d308bf5a8c4e" containerName="cleanup" Nov 02 06:45:23.481546 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:23.481452 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7494bbe-ebe5-490e-9861-9d573c45f068" containerName="cleanup" Nov 02 06:45:23.481546 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:23.481458 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="63a2970a-3a1d-4646-8ae6-9432dc18ab15" containerName="cleanup" Nov 02 06:45:23.481546 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:23.481464 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="fcebf78f-ba62-491d-a987-b784158c1c70" containerName="cleanup" Nov 02 06:45:23.481546 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:23.481470 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2cac8cd-4ce7-4c70-b380-b9aec98431b8" containerName="cleanup" Nov 02 06:45:23.481546 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:23.481477 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d559915-7623-421a-9631-d308bf5a8c4e" containerName="cleanup" Nov 02 06:45:23.481546 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:23.481483 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2cac8cd-4ce7-4c70-b380-b9aec98431b8" containerName="cleanup" Nov 02 06:45:23.486365 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:23.486347 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3bf02ebf-whd2b" Nov 02 06:45:23.539730 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:23.539699 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 06:45:23.557935 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:23.557514 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f498cdab-f85d-4586-bda6-ff3fa9e07bfb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-3bf02ebf-whd2b\" (UID: \"f498cdab-f85d-4586-bda6-ff3fa9e07bfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3bf02ebf-whd2b" Nov 02 06:45:23.557935 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:23.557632 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szzts\" (UniqueName: \"kubernetes.io/projected/f498cdab-f85d-4586-bda6-ff3fa9e07bfb-kube-api-access-szzts\") pod \"maas-api-revoke-keys-e2e-mt-3bf02ebf-whd2b\" (UID: \"f498cdab-f85d-4586-bda6-ff3fa9e07bfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3bf02ebf-whd2b" Nov 02 06:45:23.557935 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:23.557760 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f498cdab-f85d-4586-bda6-ff3fa9e07bfb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-3bf02ebf-whd2b\" (UID: \"f498cdab-f85d-4586-bda6-ff3fa9e07bfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3bf02ebf-whd2b" Nov 02 06:45:23.564653 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:23.564624 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3bf02ebf-whd2b"] Nov 02 06:45:23.586652 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:23.586627 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-68l4r\"" Nov 02 06:45:23.659135 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:23.659102 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f498cdab-f85d-4586-bda6-ff3fa9e07bfb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-3bf02ebf-whd2b\" (UID: \"f498cdab-f85d-4586-bda6-ff3fa9e07bfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3bf02ebf-whd2b" Nov 02 06:45:23.659576 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:23.659547 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f498cdab-f85d-4586-bda6-ff3fa9e07bfb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-3bf02ebf-whd2b\" (UID: \"f498cdab-f85d-4586-bda6-ff3fa9e07bfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3bf02ebf-whd2b" Nov 02 06:45:23.661238 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:23.661208 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f498cdab-f85d-4586-bda6-ff3fa9e07bfb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-3bf02ebf-whd2b\" (UID: \"f498cdab-f85d-4586-bda6-ff3fa9e07bfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3bf02ebf-whd2b" Nov 02 06:45:23.661683 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:23.661662 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szzts\" (UniqueName: \"kubernetes.io/projected/f498cdab-f85d-4586-bda6-ff3fa9e07bfb-kube-api-access-szzts\") pod \"maas-api-revoke-keys-e2e-mt-3bf02ebf-whd2b\" (UID: \"f498cdab-f85d-4586-bda6-ff3fa9e07bfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3bf02ebf-whd2b" Nov 02 06:45:23.665020 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:23.664995 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f498cdab-f85d-4586-bda6-ff3fa9e07bfb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-3bf02ebf-whd2b\" (UID: \"f498cdab-f85d-4586-bda6-ff3fa9e07bfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3bf02ebf-whd2b" Nov 02 06:45:23.718197 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:23.718129 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-szzts\" (UniqueName: \"kubernetes.io/projected/f498cdab-f85d-4586-bda6-ff3fa9e07bfb-kube-api-access-szzts\") pod \"maas-api-revoke-keys-e2e-mt-3bf02ebf-whd2b\" (UID: \"f498cdab-f85d-4586-bda6-ff3fa9e07bfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3bf02ebf-whd2b" Nov 02 06:45:23.800293 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:23.800263 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3bf02ebf-whd2b" Nov 02 06:45:23.893382 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:23.893350 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3de4bd72-539c-4fd8-a0a1-c08383426e53" path="/var/lib/kubelet/pods/3de4bd72-539c-4fd8-a0a1-c08383426e53/volumes" Nov 02 06:45:23.893836 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:23.893821 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63a2970a-3a1d-4646-8ae6-9432dc18ab15" path="/var/lib/kubelet/pods/63a2970a-3a1d-4646-8ae6-9432dc18ab15/volumes" Nov 02 06:45:24.259956 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:24.254830 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3bf02ebf-whd2b"] Nov 02 06:45:24.263159 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:45:24.263123 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf498cdab_f85d_4586_bda6_ff3fa9e07bfb.slice/crio-6efa0cabf5470a19227ff36cbd7702f2da18acb65ce3ccdc05c97fbaa1731af9 WatchSource:0}: Error finding container 6efa0cabf5470a19227ff36cbd7702f2da18acb65ce3ccdc05c97fbaa1731af9: Status 404 returned error can't find the container with id 6efa0cabf5470a19227ff36cbd7702f2da18acb65ce3ccdc05c97fbaa1731af9 Nov 02 06:45:24.439931 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:24.436290 5916 patch_prober.go:29] interesting pod/e2e-shared-a-w2-08bbf9-openshift-default-5479f6b7c-d2hk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Nov 02 06:45:24.439931 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:24.436351 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-08bbf9-openshift-default-5479f6b7c-d2hk2" podUID="1b46fdb7-008c-45d7-92cb-8e392c54547f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Nov 02 06:45:25.111504 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:25.111411 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3bf02ebf-whd2b" event={"ID":"f498cdab-f85d-4586-bda6-ff3fa9e07bfb","Type":"ContainerStarted","Data":"9b70581fea95847fe100445e724f45952af5ea3437c8d59c15560d04bcd39b63"} Nov 02 06:45:25.111504 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:25.111492 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3bf02ebf-whd2b" event={"ID":"f498cdab-f85d-4586-bda6-ff3fa9e07bfb","Type":"ContainerStarted","Data":"6efa0cabf5470a19227ff36cbd7702f2da18acb65ce3ccdc05c97fbaa1731af9"} Nov 02 06:45:25.226711 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:25.226657 5916 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3bf02ebf-whd2b" podStartSLOduration=2.226640216 podStartE2EDuration="2.226640216s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:45:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:45:25.225331417 +0000 UTC m=+1713.497151812" watchObservedRunningTime="2026-11-02 06:45:25.226640216 +0000 UTC m=+1713.498460612" Nov 02 06:45:25.441373 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:25.441345 5916 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-08bbf9-openshift-default-5479f6b7c-d2hk2" Nov 02 06:45:25.441820 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:25.441796 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-08bbf9-openshift-default-5479f6b7c-d2hk2" Nov 02 06:45:25.442457 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:25.442439 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-08bbf9-openshift-default-5479f6b7c-d2hk2" Nov 02 06:45:26.120649 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:26.120614 5916 generic.go:356] "Generic (PLEG): container finished" podID="f498cdab-f85d-4586-bda6-ff3fa9e07bfb" containerID="9b70581fea95847fe100445e724f45952af5ea3437c8d59c15560d04bcd39b63" exitCode=0 Nov 02 06:45:26.121114 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:26.120648 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3bf02ebf-whd2b" event={"ID":"f498cdab-f85d-4586-bda6-ff3fa9e07bfb","Type":"ContainerDied","Data":"9b70581fea95847fe100445e724f45952af5ea3437c8d59c15560d04bcd39b63"} Nov 02 06:45:27.296999 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:27.296971 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3bf02ebf-whd2b" Nov 02 06:45:27.328208 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:27.328183 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f498cdab-f85d-4586-bda6-ff3fa9e07bfb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f498cdab-f85d-4586-bda6-ff3fa9e07bfb-maas-api-openshift-ca\") pod \"f498cdab-f85d-4586-bda6-ff3fa9e07bfb\" (UID: \"f498cdab-f85d-4586-bda6-ff3fa9e07bfb\") " Nov 02 06:45:27.328340 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:27.328230 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f498cdab-f85d-4586-bda6-ff3fa9e07bfb-kube-api-access-szzts\" (UniqueName: \"kubernetes.io/projected/f498cdab-f85d-4586-bda6-ff3fa9e07bfb-kube-api-access-szzts\") pod \"f498cdab-f85d-4586-bda6-ff3fa9e07bfb\" (UID: \"f498cdab-f85d-4586-bda6-ff3fa9e07bfb\") " Nov 02 06:45:27.328420 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:27.328381 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f498cdab-f85d-4586-bda6-ff3fa9e07bfb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f498cdab-f85d-4586-bda6-ff3fa9e07bfb-maas-api-xks-ca\") pod \"f498cdab-f85d-4586-bda6-ff3fa9e07bfb\" (UID: \"f498cdab-f85d-4586-bda6-ff3fa9e07bfb\") " Nov 02 06:45:27.328570 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:27.328544 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f498cdab-f85d-4586-bda6-ff3fa9e07bfb-maas-api-openshift-ca" pod "f498cdab-f85d-4586-bda6-ff3fa9e07bfb" (UID: "f498cdab-f85d-4586-bda6-ff3fa9e07bfb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:45:27.328962 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:27.328940 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f498cdab-f85d-4586-bda6-ff3fa9e07bfb-maas-api-openshift-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:45:27.330437 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:27.330411 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f498cdab-f85d-4586-bda6-ff3fa9e07bfb-kube-api-access-szzts" pod "f498cdab-f85d-4586-bda6-ff3fa9e07bfb" (UID: "f498cdab-f85d-4586-bda6-ff3fa9e07bfb"). InnerVolumeSpecName "kube-api-access-szzts". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:45:27.330732 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:27.330712 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f498cdab-f85d-4586-bda6-ff3fa9e07bfb-maas-api-xks-ca" pod "f498cdab-f85d-4586-bda6-ff3fa9e07bfb" (UID: "f498cdab-f85d-4586-bda6-ff3fa9e07bfb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:45:27.435936 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:27.430762 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szzts\" (UniqueName: \"kubernetes.io/projected/f498cdab-f85d-4586-bda6-ff3fa9e07bfb-kube-api-access-szzts\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:45:27.435936 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:27.430798 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f498cdab-f85d-4586-bda6-ff3fa9e07bfb-maas-api-xks-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:45:27.554606 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:27.554562 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e5502564-mfnrs"] Nov 02 06:45:27.555485 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:27.555470 5916 cpu_manager.go:414] "containerMap: removing container" podUID="fcebf78f-ba62-491d-a987-b784158c1c70" containerName="cleanup" Nov 02 06:45:27.555539 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:27.555488 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcebf78f-ba62-491d-a987-b784158c1c70" containerName="cleanup" Nov 02 06:45:27.555582 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:27.555536 5916 cpu_manager.go:414] "containerMap: removing container" podUID="b0419130-0781-47f1-af14-a8eb8917b1eb" containerName="cleanup" Nov 02 06:45:27.555582 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:27.555545 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0419130-0781-47f1-af14-a8eb8917b1eb" containerName="cleanup" Nov 02 06:45:27.555582 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:27.555558 5916 cpu_manager.go:414] "containerMap: removing container" podUID="63a2970a-3a1d-4646-8ae6-9432dc18ab15" containerName="cleanup" Nov 02 06:45:27.555582 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:27.555565 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63a2970a-3a1d-4646-8ae6-9432dc18ab15" containerName="cleanup" Nov 02 06:45:27.555705 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:27.555593 5916 cpu_manager.go:414] "containerMap: removing container" podUID="f498cdab-f85d-4586-bda6-ff3fa9e07bfb" containerName="revoke-keys" Nov 02 06:45:27.555705 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:27.555600 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f498cdab-f85d-4586-bda6-ff3fa9e07bfb" containerName="revoke-keys" Nov 02 06:45:27.555705 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:27.555613 5916 cpu_manager.go:414] "containerMap: removing container" podUID="4d559915-7623-421a-9631-d308bf5a8c4e" containerName="cleanup" Nov 02 06:45:27.555705 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:27.555619 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d559915-7623-421a-9631-d308bf5a8c4e" containerName="cleanup" Nov 02 06:45:27.555705 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:27.555632 5916 cpu_manager.go:414] "containerMap: removing container" podUID="3de4bd72-539c-4fd8-a0a1-c08383426e53" containerName="cleanup" Nov 02 06:45:27.555705 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:27.555638 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3de4bd72-539c-4fd8-a0a1-c08383426e53" containerName="cleanup" Nov 02 06:45:27.555865 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:27.555775 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="3de4bd72-539c-4fd8-a0a1-c08383426e53" containerName="cleanup" Nov 02 06:45:27.555865 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:27.555793 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0419130-0781-47f1-af14-a8eb8917b1eb" containerName="cleanup" Nov 02 06:45:27.555865 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:27.555806 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="f498cdab-f85d-4586-bda6-ff3fa9e07bfb" containerName="revoke-keys" Nov 02 06:45:27.560000 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:27.559980 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e5502564-mfnrs" Nov 02 06:45:27.613244 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:27.613217 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e5502564-mfnrs"] Nov 02 06:45:27.633973 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:27.633930 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e140326f-6f77-46ff-b6a0-c853c0c118e6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-e5502564-mfnrs\" (UID: \"e140326f-6f77-46ff-b6a0-c853c0c118e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e5502564-mfnrs" Nov 02 06:45:27.634099 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:27.633978 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9pc9\" (UniqueName: \"kubernetes.io/projected/e140326f-6f77-46ff-b6a0-c853c0c118e6-kube-api-access-n9pc9\") pod \"maas-api-revoke-keys-e2e-mt-e5502564-mfnrs\" (UID: \"e140326f-6f77-46ff-b6a0-c853c0c118e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e5502564-mfnrs" Nov 02 06:45:27.634099 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:27.634068 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e140326f-6f77-46ff-b6a0-c853c0c118e6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-e5502564-mfnrs\" (UID: \"e140326f-6f77-46ff-b6a0-c853c0c118e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e5502564-mfnrs" Nov 02 06:45:27.735755 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:27.735679 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e140326f-6f77-46ff-b6a0-c853c0c118e6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-e5502564-mfnrs\" (UID: \"e140326f-6f77-46ff-b6a0-c853c0c118e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e5502564-mfnrs" Nov 02 06:45:27.735755 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:27.735716 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9pc9\" (UniqueName: \"kubernetes.io/projected/e140326f-6f77-46ff-b6a0-c853c0c118e6-kube-api-access-n9pc9\") pod \"maas-api-revoke-keys-e2e-mt-e5502564-mfnrs\" (UID: \"e140326f-6f77-46ff-b6a0-c853c0c118e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e5502564-mfnrs" Nov 02 06:45:27.735755 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:27.735740 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e140326f-6f77-46ff-b6a0-c853c0c118e6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-e5502564-mfnrs\" (UID: \"e140326f-6f77-46ff-b6a0-c853c0c118e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e5502564-mfnrs" Nov 02 06:45:27.736327 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:27.736308 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e140326f-6f77-46ff-b6a0-c853c0c118e6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-e5502564-mfnrs\" (UID: \"e140326f-6f77-46ff-b6a0-c853c0c118e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e5502564-mfnrs" Nov 02 06:45:27.738121 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:27.738093 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e140326f-6f77-46ff-b6a0-c853c0c118e6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-e5502564-mfnrs\" (UID: \"e140326f-6f77-46ff-b6a0-c853c0c118e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e5502564-mfnrs" Nov 02 06:45:27.792726 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:27.792687 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9pc9\" (UniqueName: \"kubernetes.io/projected/e140326f-6f77-46ff-b6a0-c853c0c118e6-kube-api-access-n9pc9\") pod \"maas-api-revoke-keys-e2e-mt-e5502564-mfnrs\" (UID: \"e140326f-6f77-46ff-b6a0-c853c0c118e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e5502564-mfnrs" Nov 02 06:45:27.877710 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:27.877684 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e5502564-mfnrs" Nov 02 06:45:28.137623 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:28.137594 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3bf02ebf-whd2b" Nov 02 06:45:28.137807 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:28.137599 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3bf02ebf-whd2b" event={"ID":"f498cdab-f85d-4586-bda6-ff3fa9e07bfb","Type":"ContainerDied","Data":"6efa0cabf5470a19227ff36cbd7702f2da18acb65ce3ccdc05c97fbaa1731af9"} Nov 02 06:45:28.137807 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:28.137707 5916 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6efa0cabf5470a19227ff36cbd7702f2da18acb65ce3ccdc05c97fbaa1731af9" Nov 02 06:45:28.267638 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:45:28.267596 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode140326f_6f77_46ff_b6a0_c853c0c118e6.slice/crio-e427f8747b560ef3e4854229534f327b6b75c752c778750677a38782b6dc9a2d WatchSource:0}: Error finding container e427f8747b560ef3e4854229534f327b6b75c752c778750677a38782b6dc9a2d: Status 404 returned error can't find the container with id e427f8747b560ef3e4854229534f327b6b75c752c778750677a38782b6dc9a2d Nov 02 06:45:28.272784 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:28.272764 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e5502564-mfnrs"] Nov 02 06:45:29.146714 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:29.146685 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e5502564-mfnrs" event={"ID":"e140326f-6f77-46ff-b6a0-c853c0c118e6","Type":"ContainerStarted","Data":"4a945f5db597914b7a17f6945786b1b49e86b1471c6d08556cc10087eb584e30"} Nov 02 06:45:29.147101 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:29.146736 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e5502564-mfnrs" event={"ID":"e140326f-6f77-46ff-b6a0-c853c0c118e6","Type":"ContainerStarted","Data":"e427f8747b560ef3e4854229534f327b6b75c752c778750677a38782b6dc9a2d"} Nov 02 06:45:29.260822 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:29.260779 5916 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e5502564-mfnrs" podStartSLOduration=2.260765293 podStartE2EDuration="2.260765293s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:45:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:45:29.260125451 +0000 UTC m=+1717.531945846" watchObservedRunningTime="2026-11-02 06:45:29.260765293 +0000 UTC m=+1717.532585689" Nov 02 06:45:30.155967 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:30.155933 5916 generic.go:356] "Generic (PLEG): container finished" podID="e140326f-6f77-46ff-b6a0-c853c0c118e6" containerID="4a945f5db597914b7a17f6945786b1b49e86b1471c6d08556cc10087eb584e30" exitCode=0 Nov 02 06:45:30.156342 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:30.156001 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e5502564-mfnrs" event={"ID":"e140326f-6f77-46ff-b6a0-c853c0c118e6","Type":"ContainerDied","Data":"4a945f5db597914b7a17f6945786b1b49e86b1471c6d08556cc10087eb584e30"} Nov 02 06:45:31.341729 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:31.341705 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e5502564-mfnrs" Nov 02 06:45:31.382629 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:31.382589 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e140326f-6f77-46ff-b6a0-c853c0c118e6-kube-api-access-n9pc9\" (UniqueName: \"kubernetes.io/projected/e140326f-6f77-46ff-b6a0-c853c0c118e6-kube-api-access-n9pc9\") pod \"e140326f-6f77-46ff-b6a0-c853c0c118e6\" (UID: \"e140326f-6f77-46ff-b6a0-c853c0c118e6\") " Nov 02 06:45:31.382759 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:31.382702 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e140326f-6f77-46ff-b6a0-c853c0c118e6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e140326f-6f77-46ff-b6a0-c853c0c118e6-maas-api-xks-ca\") pod \"e140326f-6f77-46ff-b6a0-c853c0c118e6\" (UID: \"e140326f-6f77-46ff-b6a0-c853c0c118e6\") " Nov 02 06:45:31.383039 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:31.383020 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e140326f-6f77-46ff-b6a0-c853c0c118e6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e140326f-6f77-46ff-b6a0-c853c0c118e6-maas-api-openshift-ca\") pod \"e140326f-6f77-46ff-b6a0-c853c0c118e6\" (UID: \"e140326f-6f77-46ff-b6a0-c853c0c118e6\") " Nov 02 06:45:31.383418 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:31.383389 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e140326f-6f77-46ff-b6a0-c853c0c118e6-maas-api-openshift-ca" pod "e140326f-6f77-46ff-b6a0-c853c0c118e6" (UID: "e140326f-6f77-46ff-b6a0-c853c0c118e6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:45:31.383681 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:31.383657 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e140326f-6f77-46ff-b6a0-c853c0c118e6-maas-api-openshift-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:45:31.384840 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:31.384814 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e140326f-6f77-46ff-b6a0-c853c0c118e6-maas-api-xks-ca" pod "e140326f-6f77-46ff-b6a0-c853c0c118e6" (UID: "e140326f-6f77-46ff-b6a0-c853c0c118e6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:45:31.385184 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:31.385162 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e140326f-6f77-46ff-b6a0-c853c0c118e6-kube-api-access-n9pc9" pod "e140326f-6f77-46ff-b6a0-c853c0c118e6" (UID: "e140326f-6f77-46ff-b6a0-c853c0c118e6"). InnerVolumeSpecName "kube-api-access-n9pc9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:45:31.485658 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:31.485587 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9pc9\" (UniqueName: \"kubernetes.io/projected/e140326f-6f77-46ff-b6a0-c853c0c118e6-kube-api-access-n9pc9\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:45:31.485658 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:31.485617 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e140326f-6f77-46ff-b6a0-c853c0c118e6-maas-api-xks-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:45:32.174481 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:32.174453 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e5502564-mfnrs" Nov 02 06:45:32.174751 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:32.174497 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e5502564-mfnrs" event={"ID":"e140326f-6f77-46ff-b6a0-c853c0c118e6","Type":"ContainerDied","Data":"e427f8747b560ef3e4854229534f327b6b75c752c778750677a38782b6dc9a2d"} Nov 02 06:45:32.174875 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:32.174774 5916 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e427f8747b560ef3e4854229534f327b6b75c752c778750677a38782b6dc9a2d" Nov 02 06:45:49.900224 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:45:49.900160 5916 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p98cq" containerName="extractor" Nov 02 06:45:54.555515 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:54.555478 5916 scope.go:122] "RemoveContainer" containerID="d76dedb075972834d8183084e1734fb0d70e7ae3421839837177d51653bb2112" Nov 02 06:45:54.567399 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:54.567376 5916 scope.go:122] "RemoveContainer" containerID="cffe55ce0d9e19e10f54d01ecced5cc89a635087d64a8a59fa5a98ba11445bc8" Nov 02 06:45:54.576524 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:45:54.576503 5916 scope.go:122] "RemoveContainer" containerID="67164a0c533d5a518b740df7d0112244aa85f7c16aa52e3d01808def74144292" Nov 02 06:46:05.986278 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:05.986232 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-rvnb6"] Nov 02 06:46:05.986775 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:05.986486 5916 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-rvnb6" podUID="0d4ba27f-3588-4c0b-9446-700fcff8bb14" containerName="istio-proxy" containerID="cri-o://570465c34d626fe369b828e443e8a02d668daa6d71144a85af9c6112d43a4493" gracePeriod=30 Nov 02 06:46:06.481499 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:06.481467 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-x2qpj"] Nov 02 06:46:06.481681 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:06.481659 5916 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-x2qpj" podUID="33373aea-62e6-4fa4-870d-707c3e7bc858" containerName="istio-proxy" containerID="cri-o://970c1dad82dd817e41dd56fb156bf90e0007567dcd40ff9c2b89cac5ad991715" gracePeriod=30 Nov 02 06:46:11.277075 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:11.277047 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-rvnb6" Nov 02 06:46:11.294186 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:11.294153 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d4ba27f-3588-4c0b-9446-700fcff8bb14-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d4ba27f-3588-4c0b-9446-700fcff8bb14-workload-certs\") pod \"0d4ba27f-3588-4c0b-9446-700fcff8bb14\" (UID: \"0d4ba27f-3588-4c0b-9446-700fcff8bb14\") " Nov 02 06:46:11.294311 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:11.294196 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d4ba27f-3588-4c0b-9446-700fcff8bb14-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d4ba27f-3588-4c0b-9446-700fcff8bb14-istio-data\") pod \"0d4ba27f-3588-4c0b-9446-700fcff8bb14\" (UID: \"0d4ba27f-3588-4c0b-9446-700fcff8bb14\") " Nov 02 06:46:11.294311 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:11.294238 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0d4ba27f-3588-4c0b-9446-700fcff8bb14-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d4ba27f-3588-4c0b-9446-700fcff8bb14-istio-podinfo\") pod \"0d4ba27f-3588-4c0b-9446-700fcff8bb14\" (UID: \"0d4ba27f-3588-4c0b-9446-700fcff8bb14\") " Nov 02 06:46:11.294311 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:11.294275 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d4ba27f-3588-4c0b-9446-700fcff8bb14-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d4ba27f-3588-4c0b-9446-700fcff8bb14-workload-socket\") pod \"0d4ba27f-3588-4c0b-9446-700fcff8bb14\" (UID: \"0d4ba27f-3588-4c0b-9446-700fcff8bb14\") " Nov 02 06:46:11.294493 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:11.294327 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d4ba27f-3588-4c0b-9446-700fcff8bb14-istio-token\" (UniqueName: \"kubernetes.io/projected/0d4ba27f-3588-4c0b-9446-700fcff8bb14-istio-token\") pod \"0d4ba27f-3588-4c0b-9446-700fcff8bb14\" (UID: \"0d4ba27f-3588-4c0b-9446-700fcff8bb14\") " Nov 02 06:46:11.294493 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:11.294356 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d4ba27f-3588-4c0b-9446-700fcff8bb14-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d4ba27f-3588-4c0b-9446-700fcff8bb14-credential-socket\") pod \"0d4ba27f-3588-4c0b-9446-700fcff8bb14\" (UID: \"0d4ba27f-3588-4c0b-9446-700fcff8bb14\") " Nov 02 06:46:11.294493 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:11.294409 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d4ba27f-3588-4c0b-9446-700fcff8bb14-workload-certs" pod "0d4ba27f-3588-4c0b-9446-700fcff8bb14" (UID: "0d4ba27f-3588-4c0b-9446-700fcff8bb14"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:46:11.294493 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:11.294448 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d4ba27f-3588-4c0b-9446-700fcff8bb14-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d4ba27f-3588-4c0b-9446-700fcff8bb14-istio-envoy\") pod \"0d4ba27f-3588-4c0b-9446-700fcff8bb14\" (UID: \"0d4ba27f-3588-4c0b-9446-700fcff8bb14\") " Nov 02 06:46:11.294716 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:11.294517 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d4ba27f-3588-4c0b-9446-700fcff8bb14-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d4ba27f-3588-4c0b-9446-700fcff8bb14-istiod-ca-cert\") pod \"0d4ba27f-3588-4c0b-9446-700fcff8bb14\" (UID: \"0d4ba27f-3588-4c0b-9446-700fcff8bb14\") " Nov 02 06:46:11.294716 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:11.294513 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d4ba27f-3588-4c0b-9446-700fcff8bb14-istio-data" pod "0d4ba27f-3588-4c0b-9446-700fcff8bb14" (UID: "0d4ba27f-3588-4c0b-9446-700fcff8bb14"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:46:11.294716 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:11.294605 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d4ba27f-3588-4c0b-9446-700fcff8bb14-kube-api-access-zm9gf\" (UniqueName: \"kubernetes.io/projected/0d4ba27f-3588-4c0b-9446-700fcff8bb14-kube-api-access-zm9gf\") pod \"0d4ba27f-3588-4c0b-9446-700fcff8bb14\" (UID: \"0d4ba27f-3588-4c0b-9446-700fcff8bb14\") " Nov 02 06:46:11.294716 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:11.294587 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d4ba27f-3588-4c0b-9446-700fcff8bb14-workload-socket" pod "0d4ba27f-3588-4c0b-9446-700fcff8bb14" (UID: "0d4ba27f-3588-4c0b-9446-700fcff8bb14"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:46:11.294985 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:11.294697 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d4ba27f-3588-4c0b-9446-700fcff8bb14-credential-socket" pod "0d4ba27f-3588-4c0b-9446-700fcff8bb14" (UID: "0d4ba27f-3588-4c0b-9446-700fcff8bb14"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:46:11.294985 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:11.294850 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d4ba27f-3588-4c0b-9446-700fcff8bb14-istiod-ca-cert" pod "0d4ba27f-3588-4c0b-9446-700fcff8bb14" (UID: "0d4ba27f-3588-4c0b-9446-700fcff8bb14"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:46:11.295273 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:11.295254 5916 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d4ba27f-3588-4c0b-9446-700fcff8bb14-istiod-ca-cert\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:46:11.295273 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:11.295273 5916 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d4ba27f-3588-4c0b-9446-700fcff8bb14-workload-certs\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:46:11.295425 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:11.295283 5916 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d4ba27f-3588-4c0b-9446-700fcff8bb14-istio-data\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:46:11.295425 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:11.295293 5916 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d4ba27f-3588-4c0b-9446-700fcff8bb14-workload-socket\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:46:11.295425 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:11.295301 5916 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d4ba27f-3588-4c0b-9446-700fcff8bb14-credential-socket\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:46:11.296648 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:11.296625 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d4ba27f-3588-4c0b-9446-700fcff8bb14-kube-api-access-zm9gf" pod "0d4ba27f-3588-4c0b-9446-700fcff8bb14" (UID: "0d4ba27f-3588-4c0b-9446-700fcff8bb14"). InnerVolumeSpecName "kube-api-access-zm9gf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:46:11.296762 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:11.296678 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d4ba27f-3588-4c0b-9446-700fcff8bb14-istio-envoy" pod "0d4ba27f-3588-4c0b-9446-700fcff8bb14" (UID: "0d4ba27f-3588-4c0b-9446-700fcff8bb14"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:46:11.296901 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:11.296878 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d4ba27f-3588-4c0b-9446-700fcff8bb14-istio-token" pod "0d4ba27f-3588-4c0b-9446-700fcff8bb14" (UID: "0d4ba27f-3588-4c0b-9446-700fcff8bb14"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:46:11.297032 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:11.296888 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0d4ba27f-3588-4c0b-9446-700fcff8bb14-istio-podinfo" pod "0d4ba27f-3588-4c0b-9446-700fcff8bb14" (UID: "0d4ba27f-3588-4c0b-9446-700fcff8bb14"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 06:46:11.397051 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:11.397014 5916 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d4ba27f-3588-4c0b-9446-700fcff8bb14-istio-token\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:46:11.397051 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:11.397049 5916 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d4ba27f-3588-4c0b-9446-700fcff8bb14-istio-envoy\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:46:11.398052 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:11.397064 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zm9gf\" (UniqueName: \"kubernetes.io/projected/0d4ba27f-3588-4c0b-9446-700fcff8bb14-kube-api-access-zm9gf\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:46:11.398052 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:11.397078 5916 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d4ba27f-3588-4c0b-9446-700fcff8bb14-istio-podinfo\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:46:11.762390 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:11.762350 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-x2qpj" Nov 02 06:46:11.764378 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:11.764341 5916 generic.go:356] "Generic (PLEG): container finished" podID="0d4ba27f-3588-4c0b-9446-700fcff8bb14" containerID="570465c34d626fe369b828e443e8a02d668daa6d71144a85af9c6112d43a4493" exitCode=0 Nov 02 06:46:11.764505 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:11.764406 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-rvnb6" Nov 02 06:46:11.764505 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:11.764419 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-rvnb6" event={"ID":"0d4ba27f-3588-4c0b-9446-700fcff8bb14","Type":"ContainerDied","Data":"570465c34d626fe369b828e443e8a02d668daa6d71144a85af9c6112d43a4493"} Nov 02 06:46:11.764505 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:11.764493 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-rvnb6" event={"ID":"0d4ba27f-3588-4c0b-9446-700fcff8bb14","Type":"ContainerDied","Data":"a530747df764043f66602972d8730601929abb60f7205b4d0880216fd5cac3c8"} Nov 02 06:46:11.764668 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:11.764515 5916 scope.go:122] "RemoveContainer" containerID="570465c34d626fe369b828e443e8a02d668daa6d71144a85af9c6112d43a4493" Nov 02 06:46:11.766265 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:11.766208 5916 generic.go:356] "Generic (PLEG): container finished" podID="33373aea-62e6-4fa4-870d-707c3e7bc858" containerID="970c1dad82dd817e41dd56fb156bf90e0007567dcd40ff9c2b89cac5ad991715" exitCode=0 Nov 02 06:46:11.766384 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:11.766261 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-x2qpj" Nov 02 06:46:11.766384 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:11.766305 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-x2qpj" event={"ID":"33373aea-62e6-4fa4-870d-707c3e7bc858","Type":"ContainerDied","Data":"970c1dad82dd817e41dd56fb156bf90e0007567dcd40ff9c2b89cac5ad991715"} Nov 02 06:46:11.766384 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:11.766340 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-x2qpj" event={"ID":"33373aea-62e6-4fa4-870d-707c3e7bc858","Type":"ContainerDied","Data":"8b0b5b283c8dfec362e0b0c3be2f6486cd97f92bf8b6b2acd4849d891613b462"} Nov 02 06:46:11.777606 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:11.777582 5916 scope.go:122] "RemoveContainer" containerID="570465c34d626fe369b828e443e8a02d668daa6d71144a85af9c6112d43a4493" Nov 02 06:46:11.782425 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:46:11.782396 5916 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"570465c34d626fe369b828e443e8a02d668daa6d71144a85af9c6112d43a4493\": container with ID starting with 570465c34d626fe369b828e443e8a02d668daa6d71144a85af9c6112d43a4493 not found: ID does not exist" containerID="570465c34d626fe369b828e443e8a02d668daa6d71144a85af9c6112d43a4493" Nov 02 06:46:11.782509 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:11.782446 5916 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"570465c34d626fe369b828e443e8a02d668daa6d71144a85af9c6112d43a4493"} err="failed to get container status \"570465c34d626fe369b828e443e8a02d668daa6d71144a85af9c6112d43a4493\": rpc error: code = NotFound desc = could not find container \"570465c34d626fe369b828e443e8a02d668daa6d71144a85af9c6112d43a4493\": container with ID starting with 570465c34d626fe369b828e443e8a02d668daa6d71144a85af9c6112d43a4493 not found: ID does not exist" Nov 02 06:46:11.782509 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:11.782471 5916 scope.go:122] "RemoveContainer" containerID="970c1dad82dd817e41dd56fb156bf90e0007567dcd40ff9c2b89cac5ad991715" Nov 02 06:46:11.795895 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:11.795877 5916 scope.go:122] "RemoveContainer" containerID="970c1dad82dd817e41dd56fb156bf90e0007567dcd40ff9c2b89cac5ad991715" Nov 02 06:46:11.796152 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:46:11.796133 5916 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"970c1dad82dd817e41dd56fb156bf90e0007567dcd40ff9c2b89cac5ad991715\": container with ID starting with 970c1dad82dd817e41dd56fb156bf90e0007567dcd40ff9c2b89cac5ad991715 not found: ID does not exist" containerID="970c1dad82dd817e41dd56fb156bf90e0007567dcd40ff9c2b89cac5ad991715" Nov 02 06:46:11.796218 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:11.796156 5916 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"970c1dad82dd817e41dd56fb156bf90e0007567dcd40ff9c2b89cac5ad991715"} err="failed to get container status \"970c1dad82dd817e41dd56fb156bf90e0007567dcd40ff9c2b89cac5ad991715\": rpc error: code = NotFound desc = could not find container \"970c1dad82dd817e41dd56fb156bf90e0007567dcd40ff9c2b89cac5ad991715\": container with ID starting with 970c1dad82dd817e41dd56fb156bf90e0007567dcd40ff9c2b89cac5ad991715 not found: ID does not exist" Nov 02 06:46:11.803009 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:11.802986 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33373aea-62e6-4fa4-870d-707c3e7bc858-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33373aea-62e6-4fa4-870d-707c3e7bc858-workload-socket\") pod \"33373aea-62e6-4fa4-870d-707c3e7bc858\" (UID: \"33373aea-62e6-4fa4-870d-707c3e7bc858\") " Nov 02 06:46:11.803124 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:11.803024 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33373aea-62e6-4fa4-870d-707c3e7bc858-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33373aea-62e6-4fa4-870d-707c3e7bc858-istio-envoy\") pod \"33373aea-62e6-4fa4-870d-707c3e7bc858\" (UID: \"33373aea-62e6-4fa4-870d-707c3e7bc858\") " Nov 02 06:46:11.803124 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:11.803072 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33373aea-62e6-4fa4-870d-707c3e7bc858-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33373aea-62e6-4fa4-870d-707c3e7bc858-istiod-ca-cert\") pod \"33373aea-62e6-4fa4-870d-707c3e7bc858\" (UID: \"33373aea-62e6-4fa4-870d-707c3e7bc858\") " Nov 02 06:46:11.803241 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:11.803156 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33373aea-62e6-4fa4-870d-707c3e7bc858-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33373aea-62e6-4fa4-870d-707c3e7bc858-credential-socket\") pod \"33373aea-62e6-4fa4-870d-707c3e7bc858\" (UID: \"33373aea-62e6-4fa4-870d-707c3e7bc858\") " Nov 02 06:46:11.803303 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:11.803242 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33373aea-62e6-4fa4-870d-707c3e7bc858-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33373aea-62e6-4fa4-870d-707c3e7bc858-workload-certs\") pod \"33373aea-62e6-4fa4-870d-707c3e7bc858\" (UID: \"33373aea-62e6-4fa4-870d-707c3e7bc858\") " Nov 02 06:46:11.803359 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:11.803308 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33373aea-62e6-4fa4-870d-707c3e7bc858-workload-socket" pod "33373aea-62e6-4fa4-870d-707c3e7bc858" (UID: "33373aea-62e6-4fa4-870d-707c3e7bc858"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:46:11.803359 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:11.803320 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/33373aea-62e6-4fa4-870d-707c3e7bc858-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33373aea-62e6-4fa4-870d-707c3e7bc858-istio-podinfo\") pod \"33373aea-62e6-4fa4-870d-707c3e7bc858\" (UID: \"33373aea-62e6-4fa4-870d-707c3e7bc858\") " Nov 02 06:46:11.803474 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:11.803376 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33373aea-62e6-4fa4-870d-707c3e7bc858-kube-api-access-bn7pw\" (UniqueName: \"kubernetes.io/projected/33373aea-62e6-4fa4-870d-707c3e7bc858-kube-api-access-bn7pw\") pod \"33373aea-62e6-4fa4-870d-707c3e7bc858\" (UID: \"33373aea-62e6-4fa4-870d-707c3e7bc858\") " Nov 02 06:46:11.803474 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:11.803423 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33373aea-62e6-4fa4-870d-707c3e7bc858-istio-token\" (UniqueName: \"kubernetes.io/projected/33373aea-62e6-4fa4-870d-707c3e7bc858-istio-token\") pod \"33373aea-62e6-4fa4-870d-707c3e7bc858\" (UID: \"33373aea-62e6-4fa4-870d-707c3e7bc858\") " Nov 02 06:46:11.803581 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:11.803479 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33373aea-62e6-4fa4-870d-707c3e7bc858-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33373aea-62e6-4fa4-870d-707c3e7bc858-istio-data\") pod \"33373aea-62e6-4fa4-870d-707c3e7bc858\" (UID: \"33373aea-62e6-4fa4-870d-707c3e7bc858\") " Nov 02 06:46:11.803581 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:11.803556 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33373aea-62e6-4fa4-870d-707c3e7bc858-credential-socket" pod "33373aea-62e6-4fa4-870d-707c3e7bc858" (UID: "33373aea-62e6-4fa4-870d-707c3e7bc858"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:46:11.803581 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:11.803568 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33373aea-62e6-4fa4-870d-707c3e7bc858-workload-certs" pod "33373aea-62e6-4fa4-870d-707c3e7bc858" (UID: "33373aea-62e6-4fa4-870d-707c3e7bc858"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:46:11.805863 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:11.803950 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33373aea-62e6-4fa4-870d-707c3e7bc858-istiod-ca-cert" pod "33373aea-62e6-4fa4-870d-707c3e7bc858" (UID: "33373aea-62e6-4fa4-870d-707c3e7bc858"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:46:11.805863 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:11.804168 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33373aea-62e6-4fa4-870d-707c3e7bc858-istio-data" pod "33373aea-62e6-4fa4-870d-707c3e7bc858" (UID: "33373aea-62e6-4fa4-870d-707c3e7bc858"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:46:11.805863 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:11.804291 5916 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33373aea-62e6-4fa4-870d-707c3e7bc858-workload-certs\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:46:11.805863 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:11.804310 5916 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33373aea-62e6-4fa4-870d-707c3e7bc858-istio-data\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:46:11.805863 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:11.804324 5916 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33373aea-62e6-4fa4-870d-707c3e7bc858-workload-socket\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:46:11.805863 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:11.804338 5916 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33373aea-62e6-4fa4-870d-707c3e7bc858-istiod-ca-cert\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:46:11.805863 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:11.804352 5916 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33373aea-62e6-4fa4-870d-707c3e7bc858-credential-socket\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:46:11.805863 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:11.805674 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/33373aea-62e6-4fa4-870d-707c3e7bc858-istio-podinfo" pod "33373aea-62e6-4fa4-870d-707c3e7bc858" (UID: "33373aea-62e6-4fa4-870d-707c3e7bc858"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 06:46:11.805863 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:11.805878 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33373aea-62e6-4fa4-870d-707c3e7bc858-istio-token" pod "33373aea-62e6-4fa4-870d-707c3e7bc858" (UID: "33373aea-62e6-4fa4-870d-707c3e7bc858"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:46:11.806490 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:11.806154 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33373aea-62e6-4fa4-870d-707c3e7bc858-kube-api-access-bn7pw" pod "33373aea-62e6-4fa4-870d-707c3e7bc858" (UID: "33373aea-62e6-4fa4-870d-707c3e7bc858"). InnerVolumeSpecName "kube-api-access-bn7pw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:46:11.807091 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:11.807072 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33373aea-62e6-4fa4-870d-707c3e7bc858-istio-envoy" pod "33373aea-62e6-4fa4-870d-707c3e7bc858" (UID: "33373aea-62e6-4fa4-870d-707c3e7bc858"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:46:12.020982 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:12.010777 5916 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33373aea-62e6-4fa4-870d-707c3e7bc858-istio-envoy\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:46:12.020982 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:12.010820 5916 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33373aea-62e6-4fa4-870d-707c3e7bc858-istio-podinfo\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:46:12.020982 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:12.010834 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bn7pw\" (UniqueName: \"kubernetes.io/projected/33373aea-62e6-4fa4-870d-707c3e7bc858-kube-api-access-bn7pw\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:46:12.020982 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:12.010848 5916 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33373aea-62e6-4fa4-870d-707c3e7bc858-istio-token\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:46:12.192733 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:12.192702 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-rvnb6"] Nov 02 06:46:12.196986 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:12.196964 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-rvnb6"] Nov 02 06:46:12.306357 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:12.306249 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-x2qpj"] Nov 02 06:46:12.310181 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:12.310154 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-x2qpj"] Nov 02 06:46:13.848252 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:13.848213 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d4ba27f-3588-4c0b-9446-700fcff8bb14" path="/var/lib/kubelet/pods/0d4ba27f-3588-4c0b-9446-700fcff8bb14/volumes" Nov 02 06:46:13.848986 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:13.848963 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33373aea-62e6-4fa4-870d-707c3e7bc858" path="/var/lib/kubelet/pods/33373aea-62e6-4fa4-870d-707c3e7bc858/volumes" Nov 02 06:46:17.176739 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:17.176703 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5fe8b848-ght6b"] Nov 02 06:46:17.178383 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:17.178350 5916 cpu_manager.go:414] "containerMap: removing container" podUID="e140326f-6f77-46ff-b6a0-c853c0c118e6" containerName="revoke-keys" Nov 02 06:46:17.178383 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:17.178382 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e140326f-6f77-46ff-b6a0-c853c0c118e6" containerName="revoke-keys" Nov 02 06:46:17.178539 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:17.178395 5916 cpu_manager.go:414] "containerMap: removing container" podUID="33373aea-62e6-4fa4-870d-707c3e7bc858" containerName="istio-proxy" Nov 02 06:46:17.178539 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:17.178401 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33373aea-62e6-4fa4-870d-707c3e7bc858" containerName="istio-proxy" Nov 02 06:46:17.178539 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:17.178435 5916 cpu_manager.go:414] "containerMap: removing container" podUID="0d4ba27f-3588-4c0b-9446-700fcff8bb14" containerName="istio-proxy" Nov 02 06:46:17.178539 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:17.178440 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d4ba27f-3588-4c0b-9446-700fcff8bb14" containerName="istio-proxy" Nov 02 06:46:17.178673 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:17.178558 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d4ba27f-3588-4c0b-9446-700fcff8bb14" containerName="istio-proxy" Nov 02 06:46:17.178673 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:17.178570 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="e140326f-6f77-46ff-b6a0-c853c0c118e6" containerName="revoke-keys" Nov 02 06:46:17.178673 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:17.178576 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="33373aea-62e6-4fa4-870d-707c3e7bc858" containerName="istio-proxy" Nov 02 06:46:17.183634 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:17.183614 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fe8b848-ght6b" Nov 02 06:46:17.240614 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:17.240536 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5fe8b848-ght6b"] Nov 02 06:46:17.292928 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:17.285207 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 06:46:17.292928 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:17.285369 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-68l4r\"" Nov 02 06:46:17.308968 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:17.307933 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26d48294-0eb7-48c7-868c-73068ae0b883-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5fe8b848-ght6b\" (UID: \"26d48294-0eb7-48c7-868c-73068ae0b883\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fe8b848-ght6b" Nov 02 06:46:17.308968 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:17.308001 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znms5\" (UniqueName: \"kubernetes.io/projected/26d48294-0eb7-48c7-868c-73068ae0b883-kube-api-access-znms5\") pod \"maas-api-revoke-sub-5fe8b848-ght6b\" (UID: \"26d48294-0eb7-48c7-868c-73068ae0b883\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fe8b848-ght6b" Nov 02 06:46:17.308968 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:17.308133 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26d48294-0eb7-48c7-868c-73068ae0b883-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5fe8b848-ght6b\" (UID: \"26d48294-0eb7-48c7-868c-73068ae0b883\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fe8b848-ght6b" Nov 02 06:46:17.416931 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:17.414502 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26d48294-0eb7-48c7-868c-73068ae0b883-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5fe8b848-ght6b\" (UID: \"26d48294-0eb7-48c7-868c-73068ae0b883\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fe8b848-ght6b" Nov 02 06:46:17.416931 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:17.414571 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znms5\" (UniqueName: \"kubernetes.io/projected/26d48294-0eb7-48c7-868c-73068ae0b883-kube-api-access-znms5\") pod \"maas-api-revoke-sub-5fe8b848-ght6b\" (UID: \"26d48294-0eb7-48c7-868c-73068ae0b883\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fe8b848-ght6b" Nov 02 06:46:17.416931 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:17.414701 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26d48294-0eb7-48c7-868c-73068ae0b883-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5fe8b848-ght6b\" (UID: \"26d48294-0eb7-48c7-868c-73068ae0b883\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fe8b848-ght6b" Nov 02 06:46:17.416931 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:17.415619 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26d48294-0eb7-48c7-868c-73068ae0b883-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5fe8b848-ght6b\" (UID: \"26d48294-0eb7-48c7-868c-73068ae0b883\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fe8b848-ght6b" Nov 02 06:46:17.498656 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:17.496453 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26d48294-0eb7-48c7-868c-73068ae0b883-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5fe8b848-ght6b\" (UID: \"26d48294-0eb7-48c7-868c-73068ae0b883\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fe8b848-ght6b" Nov 02 06:46:17.527338 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:17.527303 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-znms5\" (UniqueName: \"kubernetes.io/projected/26d48294-0eb7-48c7-868c-73068ae0b883-kube-api-access-znms5\") pod \"maas-api-revoke-sub-5fe8b848-ght6b\" (UID: \"26d48294-0eb7-48c7-868c-73068ae0b883\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fe8b848-ght6b" Nov 02 06:46:17.807278 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:17.807181 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fe8b848-ght6b" Nov 02 06:46:18.268094 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:18.267083 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5fe8b848-ght6b"] Nov 02 06:46:18.269494 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:46:18.269469 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26d48294_0eb7_48c7_868c_73068ae0b883.slice/crio-ca49e2d76aee19cc7d6f75c8570b7c814889c3f5d5c0b10933ec3dbcedbf8254 WatchSource:0}: Error finding container ca49e2d76aee19cc7d6f75c8570b7c814889c3f5d5c0b10933ec3dbcedbf8254: Status 404 returned error can't find the container with id ca49e2d76aee19cc7d6f75c8570b7c814889c3f5d5c0b10933ec3dbcedbf8254 Nov 02 06:46:18.855892 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:18.855799 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fe8b848-ght6b" event={"ID":"26d48294-0eb7-48c7-868c-73068ae0b883","Type":"ContainerStarted","Data":"e665c4a934ceb1ef67cd50b2b11ce22fe989ee8bbf86d6fb82c5e059b539f54d"} Nov 02 06:46:18.855892 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:18.855857 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fe8b848-ght6b" event={"ID":"26d48294-0eb7-48c7-868c-73068ae0b883","Type":"ContainerStarted","Data":"ca49e2d76aee19cc7d6f75c8570b7c814889c3f5d5c0b10933ec3dbcedbf8254"} Nov 02 06:46:18.972580 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:18.972524 5916 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fe8b848-ght6b" podStartSLOduration=1.972509442 podStartE2EDuration="1.972509442s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:46:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:46:18.971484217 +0000 UTC m=+1767.243304610" watchObservedRunningTime="2026-11-02 06:46:18.972509442 +0000 UTC m=+1767.244329835" Nov 02 06:46:19.864972 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:19.864897 5916 generic.go:356] "Generic (PLEG): container finished" podID="26d48294-0eb7-48c7-868c-73068ae0b883" containerID="e665c4a934ceb1ef67cd50b2b11ce22fe989ee8bbf86d6fb82c5e059b539f54d" exitCode=0 Nov 02 06:46:19.865407 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:19.865013 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fe8b848-ght6b" event={"ID":"26d48294-0eb7-48c7-868c-73068ae0b883","Type":"ContainerDied","Data":"e665c4a934ceb1ef67cd50b2b11ce22fe989ee8bbf86d6fb82c5e059b539f54d"} Nov 02 06:46:19.895614 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:46:19.895583 5916 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p98cq" containerName="extractor" Nov 02 06:46:21.044332 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:21.044306 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fe8b848-ght6b" Nov 02 06:46:21.178940 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:21.178870 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/26d48294-0eb7-48c7-868c-73068ae0b883-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26d48294-0eb7-48c7-868c-73068ae0b883-maas-api-xks-ca\") pod \"26d48294-0eb7-48c7-868c-73068ae0b883\" (UID: \"26d48294-0eb7-48c7-868c-73068ae0b883\") " Nov 02 06:46:21.179202 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:21.179177 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26d48294-0eb7-48c7-868c-73068ae0b883-kube-api-access-znms5\" (UniqueName: \"kubernetes.io/projected/26d48294-0eb7-48c7-868c-73068ae0b883-kube-api-access-znms5\") pod \"26d48294-0eb7-48c7-868c-73068ae0b883\" (UID: \"26d48294-0eb7-48c7-868c-73068ae0b883\") " Nov 02 06:46:21.179360 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:21.179228 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26d48294-0eb7-48c7-868c-73068ae0b883-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26d48294-0eb7-48c7-868c-73068ae0b883-maas-api-openshift-ca\") pod \"26d48294-0eb7-48c7-868c-73068ae0b883\" (UID: \"26d48294-0eb7-48c7-868c-73068ae0b883\") " Nov 02 06:46:21.179981 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:21.179949 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26d48294-0eb7-48c7-868c-73068ae0b883-maas-api-openshift-ca" pod "26d48294-0eb7-48c7-868c-73068ae0b883" (UID: "26d48294-0eb7-48c7-868c-73068ae0b883"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:46:21.181404 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:21.181382 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26d48294-0eb7-48c7-868c-73068ae0b883-kube-api-access-znms5" pod "26d48294-0eb7-48c7-868c-73068ae0b883" (UID: "26d48294-0eb7-48c7-868c-73068ae0b883"). InnerVolumeSpecName "kube-api-access-znms5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:46:21.181652 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:21.181635 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26d48294-0eb7-48c7-868c-73068ae0b883-maas-api-xks-ca" pod "26d48294-0eb7-48c7-868c-73068ae0b883" (UID: "26d48294-0eb7-48c7-868c-73068ae0b883"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:46:21.281574 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:21.281549 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-znms5\" (UniqueName: \"kubernetes.io/projected/26d48294-0eb7-48c7-868c-73068ae0b883-kube-api-access-znms5\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:46:21.281574 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:21.281574 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26d48294-0eb7-48c7-868c-73068ae0b883-maas-api-openshift-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:46:21.281747 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:21.281584 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26d48294-0eb7-48c7-868c-73068ae0b883-maas-api-xks-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:46:21.883125 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:21.883093 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fe8b848-ght6b" Nov 02 06:46:21.883125 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:21.883110 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fe8b848-ght6b" event={"ID":"26d48294-0eb7-48c7-868c-73068ae0b883","Type":"ContainerDied","Data":"ca49e2d76aee19cc7d6f75c8570b7c814889c3f5d5c0b10933ec3dbcedbf8254"} Nov 02 06:46:21.883321 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:21.883175 5916 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ca49e2d76aee19cc7d6f75c8570b7c814889c3f5d5c0b10933ec3dbcedbf8254" Nov 02 06:46:32.428972 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:32.428899 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e5502564-65855c7c98-q8rn7"] Nov 02 06:46:32.429442 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:32.429130 5916 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e5502564-65855c7c98-q8rn7" podUID="9edf52d8-d09b-40c8-84df-32020447afd2" containerName="maas-api" containerID="cri-o://0afbcc0dd88b8237c95dd526254ab33d56af99d0e678228800ddff5a8579fd71" gracePeriod=30 Nov 02 06:46:32.816101 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:32.816075 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e5502564-65855c7c98-q8rn7" Nov 02 06:46:32.919937 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:32.919207 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9edf52d8-d09b-40c8-84df-32020447afd2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9edf52d8-d09b-40c8-84df-32020447afd2-metrics-certs\") pod \"9edf52d8-d09b-40c8-84df-32020447afd2\" (UID: \"9edf52d8-d09b-40c8-84df-32020447afd2\") " Nov 02 06:46:32.960953 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:32.940355 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9edf52d8-d09b-40c8-84df-32020447afd2-metrics-certs" pod "9edf52d8-d09b-40c8-84df-32020447afd2" (UID: "9edf52d8-d09b-40c8-84df-32020447afd2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:46:33.023760 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:33.023684 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9edf52d8-d09b-40c8-84df-32020447afd2-kube-api-access-gkspc\" (UniqueName: \"kubernetes.io/projected/9edf52d8-d09b-40c8-84df-32020447afd2-kube-api-access-gkspc\") pod \"9edf52d8-d09b-40c8-84df-32020447afd2\" (UID: \"9edf52d8-d09b-40c8-84df-32020447afd2\") " Nov 02 06:46:33.024321 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:33.024215 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9edf52d8-d09b-40c8-84df-32020447afd2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9edf52d8-d09b-40c8-84df-32020447afd2-openshift-service-ca\") pod \"9edf52d8-d09b-40c8-84df-32020447afd2\" (UID: \"9edf52d8-d09b-40c8-84df-32020447afd2\") " Nov 02 06:46:33.024544 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:33.024299 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9edf52d8-d09b-40c8-84df-32020447afd2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9edf52d8-d09b-40c8-84df-32020447afd2-maas-api-tls\") pod \"9edf52d8-d09b-40c8-84df-32020447afd2\" (UID: \"9edf52d8-d09b-40c8-84df-32020447afd2\") " Nov 02 06:46:33.025221 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:33.025194 5916 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9edf52d8-d09b-40c8-84df-32020447afd2-metrics-certs\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:46:33.027067 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:33.027016 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9edf52d8-d09b-40c8-84df-32020447afd2-openshift-service-ca" pod "9edf52d8-d09b-40c8-84df-32020447afd2" (UID: "9edf52d8-d09b-40c8-84df-32020447afd2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:46:33.049071 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:33.045689 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9edf52d8-d09b-40c8-84df-32020447afd2-kube-api-access-gkspc" pod "9edf52d8-d09b-40c8-84df-32020447afd2" (UID: "9edf52d8-d09b-40c8-84df-32020447afd2"). InnerVolumeSpecName "kube-api-access-gkspc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:46:33.080785 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:33.080749 5916 generic.go:356] "Generic (PLEG): container finished" podID="9edf52d8-d09b-40c8-84df-32020447afd2" containerID="0afbcc0dd88b8237c95dd526254ab33d56af99d0e678228800ddff5a8579fd71" exitCode=0 Nov 02 06:46:33.081281 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:33.081254 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e5502564-65855c7c98-q8rn7" event={"ID":"9edf52d8-d09b-40c8-84df-32020447afd2","Type":"ContainerDied","Data":"0afbcc0dd88b8237c95dd526254ab33d56af99d0e678228800ddff5a8579fd71"} Nov 02 06:46:33.082788 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:33.082756 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e5502564-65855c7c98-q8rn7" event={"ID":"9edf52d8-d09b-40c8-84df-32020447afd2","Type":"ContainerDied","Data":"e2de3aa80d655d3da850431cddaef11eddd2f14d88317e32362829f29295aff5"} Nov 02 06:46:33.100952 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:33.084946 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e5502564-65855c7c98-q8rn7" Nov 02 06:46:33.101700 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:33.084970 5916 scope.go:122] "RemoveContainer" containerID="0afbcc0dd88b8237c95dd526254ab33d56af99d0e678228800ddff5a8579fd71" Nov 02 06:46:33.114959 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:33.107497 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5151c1f-7q6dl"] Nov 02 06:46:33.134221 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:33.133464 5916 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9edf52d8-d09b-40c8-84df-32020447afd2-openshift-service-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:46:33.134221 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:33.133501 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gkspc\" (UniqueName: \"kubernetes.io/projected/9edf52d8-d09b-40c8-84df-32020447afd2-kube-api-access-gkspc\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:46:33.143597 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:33.136435 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9edf52d8-d09b-40c8-84df-32020447afd2-maas-api-tls" pod "9edf52d8-d09b-40c8-84df-32020447afd2" (UID: "9edf52d8-d09b-40c8-84df-32020447afd2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:46:33.143597 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:33.137628 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5151c1f-7q6dl"] Nov 02 06:46:33.165090 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:33.165039 5916 scope.go:122] "RemoveContainer" containerID="0afbcc0dd88b8237c95dd526254ab33d56af99d0e678228800ddff5a8579fd71" Nov 02 06:46:33.165751 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:46:33.165711 5916 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0afbcc0dd88b8237c95dd526254ab33d56af99d0e678228800ddff5a8579fd71\": container with ID starting with 0afbcc0dd88b8237c95dd526254ab33d56af99d0e678228800ddff5a8579fd71 not found: ID does not exist" containerID="0afbcc0dd88b8237c95dd526254ab33d56af99d0e678228800ddff5a8579fd71" Nov 02 06:46:33.165947 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:33.165759 5916 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0afbcc0dd88b8237c95dd526254ab33d56af99d0e678228800ddff5a8579fd71"} err="failed to get container status \"0afbcc0dd88b8237c95dd526254ab33d56af99d0e678228800ddff5a8579fd71\": rpc error: code = NotFound desc = could not find container \"0afbcc0dd88b8237c95dd526254ab33d56af99d0e678228800ddff5a8579fd71\": container with ID starting with 0afbcc0dd88b8237c95dd526254ab33d56af99d0e678228800ddff5a8579fd71 not found: ID does not exist" Nov 02 06:46:33.234595 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:33.234567 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9edf52d8-d09b-40c8-84df-32020447afd2-maas-api-tls\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:46:33.524149 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:33.524118 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e5502564-65855c7c98-q8rn7"] Nov 02 06:46:33.573565 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:33.573534 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e5502564-65855c7c98-q8rn7"] Nov 02 06:46:33.844156 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:33.844064 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9edf52d8-d09b-40c8-84df-32020447afd2" path="/var/lib/kubelet/pods/9edf52d8-d09b-40c8-84df-32020447afd2/volumes" Nov 02 06:46:33.844770 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:33.844750 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b45f09ed-75c1-468b-a9ad-f1994301a61c" path="/var/lib/kubelet/pods/b45f09ed-75c1-468b-a9ad-f1994301a61c/volumes" Nov 02 06:46:34.803144 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:34.803111 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e5502564-mfnrs"] Nov 02 06:46:34.853387 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:34.853352 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e5502564-mfnrs"] Nov 02 06:46:35.845500 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:35.845466 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e140326f-6f77-46ff-b6a0-c853c0c118e6" path="/var/lib/kubelet/pods/e140326f-6f77-46ff-b6a0-c853c0c118e6/volumes" Nov 02 06:46:49.895248 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:46:49.895213 5916 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p98cq" containerName="extractor" Nov 02 06:46:53.098479 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:53.098449 5916 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-csfhg_17dee72b-2f47-4016-8ffe-3a89934e66ee/console-operator/2.log" Nov 02 06:46:53.108128 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:53.108100 5916 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-csfhg_17dee72b-2f47-4016-8ffe-3a89934e66ee/console-operator/2.log" Nov 02 06:46:53.993240 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:53.993206 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ead6d32-5clnf"] Nov 02 06:46:53.993852 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:53.993838 5916 cpu_manager.go:414] "containerMap: removing container" podUID="26d48294-0eb7-48c7-868c-73068ae0b883" containerName="revoke-keys" Nov 02 06:46:53.993931 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:53.993861 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26d48294-0eb7-48c7-868c-73068ae0b883" containerName="revoke-keys" Nov 02 06:46:53.993931 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:53.993889 5916 cpu_manager.go:414] "containerMap: removing container" podUID="9edf52d8-d09b-40c8-84df-32020447afd2" containerName="maas-api" Nov 02 06:46:53.993931 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:53.993895 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9edf52d8-d09b-40c8-84df-32020447afd2" containerName="maas-api" Nov 02 06:46:53.994049 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:53.994038 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="26d48294-0eb7-48c7-868c-73068ae0b883" containerName="revoke-keys" Nov 02 06:46:53.994087 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:53.994054 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="9edf52d8-d09b-40c8-84df-32020447afd2" containerName="maas-api" Nov 02 06:46:53.997797 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:53.997780 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ead6d32-5clnf" Nov 02 06:46:54.055993 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:54.055954 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s599h\" (UniqueName: \"kubernetes.io/projected/44fa3493-ecc1-4bcb-a76a-b7d0c2d53540-kube-api-access-s599h\") pod \"maas-api-revoke-sub-8ead6d32-5clnf\" (UID: \"44fa3493-ecc1-4bcb-a76a-b7d0c2d53540\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ead6d32-5clnf" Nov 02 06:46:54.056216 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:54.056064 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44fa3493-ecc1-4bcb-a76a-b7d0c2d53540-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8ead6d32-5clnf\" (UID: \"44fa3493-ecc1-4bcb-a76a-b7d0c2d53540\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ead6d32-5clnf" Nov 02 06:46:54.056216 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:54.056107 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44fa3493-ecc1-4bcb-a76a-b7d0c2d53540-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8ead6d32-5clnf\" (UID: \"44fa3493-ecc1-4bcb-a76a-b7d0c2d53540\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ead6d32-5clnf" Nov 02 06:46:54.056469 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:54.056449 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ead6d32-5clnf"] Nov 02 06:46:54.099966 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:54.099932 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 06:46:54.099966 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:54.099945 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-68l4r\"" Nov 02 06:46:54.157338 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:54.157298 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s599h\" (UniqueName: \"kubernetes.io/projected/44fa3493-ecc1-4bcb-a76a-b7d0c2d53540-kube-api-access-s599h\") pod \"maas-api-revoke-sub-8ead6d32-5clnf\" (UID: \"44fa3493-ecc1-4bcb-a76a-b7d0c2d53540\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ead6d32-5clnf" Nov 02 06:46:54.157526 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:54.157409 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44fa3493-ecc1-4bcb-a76a-b7d0c2d53540-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8ead6d32-5clnf\" (UID: \"44fa3493-ecc1-4bcb-a76a-b7d0c2d53540\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ead6d32-5clnf" Nov 02 06:46:54.157526 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:54.157452 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44fa3493-ecc1-4bcb-a76a-b7d0c2d53540-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8ead6d32-5clnf\" (UID: \"44fa3493-ecc1-4bcb-a76a-b7d0c2d53540\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ead6d32-5clnf" Nov 02 06:46:54.158138 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:54.158112 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44fa3493-ecc1-4bcb-a76a-b7d0c2d53540-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8ead6d32-5clnf\" (UID: \"44fa3493-ecc1-4bcb-a76a-b7d0c2d53540\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ead6d32-5clnf" Nov 02 06:46:54.160031 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:54.160003 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44fa3493-ecc1-4bcb-a76a-b7d0c2d53540-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8ead6d32-5clnf\" (UID: \"44fa3493-ecc1-4bcb-a76a-b7d0c2d53540\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ead6d32-5clnf" Nov 02 06:46:54.214434 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:54.214408 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s599h\" (UniqueName: \"kubernetes.io/projected/44fa3493-ecc1-4bcb-a76a-b7d0c2d53540-kube-api-access-s599h\") pod \"maas-api-revoke-sub-8ead6d32-5clnf\" (UID: \"44fa3493-ecc1-4bcb-a76a-b7d0c2d53540\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ead6d32-5clnf" Nov 02 06:46:54.311624 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:54.311540 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ead6d32-5clnf" Nov 02 06:46:54.658651 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:54.658616 5916 scope.go:122] "RemoveContainer" containerID="2813e871dfb65bea620ed582fdb67b92a5e845d0da5db47b6770fe6763ef68f3" Nov 02 06:46:54.668957 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:54.668931 5916 scope.go:122] "RemoveContainer" containerID="f4fd08872d3f9a860ecbd8cfe89ae32d70f6bda5f5d9990ff5ef4a008f698b70" Nov 02 06:46:54.702778 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:46:54.702751 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44fa3493_ecc1_4bcb_a76a_b7d0c2d53540.slice/crio-d5961aa6e720239b47aa38ee71a1591c7c860ce985c606931d04a453003f5e3c WatchSource:0}: Error finding container d5961aa6e720239b47aa38ee71a1591c7c860ce985c606931d04a453003f5e3c: Status 404 returned error can't find the container with id d5961aa6e720239b47aa38ee71a1591c7c860ce985c606931d04a453003f5e3c Nov 02 06:46:54.705346 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:54.705310 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ead6d32-5clnf"] Nov 02 06:46:55.328358 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:55.327205 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ead6d32-5clnf" event={"ID":"44fa3493-ecc1-4bcb-a76a-b7d0c2d53540","Type":"ContainerStarted","Data":"5fdc52ba9477367a2a7cb6fe24e449629b9df2540c07030e4fc4997c648b534e"} Nov 02 06:46:55.328358 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:55.327273 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ead6d32-5clnf" event={"ID":"44fa3493-ecc1-4bcb-a76a-b7d0c2d53540","Type":"ContainerStarted","Data":"d5961aa6e720239b47aa38ee71a1591c7c860ce985c606931d04a453003f5e3c"} Nov 02 06:46:55.442132 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:55.442076 5916 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ead6d32-5clnf" podStartSLOduration=2.442057853 podStartE2EDuration="2.442057853s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:46:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:46:55.440860164 +0000 UTC m=+1803.712680561" watchObservedRunningTime="2026-11-02 06:46:55.442057853 +0000 UTC m=+1803.713878247" Nov 02 06:46:56.338593 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:56.338558 5916 generic.go:356] "Generic (PLEG): container finished" podID="44fa3493-ecc1-4bcb-a76a-b7d0c2d53540" containerID="5fdc52ba9477367a2a7cb6fe24e449629b9df2540c07030e4fc4997c648b534e" exitCode=0 Nov 02 06:46:56.339136 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:56.338616 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ead6d32-5clnf" event={"ID":"44fa3493-ecc1-4bcb-a76a-b7d0c2d53540","Type":"ContainerDied","Data":"5fdc52ba9477367a2a7cb6fe24e449629b9df2540c07030e4fc4997c648b534e"} Nov 02 06:46:57.510159 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:57.510130 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ead6d32-5clnf" Nov 02 06:46:57.611784 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:57.611705 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/44fa3493-ecc1-4bcb-a76a-b7d0c2d53540-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44fa3493-ecc1-4bcb-a76a-b7d0c2d53540-maas-api-xks-ca\") pod \"44fa3493-ecc1-4bcb-a76a-b7d0c2d53540\" (UID: \"44fa3493-ecc1-4bcb-a76a-b7d0c2d53540\") " Nov 02 06:46:57.611969 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:57.611873 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44fa3493-ecc1-4bcb-a76a-b7d0c2d53540-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44fa3493-ecc1-4bcb-a76a-b7d0c2d53540-maas-api-openshift-ca\") pod \"44fa3493-ecc1-4bcb-a76a-b7d0c2d53540\" (UID: \"44fa3493-ecc1-4bcb-a76a-b7d0c2d53540\") " Nov 02 06:46:57.611969 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:57.611958 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44fa3493-ecc1-4bcb-a76a-b7d0c2d53540-kube-api-access-s599h\" (UniqueName: \"kubernetes.io/projected/44fa3493-ecc1-4bcb-a76a-b7d0c2d53540-kube-api-access-s599h\") pod \"44fa3493-ecc1-4bcb-a76a-b7d0c2d53540\" (UID: \"44fa3493-ecc1-4bcb-a76a-b7d0c2d53540\") " Nov 02 06:46:57.612211 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:57.612186 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44fa3493-ecc1-4bcb-a76a-b7d0c2d53540-maas-api-openshift-ca" pod "44fa3493-ecc1-4bcb-a76a-b7d0c2d53540" (UID: "44fa3493-ecc1-4bcb-a76a-b7d0c2d53540"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:46:57.612459 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:57.612442 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44fa3493-ecc1-4bcb-a76a-b7d0c2d53540-maas-api-openshift-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:46:57.613787 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:57.613764 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44fa3493-ecc1-4bcb-a76a-b7d0c2d53540-maas-api-xks-ca" pod "44fa3493-ecc1-4bcb-a76a-b7d0c2d53540" (UID: "44fa3493-ecc1-4bcb-a76a-b7d0c2d53540"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:46:57.614053 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:57.614034 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44fa3493-ecc1-4bcb-a76a-b7d0c2d53540-kube-api-access-s599h" pod "44fa3493-ecc1-4bcb-a76a-b7d0c2d53540" (UID: "44fa3493-ecc1-4bcb-a76a-b7d0c2d53540"). InnerVolumeSpecName "kube-api-access-s599h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:46:57.714005 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:57.713973 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s599h\" (UniqueName: \"kubernetes.io/projected/44fa3493-ecc1-4bcb-a76a-b7d0c2d53540-kube-api-access-s599h\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:46:57.714005 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:57.714000 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44fa3493-ecc1-4bcb-a76a-b7d0c2d53540-maas-api-xks-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:46:58.360037 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:58.360005 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ead6d32-5clnf" Nov 02 06:46:58.360250 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:58.360030 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ead6d32-5clnf" event={"ID":"44fa3493-ecc1-4bcb-a76a-b7d0c2d53540","Type":"ContainerDied","Data":"d5961aa6e720239b47aa38ee71a1591c7c860ce985c606931d04a453003f5e3c"} Nov 02 06:46:58.360250 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:46:58.360082 5916 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d5961aa6e720239b47aa38ee71a1591c7c860ce985c606931d04a453003f5e3c" Nov 02 06:47:06.652251 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:47:06.652212 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-wk9b2"] Nov 02 06:47:06.652749 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:47:06.652447 5916 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-wk9b2" podUID="6d75560c-3f13-4f05-9e64-b78fdb08e49c" containerName="istio-proxy" containerID="cri-o://750beb3a20eea2be0e840f58c20768ab933dc29bf3da021ae61f7b76635e1368" gracePeriod=30 Nov 02 06:47:07.949695 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:47:07.949650 5916 patch_prober.go:29] interesting pod/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-wk9b2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 06:47:07.950265 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:47:07.949735 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-wk9b2" podUID="6d75560c-3f13-4f05-9e64-b78fdb08e49c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 06:47:09.076096 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:47:09.076052 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-37b53e-765d6b9f5c-7pb64"] Nov 02 06:47:09.076776 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:47:09.076756 5916 cpu_manager.go:414] "containerMap: removing container" podUID="44fa3493-ecc1-4bcb-a76a-b7d0c2d53540" containerName="revoke-keys" Nov 02 06:47:09.076776 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:47:09.076774 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44fa3493-ecc1-4bcb-a76a-b7d0c2d53540" containerName="revoke-keys" Nov 02 06:47:09.076971 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:47:09.076873 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="44fa3493-ecc1-4bcb-a76a-b7d0c2d53540" containerName="revoke-keys" Nov 02 06:47:09.080378 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:47:09.080359 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-37b53e-765d6b9f5c-7pb64" Nov 02 06:47:09.186150 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:47:09.186116 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-37b53e-serving-cert\"" Nov 02 06:47:09.190373 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:47:09.190348 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-37b53e-765d6b9f5c-7pb64"] Nov 02 06:47:09.214012 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:47:09.213979 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1db3e7e8-ea83-42fa-a1d0-88ea901f78aa-maas-api-tls\") pod \"maas-api-e2e-api-b-37b53e-765d6b9f5c-7pb64\" (UID: \"1db3e7e8-ea83-42fa-a1d0-88ea901f78aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-37b53e-765d6b9f5c-7pb64" Nov 02 06:47:09.214216 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:47:09.214144 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1db3e7e8-ea83-42fa-a1d0-88ea901f78aa-openshift-service-ca\") pod \"maas-api-e2e-api-b-37b53e-765d6b9f5c-7pb64\" (UID: \"1db3e7e8-ea83-42fa-a1d0-88ea901f78aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-37b53e-765d6b9f5c-7pb64" Nov 02 06:47:09.214284 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:47:09.214249 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1db3e7e8-ea83-42fa-a1d0-88ea901f78aa-metrics-certs\") pod \"maas-api-e2e-api-b-37b53e-765d6b9f5c-7pb64\" (UID: \"1db3e7e8-ea83-42fa-a1d0-88ea901f78aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-37b53e-765d6b9f5c-7pb64" Nov 02 06:47:09.214342 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:47:09.214322 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kf5px\" (UniqueName: \"kubernetes.io/projected/1db3e7e8-ea83-42fa-a1d0-88ea901f78aa-kube-api-access-kf5px\") pod \"maas-api-e2e-api-b-37b53e-765d6b9f5c-7pb64\" (UID: \"1db3e7e8-ea83-42fa-a1d0-88ea901f78aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-37b53e-765d6b9f5c-7pb64" Nov 02 06:47:09.317519 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:47:09.317216 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1db3e7e8-ea83-42fa-a1d0-88ea901f78aa-openshift-service-ca\") pod \"maas-api-e2e-api-b-37b53e-765d6b9f5c-7pb64\" (UID: \"1db3e7e8-ea83-42fa-a1d0-88ea901f78aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-37b53e-765d6b9f5c-7pb64" Nov 02 06:47:09.317882 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:47:09.317857 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1db3e7e8-ea83-42fa-a1d0-88ea901f78aa-metrics-certs\") pod \"maas-api-e2e-api-b-37b53e-765d6b9f5c-7pb64\" (UID: \"1db3e7e8-ea83-42fa-a1d0-88ea901f78aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-37b53e-765d6b9f5c-7pb64" Nov 02 06:47:09.318124 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:47:09.318094 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kf5px\" (UniqueName: \"kubernetes.io/projected/1db3e7e8-ea83-42fa-a1d0-88ea901f78aa-kube-api-access-kf5px\") pod \"maas-api-e2e-api-b-37b53e-765d6b9f5c-7pb64\" (UID: \"1db3e7e8-ea83-42fa-a1d0-88ea901f78aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-37b53e-765d6b9f5c-7pb64" Nov 02 06:47:09.327635 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:47:09.327556 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1db3e7e8-ea83-42fa-a1d0-88ea901f78aa-maas-api-tls\") pod \"maas-api-e2e-api-b-37b53e-765d6b9f5c-7pb64\" (UID: \"1db3e7e8-ea83-42fa-a1d0-88ea901f78aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-37b53e-765d6b9f5c-7pb64" Nov 02 06:47:09.349928 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:47:09.320200 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1db3e7e8-ea83-42fa-a1d0-88ea901f78aa-openshift-service-ca\") pod \"maas-api-e2e-api-b-37b53e-765d6b9f5c-7pb64\" (UID: \"1db3e7e8-ea83-42fa-a1d0-88ea901f78aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-37b53e-765d6b9f5c-7pb64" Nov 02 06:47:09.353928 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:47:09.353307 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1db3e7e8-ea83-42fa-a1d0-88ea901f78aa-metrics-certs\") pod \"maas-api-e2e-api-b-37b53e-765d6b9f5c-7pb64\" (UID: \"1db3e7e8-ea83-42fa-a1d0-88ea901f78aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-37b53e-765d6b9f5c-7pb64" Nov 02 06:47:09.363095 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:47:09.363038 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1db3e7e8-ea83-42fa-a1d0-88ea901f78aa-maas-api-tls\") pod \"maas-api-e2e-api-b-37b53e-765d6b9f5c-7pb64\" (UID: \"1db3e7e8-ea83-42fa-a1d0-88ea901f78aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-37b53e-765d6b9f5c-7pb64" Nov 02 06:47:09.392844 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:47:09.391304 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kf5px\" (UniqueName: \"kubernetes.io/projected/1db3e7e8-ea83-42fa-a1d0-88ea901f78aa-kube-api-access-kf5px\") pod \"maas-api-e2e-api-b-37b53e-765d6b9f5c-7pb64\" (UID: \"1db3e7e8-ea83-42fa-a1d0-88ea901f78aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-37b53e-765d6b9f5c-7pb64" Nov 02 06:47:09.395099 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:47:09.395073 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-37b53e-765d6b9f5c-7pb64" Nov 02 06:47:09.792162 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:47:09.792139 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-37b53e-765d6b9f5c-7pb64"] Nov 02 06:47:09.792820 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:47:09.792795 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1db3e7e8_ea83_42fa_a1d0_88ea901f78aa.slice/crio-7c872318fcebdd9ce33a5064d00ccca4ca35c093104d18a6524379159f98d68a WatchSource:0}: Error finding container 7c872318fcebdd9ce33a5064d00ccca4ca35c093104d18a6524379159f98d68a: Status 404 returned error can't find the container with id 7c872318fcebdd9ce33a5064d00ccca4ca35c093104d18a6524379159f98d68a Nov 02 06:47:09.794180 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:47:09.794163 5916 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 06:47:10.510470 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:47:10.510437 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-37b53e-765d6b9f5c-7pb64" event={"ID":"1db3e7e8-ea83-42fa-a1d0-88ea901f78aa","Type":"ContainerStarted","Data":"7c872318fcebdd9ce33a5064d00ccca4ca35c093104d18a6524379159f98d68a"} Nov 02 06:47:11.520001 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:47:11.519306 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-37b53e-765d6b9f5c-7pb64" event={"ID":"1db3e7e8-ea83-42fa-a1d0-88ea901f78aa","Type":"ContainerStarted","Data":"dcffa6b2a75bca2f17cfd038bc6f5480c60f4671bd24e955cdf06fe11f0d807e"} Nov 02 06:47:11.520001 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:47:11.519365 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-37b53e-765d6b9f5c-7pb64" Nov 02 06:47:11.642347 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:47:11.642286 5916 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-37b53e-765d6b9f5c-7pb64" podStartSLOduration=1.7448991409999999 podStartE2EDuration="2.642268762s" totalImagesPullingTime="897.369621ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:47:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:47:11.640633356 +0000 UTC m=+1819.912453750" watchObservedRunningTime="2026-11-02 06:47:11.642268762 +0000 UTC m=+1819.914089156" Nov 02 06:47:12.025948 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:47:12.025875 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-wk9b2" Nov 02 06:47:12.102554 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:47:12.102469 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d75560c-3f13-4f05-9e64-b78fdb08e49c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d75560c-3f13-4f05-9e64-b78fdb08e49c-credential-socket\") pod \"6d75560c-3f13-4f05-9e64-b78fdb08e49c\" (UID: \"6d75560c-3f13-4f05-9e64-b78fdb08e49c\") " Nov 02 06:47:12.105950 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:47:12.102850 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d75560c-3f13-4f05-9e64-b78fdb08e49c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d75560c-3f13-4f05-9e64-b78fdb08e49c-istio-data\") pod \"6d75560c-3f13-4f05-9e64-b78fdb08e49c\" (UID: \"6d75560c-3f13-4f05-9e64-b78fdb08e49c\") " Nov 02 06:47:12.106308 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:47:12.106285 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d75560c-3f13-4f05-9e64-b78fdb08e49c-kube-api-access-26kdr\" (UniqueName: \"kubernetes.io/projected/6d75560c-3f13-4f05-9e64-b78fdb08e49c-kube-api-access-26kdr\") pod \"6d75560c-3f13-4f05-9e64-b78fdb08e49c\" (UID: \"6d75560c-3f13-4f05-9e64-b78fdb08e49c\") " Nov 02 06:47:12.106551 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:47:12.106532 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d75560c-3f13-4f05-9e64-b78fdb08e49c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d75560c-3f13-4f05-9e64-b78fdb08e49c-workload-certs\") pod \"6d75560c-3f13-4f05-9e64-b78fdb08e49c\" (UID: \"6d75560c-3f13-4f05-9e64-b78fdb08e49c\") " Nov 02 06:47:12.106728 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:47:12.106702 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d75560c-3f13-4f05-9e64-b78fdb08e49c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d75560c-3f13-4f05-9e64-b78fdb08e49c-istio-envoy\") pod \"6d75560c-3f13-4f05-9e64-b78fdb08e49c\" (UID: \"6d75560c-3f13-4f05-9e64-b78fdb08e49c\") " Nov 02 06:47:12.108226 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:47:12.105542 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d75560c-3f13-4f05-9e64-b78fdb08e49c-istio-data" pod "6d75560c-3f13-4f05-9e64-b78fdb08e49c" (UID: "6d75560c-3f13-4f05-9e64-b78fdb08e49c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:47:12.108490 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:47:12.106805 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d75560c-3f13-4f05-9e64-b78fdb08e49c-credential-socket" pod "6d75560c-3f13-4f05-9e64-b78fdb08e49c" (UID: "6d75560c-3f13-4f05-9e64-b78fdb08e49c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:47:12.108667 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:47:12.108198 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d75560c-3f13-4f05-9e64-b78fdb08e49c-workload-certs" pod "6d75560c-3f13-4f05-9e64-b78fdb08e49c" (UID: "6d75560c-3f13-4f05-9e64-b78fdb08e49c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:47:12.108894 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:47:12.108877 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d75560c-3f13-4f05-9e64-b78fdb08e49c-istio-token\" (UniqueName: \"kubernetes.io/projected/6d75560c-3f13-4f05-9e64-b78fdb08e49c-istio-token\") pod \"6d75560c-3f13-4f05-9e64-b78fdb08e49c\" (UID: \"6d75560c-3f13-4f05-9e64-b78fdb08e49c\") " Nov 02 06:47:12.109661 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:47:12.109640 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d75560c-3f13-4f05-9e64-b78fdb08e49c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d75560c-3f13-4f05-9e64-b78fdb08e49c-workload-socket\") pod \"6d75560c-3f13-4f05-9e64-b78fdb08e49c\" (UID: \"6d75560c-3f13-4f05-9e64-b78fdb08e49c\") " Nov 02 06:47:12.110535 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:47:12.110514 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6d75560c-3f13-4f05-9e64-b78fdb08e49c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d75560c-3f13-4f05-9e64-b78fdb08e49c-istio-podinfo\") pod \"6d75560c-3f13-4f05-9e64-b78fdb08e49c\" (UID: \"6d75560c-3f13-4f05-9e64-b78fdb08e49c\") " Nov 02 06:47:12.110801 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:47:12.110783 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d75560c-3f13-4f05-9e64-b78fdb08e49c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d75560c-3f13-4f05-9e64-b78fdb08e49c-istiod-ca-cert\") pod \"6d75560c-3f13-4f05-9e64-b78fdb08e49c\" (UID: \"6d75560c-3f13-4f05-9e64-b78fdb08e49c\") " Nov 02 06:47:12.111623 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:47:12.111324 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d75560c-3f13-4f05-9e64-b78fdb08e49c-istiod-ca-cert" pod "6d75560c-3f13-4f05-9e64-b78fdb08e49c" (UID: "6d75560c-3f13-4f05-9e64-b78fdb08e49c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:47:12.111830 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:47:12.111564 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d75560c-3f13-4f05-9e64-b78fdb08e49c-workload-socket" pod "6d75560c-3f13-4f05-9e64-b78fdb08e49c" (UID: "6d75560c-3f13-4f05-9e64-b78fdb08e49c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:47:12.114085 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:47:12.114051 5916 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d75560c-3f13-4f05-9e64-b78fdb08e49c-workload-socket\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:47:12.114216 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:47:12.114203 5916 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d75560c-3f13-4f05-9e64-b78fdb08e49c-istiod-ca-cert\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:47:12.114333 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:47:12.114321 5916 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d75560c-3f13-4f05-9e64-b78fdb08e49c-credential-socket\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:47:12.114431 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:47:12.114420 5916 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d75560c-3f13-4f05-9e64-b78fdb08e49c-istio-data\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:47:12.114536 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:47:12.114522 5916 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d75560c-3f13-4f05-9e64-b78fdb08e49c-workload-certs\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:47:12.151202 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:47:12.147665 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6d75560c-3f13-4f05-9e64-b78fdb08e49c-istio-podinfo" pod "6d75560c-3f13-4f05-9e64-b78fdb08e49c" (UID: "6d75560c-3f13-4f05-9e64-b78fdb08e49c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 06:47:12.174585 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:47:12.147901 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d75560c-3f13-4f05-9e64-b78fdb08e49c-istio-token" pod "6d75560c-3f13-4f05-9e64-b78fdb08e49c" (UID: "6d75560c-3f13-4f05-9e64-b78fdb08e49c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:47:12.236203 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:47:12.236172 5916 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d75560c-3f13-4f05-9e64-b78fdb08e49c-istio-podinfo\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:47:12.236457 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:47:12.236440 5916 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d75560c-3f13-4f05-9e64-b78fdb08e49c-istio-token\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:47:12.237218 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:47:12.237152 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d75560c-3f13-4f05-9e64-b78fdb08e49c-istio-envoy" pod "6d75560c-3f13-4f05-9e64-b78fdb08e49c" (UID: "6d75560c-3f13-4f05-9e64-b78fdb08e49c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:47:12.242336 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:47:12.242101 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d75560c-3f13-4f05-9e64-b78fdb08e49c-kube-api-access-26kdr" pod "6d75560c-3f13-4f05-9e64-b78fdb08e49c" (UID: "6d75560c-3f13-4f05-9e64-b78fdb08e49c"). InnerVolumeSpecName "kube-api-access-26kdr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:47:12.338623 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:47:12.338575 5916 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d75560c-3f13-4f05-9e64-b78fdb08e49c-istio-envoy\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:47:12.338623 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:47:12.338613 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26kdr\" (UniqueName: \"kubernetes.io/projected/6d75560c-3f13-4f05-9e64-b78fdb08e49c-kube-api-access-26kdr\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:47:12.528250 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:47:12.528219 5916 generic.go:356] "Generic (PLEG): container finished" podID="6d75560c-3f13-4f05-9e64-b78fdb08e49c" containerID="750beb3a20eea2be0e840f58c20768ab933dc29bf3da021ae61f7b76635e1368" exitCode=0 Nov 02 06:47:12.528759 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:47:12.528285 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-wk9b2" Nov 02 06:47:12.528759 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:47:12.528303 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-wk9b2" event={"ID":"6d75560c-3f13-4f05-9e64-b78fdb08e49c","Type":"ContainerDied","Data":"750beb3a20eea2be0e840f58c20768ab933dc29bf3da021ae61f7b76635e1368"} Nov 02 06:47:12.528759 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:47:12.528368 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-wk9b2" event={"ID":"6d75560c-3f13-4f05-9e64-b78fdb08e49c","Type":"ContainerDied","Data":"afd48f9f92b96ad905bc784179cbf5cd8d9b65e4d5a15221c2ef81b5e1aa90c2"} Nov 02 06:47:12.528759 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:47:12.528396 5916 scope.go:122] "RemoveContainer" containerID="750beb3a20eea2be0e840f58c20768ab933dc29bf3da021ae61f7b76635e1368" Nov 02 06:47:12.543212 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:47:12.542399 5916 scope.go:122] "RemoveContainer" containerID="750beb3a20eea2be0e840f58c20768ab933dc29bf3da021ae61f7b76635e1368" Nov 02 06:47:12.543212 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:47:12.542737 5916 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"750beb3a20eea2be0e840f58c20768ab933dc29bf3da021ae61f7b76635e1368\": container with ID starting with 750beb3a20eea2be0e840f58c20768ab933dc29bf3da021ae61f7b76635e1368 not found: ID does not exist" containerID="750beb3a20eea2be0e840f58c20768ab933dc29bf3da021ae61f7b76635e1368" Nov 02 06:47:12.543212 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:47:12.542775 5916 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"750beb3a20eea2be0e840f58c20768ab933dc29bf3da021ae61f7b76635e1368"} err="failed to get container status \"750beb3a20eea2be0e840f58c20768ab933dc29bf3da021ae61f7b76635e1368\": rpc error: code = NotFound desc = could not find container \"750beb3a20eea2be0e840f58c20768ab933dc29bf3da021ae61f7b76635e1368\": container with ID starting with 750beb3a20eea2be0e840f58c20768ab933dc29bf3da021ae61f7b76635e1368 not found: ID does not exist" Nov 02 06:47:12.698613 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:47:12.698576 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-wk9b2"] Nov 02 06:47:12.762874 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:47:12.762826 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-wk9b2"] Nov 02 06:47:13.847572 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:47:13.847534 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d75560c-3f13-4f05-9e64-b78fdb08e49c" path="/var/lib/kubelet/pods/6d75560c-3f13-4f05-9e64-b78fdb08e49c/volumes" Nov 02 06:47:17.533856 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:47:17.533828 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-37b53e-765d6b9f5c-7pb64" Nov 02 06:47:19.896136 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:47:19.896098 5916 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p98cq" containerName="extractor" Nov 02 06:47:46.199965 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:47:46.199930 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3bf02ebf-59b5c46bd8-2d4n6"] Nov 02 06:47:46.200531 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:47:46.200129 5916 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3bf02ebf-59b5c46bd8-2d4n6" podUID="8df082e9-aecd-42e7-a7b0-a7cc929e269f" containerName="maas-api" containerID="cri-o://df69e499b533701da6fdfcd3a2b60ef0d05ec1cea35de6c93b29f0c81c11bc29" gracePeriod=30 Nov 02 06:47:46.599929 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:47:46.599839 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3bf02ebf-59b5c46bd8-2d4n6" Nov 02 06:47:46.666938 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:47:46.666590 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8df082e9-aecd-42e7-a7b0-a7cc929e269f-kube-api-access-vcz9b\" (UniqueName: \"kubernetes.io/projected/8df082e9-aecd-42e7-a7b0-a7cc929e269f-kube-api-access-vcz9b\") pod \"8df082e9-aecd-42e7-a7b0-a7cc929e269f\" (UID: \"8df082e9-aecd-42e7-a7b0-a7cc929e269f\") " Nov 02 06:47:46.666938 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:47:46.666709 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8df082e9-aecd-42e7-a7b0-a7cc929e269f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8df082e9-aecd-42e7-a7b0-a7cc929e269f-openshift-service-ca\") pod \"8df082e9-aecd-42e7-a7b0-a7cc929e269f\" (UID: \"8df082e9-aecd-42e7-a7b0-a7cc929e269f\") " Nov 02 06:47:46.666938 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:47:46.666828 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8df082e9-aecd-42e7-a7b0-a7cc929e269f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8df082e9-aecd-42e7-a7b0-a7cc929e269f-maas-api-tls\") pod \"8df082e9-aecd-42e7-a7b0-a7cc929e269f\" (UID: \"8df082e9-aecd-42e7-a7b0-a7cc929e269f\") " Nov 02 06:47:46.667293 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:47:46.666951 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8df082e9-aecd-42e7-a7b0-a7cc929e269f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8df082e9-aecd-42e7-a7b0-a7cc929e269f-metrics-certs\") pod \"8df082e9-aecd-42e7-a7b0-a7cc929e269f\" (UID: \"8df082e9-aecd-42e7-a7b0-a7cc929e269f\") " Nov 02 06:47:46.678927 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:47:46.674144 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8df082e9-aecd-42e7-a7b0-a7cc929e269f-openshift-service-ca" pod "8df082e9-aecd-42e7-a7b0-a7cc929e269f" (UID: "8df082e9-aecd-42e7-a7b0-a7cc929e269f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:47:46.682127 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:47:46.682084 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8df082e9-aecd-42e7-a7b0-a7cc929e269f-maas-api-tls" pod "8df082e9-aecd-42e7-a7b0-a7cc929e269f" (UID: "8df082e9-aecd-42e7-a7b0-a7cc929e269f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:47:46.682427 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:47:46.682401 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8df082e9-aecd-42e7-a7b0-a7cc929e269f-metrics-certs" pod "8df082e9-aecd-42e7-a7b0-a7cc929e269f" (UID: "8df082e9-aecd-42e7-a7b0-a7cc929e269f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:47:46.709232 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:47:46.709190 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8df082e9-aecd-42e7-a7b0-a7cc929e269f-kube-api-access-vcz9b" pod "8df082e9-aecd-42e7-a7b0-a7cc929e269f" (UID: "8df082e9-aecd-42e7-a7b0-a7cc929e269f"). InnerVolumeSpecName "kube-api-access-vcz9b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:47:46.769572 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:47:46.769537 5916 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8df082e9-aecd-42e7-a7b0-a7cc929e269f-metrics-certs\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:47:46.769572 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:47:46.769570 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vcz9b\" (UniqueName: \"kubernetes.io/projected/8df082e9-aecd-42e7-a7b0-a7cc929e269f-kube-api-access-vcz9b\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:47:46.769572 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:47:46.769581 5916 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8df082e9-aecd-42e7-a7b0-a7cc929e269f-openshift-service-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:47:46.769873 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:47:46.769590 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8df082e9-aecd-42e7-a7b0-a7cc929e269f-maas-api-tls\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:47:46.861701 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:47:46.861595 5916 generic.go:356] "Generic (PLEG): container finished" podID="8df082e9-aecd-42e7-a7b0-a7cc929e269f" containerID="df69e499b533701da6fdfcd3a2b60ef0d05ec1cea35de6c93b29f0c81c11bc29" exitCode=0 Nov 02 06:47:46.861701 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:47:46.861645 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3bf02ebf-59b5c46bd8-2d4n6" Nov 02 06:47:46.861701 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:47:46.861684 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3bf02ebf-59b5c46bd8-2d4n6" event={"ID":"8df082e9-aecd-42e7-a7b0-a7cc929e269f","Type":"ContainerDied","Data":"df69e499b533701da6fdfcd3a2b60ef0d05ec1cea35de6c93b29f0c81c11bc29"} Nov 02 06:47:46.861981 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:47:46.861740 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3bf02ebf-59b5c46bd8-2d4n6" event={"ID":"8df082e9-aecd-42e7-a7b0-a7cc929e269f","Type":"ContainerDied","Data":"3290de6d3eba499ae9fa91f4a6e6182b4d2cd620179d8d5ff04d2b666dac3a80"} Nov 02 06:47:46.861981 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:47:46.861772 5916 scope.go:122] "RemoveContainer" containerID="df69e499b533701da6fdfcd3a2b60ef0d05ec1cea35de6c93b29f0c81c11bc29" Nov 02 06:47:46.874374 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:47:46.874355 5916 scope.go:122] "RemoveContainer" containerID="df69e499b533701da6fdfcd3a2b60ef0d05ec1cea35de6c93b29f0c81c11bc29" Nov 02 06:47:46.874662 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:47:46.874642 5916 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df69e499b533701da6fdfcd3a2b60ef0d05ec1cea35de6c93b29f0c81c11bc29\": container with ID starting with df69e499b533701da6fdfcd3a2b60ef0d05ec1cea35de6c93b29f0c81c11bc29 not found: ID does not exist" containerID="df69e499b533701da6fdfcd3a2b60ef0d05ec1cea35de6c93b29f0c81c11bc29" Nov 02 06:47:46.874726 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:47:46.874671 5916 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df69e499b533701da6fdfcd3a2b60ef0d05ec1cea35de6c93b29f0c81c11bc29"} err="failed to get container status \"df69e499b533701da6fdfcd3a2b60ef0d05ec1cea35de6c93b29f0c81c11bc29\": rpc error: code = NotFound desc = could not find container \"df69e499b533701da6fdfcd3a2b60ef0d05ec1cea35de6c93b29f0c81c11bc29\": container with ID starting with df69e499b533701da6fdfcd3a2b60ef0d05ec1cea35de6c93b29f0c81c11bc29 not found: ID does not exist" Nov 02 06:47:47.027837 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:47:47.027801 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3bf02ebf-59b5c46bd8-2d4n6"] Nov 02 06:47:47.032397 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:47:47.032372 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3bf02ebf-59b5c46bd8-2d4n6"] Nov 02 06:47:47.844465 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:47:47.844433 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8df082e9-aecd-42e7-a7b0-a7cc929e269f" path="/var/lib/kubelet/pods/8df082e9-aecd-42e7-a7b0-a7cc929e269f/volumes" Nov 02 06:47:48.426272 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:47:48.426225 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3bf02ebf-whd2b"] Nov 02 06:47:48.475814 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:47:48.475777 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3bf02ebf-whd2b"] Nov 02 06:47:49.843838 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:47:49.843804 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f498cdab-f85d-4586-bda6-ff3fa9e07bfb" path="/var/lib/kubelet/pods/f498cdab-f85d-4586-bda6-ff3fa9e07bfb/volumes" Nov 02 06:47:49.894572 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:47:49.894538 5916 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p98cq" containerName="extractor" Nov 02 06:47:54.731167 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:47:54.731131 5916 scope.go:122] "RemoveContainer" containerID="48be77aa50acf8f5a4105ec8d3925bea02af69fe3964db1df96ef6f0195a7656" Nov 02 06:48:06.858887 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:06.853749 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-tkzw9"] Nov 02 06:48:06.858887 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:06.854137 5916 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-tkzw9" podUID="3e77a579-9ee6-4304-85b6-96d292d0bb1d" containerName="istio-proxy" containerID="cri-o://aacb00e236e7ca4a84a68d6dca69058ddd11f3caa6c711d9ba0a23dbb2627d8f" gracePeriod=30 Nov 02 06:48:07.577756 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:07.559186 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-12bfb5-zhcsf"] Nov 02 06:48:07.577756 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:07.575951 5916 cpu_manager.go:414] "containerMap: removing container" podUID="6d75560c-3f13-4f05-9e64-b78fdb08e49c" containerName="istio-proxy" Nov 02 06:48:07.577756 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:07.575971 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d75560c-3f13-4f05-9e64-b78fdb08e49c" containerName="istio-proxy" Nov 02 06:48:07.577756 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:07.576001 5916 cpu_manager.go:414] "containerMap: removing container" podUID="8df082e9-aecd-42e7-a7b0-a7cc929e269f" containerName="maas-api" Nov 02 06:48:07.577756 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:07.576007 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8df082e9-aecd-42e7-a7b0-a7cc929e269f" containerName="maas-api" Nov 02 06:48:07.577756 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:07.576124 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="8df082e9-aecd-42e7-a7b0-a7cc929e269f" containerName="maas-api" Nov 02 06:48:07.577756 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:07.576139 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d75560c-3f13-4f05-9e64-b78fdb08e49c" containerName="istio-proxy" Nov 02 06:48:07.581663 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:07.581639 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-12bfb5-zhcsf" Nov 02 06:48:07.605742 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:07.605712 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-12bfb5-zhcsf"] Nov 02 06:48:07.636163 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:07.636133 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 06:48:07.650882 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:07.650857 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-68l4r\"" Nov 02 06:48:07.690195 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:07.690157 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c0da92a-5e4c-4e80-842a-da1642aabeb6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-12bfb5-zhcsf\" (UID: \"9c0da92a-5e4c-4e80-842a-da1642aabeb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-12bfb5-zhcsf" Nov 02 06:48:07.690356 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:07.690284 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppch2\" (UniqueName: \"kubernetes.io/projected/9c0da92a-5e4c-4e80-842a-da1642aabeb6-kube-api-access-ppch2\") pod \"maas-api-revoke-keys-e2e-inf-a-12bfb5-zhcsf\" (UID: \"9c0da92a-5e4c-4e80-842a-da1642aabeb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-12bfb5-zhcsf" Nov 02 06:48:07.690356 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:07.690321 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c0da92a-5e4c-4e80-842a-da1642aabeb6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-12bfb5-zhcsf\" (UID: \"9c0da92a-5e4c-4e80-842a-da1642aabeb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-12bfb5-zhcsf" Nov 02 06:48:07.791599 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:07.791561 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c0da92a-5e4c-4e80-842a-da1642aabeb6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-12bfb5-zhcsf\" (UID: \"9c0da92a-5e4c-4e80-842a-da1642aabeb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-12bfb5-zhcsf" Nov 02 06:48:07.791775 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:07.791725 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c0da92a-5e4c-4e80-842a-da1642aabeb6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-12bfb5-zhcsf\" (UID: \"9c0da92a-5e4c-4e80-842a-da1642aabeb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-12bfb5-zhcsf" Nov 02 06:48:07.791837 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:07.791795 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppch2\" (UniqueName: \"kubernetes.io/projected/9c0da92a-5e4c-4e80-842a-da1642aabeb6-kube-api-access-ppch2\") pod \"maas-api-revoke-keys-e2e-inf-a-12bfb5-zhcsf\" (UID: \"9c0da92a-5e4c-4e80-842a-da1642aabeb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-12bfb5-zhcsf" Nov 02 06:48:07.792408 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:07.792379 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c0da92a-5e4c-4e80-842a-da1642aabeb6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-12bfb5-zhcsf\" (UID: \"9c0da92a-5e4c-4e80-842a-da1642aabeb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-12bfb5-zhcsf" Nov 02 06:48:07.793878 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:07.793858 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c0da92a-5e4c-4e80-842a-da1642aabeb6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-12bfb5-zhcsf\" (UID: \"9c0da92a-5e4c-4e80-842a-da1642aabeb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-12bfb5-zhcsf" Nov 02 06:48:07.848774 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:07.848703 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppch2\" (UniqueName: \"kubernetes.io/projected/9c0da92a-5e4c-4e80-842a-da1642aabeb6-kube-api-access-ppch2\") pod \"maas-api-revoke-keys-e2e-inf-a-12bfb5-zhcsf\" (UID: \"9c0da92a-5e4c-4e80-842a-da1642aabeb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-12bfb5-zhcsf" Nov 02 06:48:07.900059 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:07.900032 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-12bfb5-zhcsf" Nov 02 06:48:08.287571 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:08.287539 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-12bfb5-zhcsf"] Nov 02 06:48:08.287741 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:48:08.287718 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c0da92a_5e4c_4e80_842a_da1642aabeb6.slice/crio-2b0828b9cd1501f0b98ca64b370a930bd5dc2adf442ab43af0dfd881e254a2ac WatchSource:0}: Error finding container 2b0828b9cd1501f0b98ca64b370a930bd5dc2adf442ab43af0dfd881e254a2ac: Status 404 returned error can't find the container with id 2b0828b9cd1501f0b98ca64b370a930bd5dc2adf442ab43af0dfd881e254a2ac Nov 02 06:48:09.117178 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:09.117145 5916 generic.go:356] "Generic (PLEG): container finished" podID="9c0da92a-5e4c-4e80-842a-da1642aabeb6" containerID="984508b8d1db03bbd0ceae5b5f7490e08d5bd6c4a66253e78d848c1de9d92cde" exitCode=0 Nov 02 06:48:09.117583 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:09.117188 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-12bfb5-zhcsf" event={"ID":"9c0da92a-5e4c-4e80-842a-da1642aabeb6","Type":"ContainerDied","Data":"984508b8d1db03bbd0ceae5b5f7490e08d5bd6c4a66253e78d848c1de9d92cde"} Nov 02 06:48:09.117583 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:09.117248 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-12bfb5-zhcsf" event={"ID":"9c0da92a-5e4c-4e80-842a-da1642aabeb6","Type":"ContainerStarted","Data":"2b0828b9cd1501f0b98ca64b370a930bd5dc2adf442ab43af0dfd881e254a2ac"} Nov 02 06:48:10.306734 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:10.306705 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-12bfb5-zhcsf" Nov 02 06:48:10.427460 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:10.427426 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c0da92a-5e4c-4e80-842a-da1642aabeb6-kube-api-access-ppch2\" (UniqueName: \"kubernetes.io/projected/9c0da92a-5e4c-4e80-842a-da1642aabeb6-kube-api-access-ppch2\") pod \"9c0da92a-5e4c-4e80-842a-da1642aabeb6\" (UID: \"9c0da92a-5e4c-4e80-842a-da1642aabeb6\") " Nov 02 06:48:10.427660 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:10.427541 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c0da92a-5e4c-4e80-842a-da1642aabeb6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c0da92a-5e4c-4e80-842a-da1642aabeb6-maas-api-xks-ca\") pod \"9c0da92a-5e4c-4e80-842a-da1642aabeb6\" (UID: \"9c0da92a-5e4c-4e80-842a-da1642aabeb6\") " Nov 02 06:48:10.427756 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:10.427739 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c0da92a-5e4c-4e80-842a-da1642aabeb6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c0da92a-5e4c-4e80-842a-da1642aabeb6-maas-api-openshift-ca\") pod \"9c0da92a-5e4c-4e80-842a-da1642aabeb6\" (UID: \"9c0da92a-5e4c-4e80-842a-da1642aabeb6\") " Nov 02 06:48:10.428297 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:10.428234 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c0da92a-5e4c-4e80-842a-da1642aabeb6-maas-api-openshift-ca" pod "9c0da92a-5e4c-4e80-842a-da1642aabeb6" (UID: "9c0da92a-5e4c-4e80-842a-da1642aabeb6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:48:10.428450 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:10.428423 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c0da92a-5e4c-4e80-842a-da1642aabeb6-maas-api-openshift-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:48:10.430149 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:10.430114 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c0da92a-5e4c-4e80-842a-da1642aabeb6-maas-api-xks-ca" pod "9c0da92a-5e4c-4e80-842a-da1642aabeb6" (UID: "9c0da92a-5e4c-4e80-842a-da1642aabeb6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:48:10.430261 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:10.430158 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c0da92a-5e4c-4e80-842a-da1642aabeb6-kube-api-access-ppch2" pod "9c0da92a-5e4c-4e80-842a-da1642aabeb6" (UID: "9c0da92a-5e4c-4e80-842a-da1642aabeb6"). InnerVolumeSpecName "kube-api-access-ppch2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:48:10.511760 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:10.511727 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bkklk"] Nov 02 06:48:10.512347 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:10.512334 5916 cpu_manager.go:414] "containerMap: removing container" podUID="9c0da92a-5e4c-4e80-842a-da1642aabeb6" containerName="revoke-keys" Nov 02 06:48:10.512400 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:10.512348 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c0da92a-5e4c-4e80-842a-da1642aabeb6" containerName="revoke-keys" Nov 02 06:48:10.512498 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:10.512488 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c0da92a-5e4c-4e80-842a-da1642aabeb6" containerName="revoke-keys" Nov 02 06:48:10.517241 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:10.517217 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bkklk" Nov 02 06:48:10.532922 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:10.530495 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ppch2\" (UniqueName: \"kubernetes.io/projected/9c0da92a-5e4c-4e80-842a-da1642aabeb6-kube-api-access-ppch2\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:48:10.532922 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:10.530528 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c0da92a-5e4c-4e80-842a-da1642aabeb6-maas-api-xks-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:48:10.631647 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:10.631603 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1173406-c493-422e-a67a-24a9580c9c0b-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-bkklk\" (UID: \"f1173406-c493-422e-a67a-24a9580c9c0b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bkklk" Nov 02 06:48:10.631813 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:10.631734 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4skqf\" (UniqueName: \"kubernetes.io/projected/f1173406-c493-422e-a67a-24a9580c9c0b-kube-api-access-4skqf\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-bkklk\" (UID: \"f1173406-c493-422e-a67a-24a9580c9c0b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bkklk" Nov 02 06:48:10.631878 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:10.631811 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1173406-c493-422e-a67a-24a9580c9c0b-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-bkklk\" (UID: \"f1173406-c493-422e-a67a-24a9580c9c0b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bkklk" Nov 02 06:48:10.631976 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:10.631928 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1173406-c493-422e-a67a-24a9580c9c0b-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-bkklk\" (UID: \"f1173406-c493-422e-a67a-24a9580c9c0b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bkklk" Nov 02 06:48:10.632058 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:10.632040 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1173406-c493-422e-a67a-24a9580c9c0b-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-bkklk\" (UID: \"f1173406-c493-422e-a67a-24a9580c9c0b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bkklk" Nov 02 06:48:10.632115 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:10.632091 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1173406-c493-422e-a67a-24a9580c9c0b-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-bkklk\" (UID: \"f1173406-c493-422e-a67a-24a9580c9c0b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bkklk" Nov 02 06:48:10.632218 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:10.632205 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1173406-c493-422e-a67a-24a9580c9c0b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-bkklk\" (UID: \"f1173406-c493-422e-a67a-24a9580c9c0b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bkklk" Nov 02 06:48:10.632278 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:10.632238 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1173406-c493-422e-a67a-24a9580c9c0b-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-bkklk\" (UID: \"f1173406-c493-422e-a67a-24a9580c9c0b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bkklk" Nov 02 06:48:10.632331 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:10.632273 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1173406-c493-422e-a67a-24a9580c9c0b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-bkklk\" (UID: \"f1173406-c493-422e-a67a-24a9580c9c0b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bkklk" Nov 02 06:48:10.716293 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:10.716216 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bkklk"] Nov 02 06:48:10.733746 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:10.733708 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1173406-c493-422e-a67a-24a9580c9c0b-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-bkklk\" (UID: \"f1173406-c493-422e-a67a-24a9580c9c0b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bkklk" Nov 02 06:48:10.733928 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:10.733788 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4skqf\" (UniqueName: \"kubernetes.io/projected/f1173406-c493-422e-a67a-24a9580c9c0b-kube-api-access-4skqf\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-bkklk\" (UID: \"f1173406-c493-422e-a67a-24a9580c9c0b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bkklk" Nov 02 06:48:10.733928 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:10.733843 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1173406-c493-422e-a67a-24a9580c9c0b-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-bkklk\" (UID: \"f1173406-c493-422e-a67a-24a9580c9c0b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bkklk" Nov 02 06:48:10.733928 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:10.733887 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1173406-c493-422e-a67a-24a9580c9c0b-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-bkklk\" (UID: \"f1173406-c493-422e-a67a-24a9580c9c0b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bkklk" Nov 02 06:48:10.734091 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:10.733981 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1173406-c493-422e-a67a-24a9580c9c0b-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-bkklk\" (UID: \"f1173406-c493-422e-a67a-24a9580c9c0b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bkklk" Nov 02 06:48:10.734091 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:10.734024 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1173406-c493-422e-a67a-24a9580c9c0b-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-bkklk\" (UID: \"f1173406-c493-422e-a67a-24a9580c9c0b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bkklk" Nov 02 06:48:10.734263 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:10.734152 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1173406-c493-422e-a67a-24a9580c9c0b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-bkklk\" (UID: \"f1173406-c493-422e-a67a-24a9580c9c0b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bkklk" Nov 02 06:48:10.734263 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:10.734180 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1173406-c493-422e-a67a-24a9580c9c0b-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-bkklk\" (UID: \"f1173406-c493-422e-a67a-24a9580c9c0b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bkklk" Nov 02 06:48:10.734263 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:10.734217 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1173406-c493-422e-a67a-24a9580c9c0b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-bkklk\" (UID: \"f1173406-c493-422e-a67a-24a9580c9c0b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bkklk" Nov 02 06:48:10.734425 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:10.734380 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1173406-c493-422e-a67a-24a9580c9c0b-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-bkklk\" (UID: \"f1173406-c493-422e-a67a-24a9580c9c0b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bkklk" Nov 02 06:48:10.734483 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:10.734435 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1173406-c493-422e-a67a-24a9580c9c0b-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-bkklk\" (UID: \"f1173406-c493-422e-a67a-24a9580c9c0b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bkklk" Nov 02 06:48:10.734733 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:10.734708 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1173406-c493-422e-a67a-24a9580c9c0b-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-bkklk\" (UID: \"f1173406-c493-422e-a67a-24a9580c9c0b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bkklk" Nov 02 06:48:10.734990 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:10.734971 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1173406-c493-422e-a67a-24a9580c9c0b-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-bkklk\" (UID: \"f1173406-c493-422e-a67a-24a9580c9c0b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bkklk" Nov 02 06:48:10.735094 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:10.735031 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1173406-c493-422e-a67a-24a9580c9c0b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-bkklk\" (UID: \"f1173406-c493-422e-a67a-24a9580c9c0b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bkklk" Nov 02 06:48:10.736895 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:10.736870 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1173406-c493-422e-a67a-24a9580c9c0b-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-bkklk\" (UID: \"f1173406-c493-422e-a67a-24a9580c9c0b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bkklk" Nov 02 06:48:10.737013 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:10.736899 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1173406-c493-422e-a67a-24a9580c9c0b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-bkklk\" (UID: \"f1173406-c493-422e-a67a-24a9580c9c0b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bkklk" Nov 02 06:48:10.792225 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:10.792198 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1173406-c493-422e-a67a-24a9580c9c0b-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-bkklk\" (UID: \"f1173406-c493-422e-a67a-24a9580c9c0b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bkklk" Nov 02 06:48:10.792353 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:10.792328 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4skqf\" (UniqueName: \"kubernetes.io/projected/f1173406-c493-422e-a67a-24a9580c9c0b-kube-api-access-4skqf\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-bkklk\" (UID: \"f1173406-c493-422e-a67a-24a9580c9c0b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bkklk" Nov 02 06:48:10.838548 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:10.838521 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bkklk" Nov 02 06:48:11.137086 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:11.137059 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-12bfb5-zhcsf" Nov 02 06:48:11.137282 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:11.137091 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-12bfb5-zhcsf" event={"ID":"9c0da92a-5e4c-4e80-842a-da1642aabeb6","Type":"ContainerDied","Data":"2b0828b9cd1501f0b98ca64b370a930bd5dc2adf442ab43af0dfd881e254a2ac"} Nov 02 06:48:11.137282 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:11.137135 5916 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2b0828b9cd1501f0b98ca64b370a930bd5dc2adf442ab43af0dfd881e254a2ac" Nov 02 06:48:11.221496 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:11.221465 5916 patch_prober.go:29] interesting pod/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-tkzw9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 06:48:11.221673 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:11.221513 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-tkzw9" podUID="3e77a579-9ee6-4304-85b6-96d292d0bb1d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 06:48:11.270694 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:11.270664 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bkklk"] Nov 02 06:48:11.271523 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:48:11.271497 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1173406_c493_422e_a67a_24a9580c9c0b.slice/crio-6b50126c3eea3c0655146190fff2135b23cd5f2dd8d0fc6410f3d0d1a40b8b52 WatchSource:0}: Error finding container 6b50126c3eea3c0655146190fff2135b23cd5f2dd8d0fc6410f3d0d1a40b8b52: Status 404 returned error can't find the container with id 6b50126c3eea3c0655146190fff2135b23cd5f2dd8d0fc6410f3d0d1a40b8b52 Nov 02 06:48:11.275192 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:11.275146 5916 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 06:48:11.275297 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:11.275236 5916 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 06:48:11.275297 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:11.275283 5916 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 06:48:12.166579 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:12.166483 5916 generic.go:356] "Generic (PLEG): container finished" podID="3e77a579-9ee6-4304-85b6-96d292d0bb1d" containerID="aacb00e236e7ca4a84a68d6dca69058ddd11f3caa6c711d9ba0a23dbb2627d8f" exitCode=0 Nov 02 06:48:12.166579 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:12.166560 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-tkzw9" event={"ID":"3e77a579-9ee6-4304-85b6-96d292d0bb1d","Type":"ContainerDied","Data":"aacb00e236e7ca4a84a68d6dca69058ddd11f3caa6c711d9ba0a23dbb2627d8f"} Nov 02 06:48:12.168518 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:12.168488 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bkklk" event={"ID":"f1173406-c493-422e-a67a-24a9580c9c0b","Type":"ContainerStarted","Data":"73b632741625a10d75b7efd1406f773f337d3b4ddad764a3f38b753c2259cd2f"} Nov 02 06:48:12.168649 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:12.168544 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bkklk" event={"ID":"f1173406-c493-422e-a67a-24a9580c9c0b","Type":"ContainerStarted","Data":"6b50126c3eea3c0655146190fff2135b23cd5f2dd8d0fc6410f3d0d1a40b8b52"} Nov 02 06:48:12.388548 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:12.388502 5916 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bkklk" podStartSLOduration=2.38848694 podStartE2EDuration="2.38848694s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:48:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:48:12.337372254 +0000 UTC m=+1880.609192649" watchObservedRunningTime="2026-11-02 06:48:12.38848694 +0000 UTC m=+1880.660307333" Nov 02 06:48:12.648931 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:12.630216 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-tkzw9" Nov 02 06:48:12.776178 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:12.776106 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e77a579-9ee6-4304-85b6-96d292d0bb1d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e77a579-9ee6-4304-85b6-96d292d0bb1d-credential-socket\") pod \"3e77a579-9ee6-4304-85b6-96d292d0bb1d\" (UID: \"3e77a579-9ee6-4304-85b6-96d292d0bb1d\") " Nov 02 06:48:12.776415 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:12.776396 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e77a579-9ee6-4304-85b6-96d292d0bb1d-kube-api-access-vdjqh\" (UniqueName: \"kubernetes.io/projected/3e77a579-9ee6-4304-85b6-96d292d0bb1d-kube-api-access-vdjqh\") pod \"3e77a579-9ee6-4304-85b6-96d292d0bb1d\" (UID: \"3e77a579-9ee6-4304-85b6-96d292d0bb1d\") " Nov 02 06:48:12.776535 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:12.776507 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e77a579-9ee6-4304-85b6-96d292d0bb1d-credential-socket" pod "3e77a579-9ee6-4304-85b6-96d292d0bb1d" (UID: "3e77a579-9ee6-4304-85b6-96d292d0bb1d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:48:12.776616 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:12.776603 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e77a579-9ee6-4304-85b6-96d292d0bb1d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e77a579-9ee6-4304-85b6-96d292d0bb1d-workload-socket\") pod \"3e77a579-9ee6-4304-85b6-96d292d0bb1d\" (UID: \"3e77a579-9ee6-4304-85b6-96d292d0bb1d\") " Nov 02 06:48:12.776733 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:12.776717 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e77a579-9ee6-4304-85b6-96d292d0bb1d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e77a579-9ee6-4304-85b6-96d292d0bb1d-istio-envoy\") pod \"3e77a579-9ee6-4304-85b6-96d292d0bb1d\" (UID: \"3e77a579-9ee6-4304-85b6-96d292d0bb1d\") " Nov 02 06:48:12.776848 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:12.776835 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e77a579-9ee6-4304-85b6-96d292d0bb1d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e77a579-9ee6-4304-85b6-96d292d0bb1d-workload-certs\") pod \"3e77a579-9ee6-4304-85b6-96d292d0bb1d\" (UID: \"3e77a579-9ee6-4304-85b6-96d292d0bb1d\") " Nov 02 06:48:12.777012 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:12.776998 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e77a579-9ee6-4304-85b6-96d292d0bb1d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e77a579-9ee6-4304-85b6-96d292d0bb1d-istiod-ca-cert\") pod \"3e77a579-9ee6-4304-85b6-96d292d0bb1d\" (UID: \"3e77a579-9ee6-4304-85b6-96d292d0bb1d\") " Nov 02 06:48:12.777155 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:12.777139 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e77a579-9ee6-4304-85b6-96d292d0bb1d-istio-token\" (UniqueName: \"kubernetes.io/projected/3e77a579-9ee6-4304-85b6-96d292d0bb1d-istio-token\") pod \"3e77a579-9ee6-4304-85b6-96d292d0bb1d\" (UID: \"3e77a579-9ee6-4304-85b6-96d292d0bb1d\") " Nov 02 06:48:12.777318 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:12.777304 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e77a579-9ee6-4304-85b6-96d292d0bb1d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e77a579-9ee6-4304-85b6-96d292d0bb1d-istio-data\") pod \"3e77a579-9ee6-4304-85b6-96d292d0bb1d\" (UID: \"3e77a579-9ee6-4304-85b6-96d292d0bb1d\") " Nov 02 06:48:12.777500 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:12.777039 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e77a579-9ee6-4304-85b6-96d292d0bb1d-workload-socket" pod "3e77a579-9ee6-4304-85b6-96d292d0bb1d" (UID: "3e77a579-9ee6-4304-85b6-96d292d0bb1d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:48:12.777620 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:12.777603 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3e77a579-9ee6-4304-85b6-96d292d0bb1d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e77a579-9ee6-4304-85b6-96d292d0bb1d-istio-podinfo\") pod \"3e77a579-9ee6-4304-85b6-96d292d0bb1d\" (UID: \"3e77a579-9ee6-4304-85b6-96d292d0bb1d\") " Nov 02 06:48:12.777734 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:12.777706 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e77a579-9ee6-4304-85b6-96d292d0bb1d-istiod-ca-cert" pod "3e77a579-9ee6-4304-85b6-96d292d0bb1d" (UID: "3e77a579-9ee6-4304-85b6-96d292d0bb1d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:48:12.778028 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:12.778003 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e77a579-9ee6-4304-85b6-96d292d0bb1d-istio-data" pod "3e77a579-9ee6-4304-85b6-96d292d0bb1d" (UID: "3e77a579-9ee6-4304-85b6-96d292d0bb1d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:48:12.778028 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:12.778016 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e77a579-9ee6-4304-85b6-96d292d0bb1d-workload-certs" pod "3e77a579-9ee6-4304-85b6-96d292d0bb1d" (UID: "3e77a579-9ee6-4304-85b6-96d292d0bb1d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:48:12.778485 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:12.778466 5916 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e77a579-9ee6-4304-85b6-96d292d0bb1d-workload-certs\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:48:12.778606 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:12.778585 5916 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e77a579-9ee6-4304-85b6-96d292d0bb1d-istiod-ca-cert\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:48:12.778702 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:12.778691 5916 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e77a579-9ee6-4304-85b6-96d292d0bb1d-istio-data\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:48:12.778785 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:12.778773 5916 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e77a579-9ee6-4304-85b6-96d292d0bb1d-credential-socket\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:48:12.778896 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:12.778881 5916 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e77a579-9ee6-4304-85b6-96d292d0bb1d-workload-socket\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:48:12.780680 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:12.780645 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3e77a579-9ee6-4304-85b6-96d292d0bb1d-istio-podinfo" pod "3e77a579-9ee6-4304-85b6-96d292d0bb1d" (UID: "3e77a579-9ee6-4304-85b6-96d292d0bb1d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 06:48:12.781176 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:12.781146 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e77a579-9ee6-4304-85b6-96d292d0bb1d-istio-envoy" pod "3e77a579-9ee6-4304-85b6-96d292d0bb1d" (UID: "3e77a579-9ee6-4304-85b6-96d292d0bb1d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:48:12.783075 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:12.783052 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e77a579-9ee6-4304-85b6-96d292d0bb1d-istio-token" pod "3e77a579-9ee6-4304-85b6-96d292d0bb1d" (UID: "3e77a579-9ee6-4304-85b6-96d292d0bb1d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:48:12.783684 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:12.783660 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e77a579-9ee6-4304-85b6-96d292d0bb1d-kube-api-access-vdjqh" pod "3e77a579-9ee6-4304-85b6-96d292d0bb1d" (UID: "3e77a579-9ee6-4304-85b6-96d292d0bb1d"). InnerVolumeSpecName "kube-api-access-vdjqh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:48:12.839928 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:12.839037 5916 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bkklk" Nov 02 06:48:12.864926 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:12.860042 5916 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-bkklk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Nov 02 06:48:12.864926 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:12.860109 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bkklk" podUID="f1173406-c493-422e-a67a-24a9580c9c0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Nov 02 06:48:12.881421 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:12.881386 5916 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e77a579-9ee6-4304-85b6-96d292d0bb1d-istio-token\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:48:12.881421 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:12.881413 5916 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e77a579-9ee6-4304-85b6-96d292d0bb1d-istio-podinfo\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:48:12.881421 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:12.881423 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdjqh\" (UniqueName: \"kubernetes.io/projected/3e77a579-9ee6-4304-85b6-96d292d0bb1d-kube-api-access-vdjqh\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:48:12.881752 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:12.881433 5916 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e77a579-9ee6-4304-85b6-96d292d0bb1d-istio-envoy\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:48:13.181227 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:13.181188 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-tkzw9" Nov 02 06:48:13.181227 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:13.181219 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-tkzw9" event={"ID":"3e77a579-9ee6-4304-85b6-96d292d0bb1d","Type":"ContainerDied","Data":"e4a76ecaf9cffe1e14227deea5946d6c53d8780390b471fe754eb0c255e429ce"} Nov 02 06:48:13.181804 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:13.181279 5916 scope.go:122] "RemoveContainer" containerID="aacb00e236e7ca4a84a68d6dca69058ddd11f3caa6c711d9ba0a23dbb2627d8f" Nov 02 06:48:13.371885 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:13.371850 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-tkzw9"] Nov 02 06:48:13.377644 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:13.376763 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-tkzw9"] Nov 02 06:48:13.853927 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:13.846246 5916 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-bkklk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Nov 02 06:48:13.853927 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:13.846304 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bkklk" podUID="f1173406-c493-422e-a67a-24a9580c9c0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Nov 02 06:48:13.864944 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:13.864779 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e77a579-9ee6-4304-85b6-96d292d0bb1d" path="/var/lib/kubelet/pods/3e77a579-9ee6-4304-85b6-96d292d0bb1d/volumes" Nov 02 06:48:14.842928 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:14.842298 5916 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-bkklk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Nov 02 06:48:14.842928 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:14.842375 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bkklk" podUID="f1173406-c493-422e-a67a-24a9580c9c0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Nov 02 06:48:15.845985 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:15.845283 5916 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-bkklk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Nov 02 06:48:15.845985 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:15.845345 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bkklk" podUID="f1173406-c493-422e-a67a-24a9580c9c0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Nov 02 06:48:16.839015 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:16.838979 5916 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-bkklk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Nov 02 06:48:16.839341 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:16.839311 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bkklk" podUID="f1173406-c493-422e-a67a-24a9580c9c0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Nov 02 06:48:17.844557 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:17.844070 5916 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-bkklk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Nov 02 06:48:17.844557 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:17.844126 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bkklk" podUID="f1173406-c493-422e-a67a-24a9580c9c0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Nov 02 06:48:19.662854 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:19.662815 5916 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bkklk" Nov 02 06:48:19.663698 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:19.663671 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bkklk" Nov 02 06:48:19.666175 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:19.666144 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bkklk" Nov 02 06:48:19.896417 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:48:19.896376 5916 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p98cq" containerName="extractor" Nov 02 06:48:35.355189 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:35.354994 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f83fbf8-fhcqz"] Nov 02 06:48:35.355728 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:35.355710 5916 cpu_manager.go:414] "containerMap: removing container" podUID="3e77a579-9ee6-4304-85b6-96d292d0bb1d" containerName="istio-proxy" Nov 02 06:48:35.355775 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:35.355730 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e77a579-9ee6-4304-85b6-96d292d0bb1d" containerName="istio-proxy" Nov 02 06:48:35.355878 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:35.355869 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e77a579-9ee6-4304-85b6-96d292d0bb1d" containerName="istio-proxy" Nov 02 06:48:35.360894 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:35.360874 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f83fbf8-fhcqz" Nov 02 06:48:35.417766 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:35.417739 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f83fbf8-fhcqz"] Nov 02 06:48:35.434571 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:35.434542 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-68l4r\"" Nov 02 06:48:35.434740 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:35.434569 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 06:48:35.444231 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:35.444204 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rbbt\" (UniqueName: \"kubernetes.io/projected/5aa6bbf7-bc9e-4c67-bbfb-bc24f450b185-kube-api-access-7rbbt\") pod \"maas-api-revoke-sub-3f83fbf8-fhcqz\" (UID: \"5aa6bbf7-bc9e-4c67-bbfb-bc24f450b185\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f83fbf8-fhcqz" Nov 02 06:48:35.444366 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:35.444344 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5aa6bbf7-bc9e-4c67-bbfb-bc24f450b185-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3f83fbf8-fhcqz\" (UID: \"5aa6bbf7-bc9e-4c67-bbfb-bc24f450b185\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f83fbf8-fhcqz" Nov 02 06:48:35.444419 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:35.444396 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5aa6bbf7-bc9e-4c67-bbfb-bc24f450b185-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3f83fbf8-fhcqz\" (UID: \"5aa6bbf7-bc9e-4c67-bbfb-bc24f450b185\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f83fbf8-fhcqz" Nov 02 06:48:35.546426 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:35.546380 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5aa6bbf7-bc9e-4c67-bbfb-bc24f450b185-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3f83fbf8-fhcqz\" (UID: \"5aa6bbf7-bc9e-4c67-bbfb-bc24f450b185\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f83fbf8-fhcqz" Nov 02 06:48:35.546627 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:35.546446 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5aa6bbf7-bc9e-4c67-bbfb-bc24f450b185-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3f83fbf8-fhcqz\" (UID: \"5aa6bbf7-bc9e-4c67-bbfb-bc24f450b185\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f83fbf8-fhcqz" Nov 02 06:48:35.546705 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:35.546686 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rbbt\" (UniqueName: \"kubernetes.io/projected/5aa6bbf7-bc9e-4c67-bbfb-bc24f450b185-kube-api-access-7rbbt\") pod \"maas-api-revoke-sub-3f83fbf8-fhcqz\" (UID: \"5aa6bbf7-bc9e-4c67-bbfb-bc24f450b185\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f83fbf8-fhcqz" Nov 02 06:48:35.547798 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:35.547746 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5aa6bbf7-bc9e-4c67-bbfb-bc24f450b185-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3f83fbf8-fhcqz\" (UID: \"5aa6bbf7-bc9e-4c67-bbfb-bc24f450b185\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f83fbf8-fhcqz" Nov 02 06:48:35.556827 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:35.550224 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5aa6bbf7-bc9e-4c67-bbfb-bc24f450b185-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3f83fbf8-fhcqz\" (UID: \"5aa6bbf7-bc9e-4c67-bbfb-bc24f450b185\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f83fbf8-fhcqz" Nov 02 06:48:35.604071 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:35.604048 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rbbt\" (UniqueName: \"kubernetes.io/projected/5aa6bbf7-bc9e-4c67-bbfb-bc24f450b185-kube-api-access-7rbbt\") pod \"maas-api-revoke-sub-3f83fbf8-fhcqz\" (UID: \"5aa6bbf7-bc9e-4c67-bbfb-bc24f450b185\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f83fbf8-fhcqz" Nov 02 06:48:35.676881 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:35.676850 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f83fbf8-fhcqz" Nov 02 06:48:36.069934 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:48:36.069884 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5aa6bbf7_bc9e_4c67_bbfb_bc24f450b185.slice/crio-120ba976bef93b884e4d0868cac9ac36ec7ecb87e56ff889b1c6cdca2652ee6d WatchSource:0}: Error finding container 120ba976bef93b884e4d0868cac9ac36ec7ecb87e56ff889b1c6cdca2652ee6d: Status 404 returned error can't find the container with id 120ba976bef93b884e4d0868cac9ac36ec7ecb87e56ff889b1c6cdca2652ee6d Nov 02 06:48:36.070071 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:36.069970 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f83fbf8-fhcqz"] Nov 02 06:48:36.171613 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:36.171569 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-25spl"] Nov 02 06:48:36.171823 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:36.171797 5916 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-25spl" podUID="3818427d-4da9-4347-8a90-83e34f7ed19d" containerName="istio-proxy" containerID="cri-o://6fb33bb629b56824d71e353bda7d0af5f016efc68d498ac0ce4736efda682b08" gracePeriod=30 Nov 02 06:48:36.384257 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:36.384228 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f83fbf8-fhcqz" event={"ID":"5aa6bbf7-bc9e-4c67-bbfb-bc24f450b185","Type":"ContainerStarted","Data":"120ba976bef93b884e4d0868cac9ac36ec7ecb87e56ff889b1c6cdca2652ee6d"} Nov 02 06:48:37.393617 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:37.393582 5916 generic.go:356] "Generic (PLEG): container finished" podID="5aa6bbf7-bc9e-4c67-bbfb-bc24f450b185" containerID="984f18719a56ab9ec0ca47d5a43d173aed717c790ef8c08289fcb5af0ae4abc1" exitCode=0 Nov 02 06:48:37.394023 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:37.393666 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f83fbf8-fhcqz" event={"ID":"5aa6bbf7-bc9e-4c67-bbfb-bc24f450b185","Type":"ContainerDied","Data":"984f18719a56ab9ec0ca47d5a43d173aed717c790ef8c08289fcb5af0ae4abc1"} Nov 02 06:48:38.565313 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:38.565285 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f83fbf8-fhcqz" Nov 02 06:48:38.688955 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:38.688852 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5aa6bbf7-bc9e-4c67-bbfb-bc24f450b185-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5aa6bbf7-bc9e-4c67-bbfb-bc24f450b185-maas-api-xks-ca\") pod \"5aa6bbf7-bc9e-4c67-bbfb-bc24f450b185\" (UID: \"5aa6bbf7-bc9e-4c67-bbfb-bc24f450b185\") " Nov 02 06:48:38.689088 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:38.688977 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5aa6bbf7-bc9e-4c67-bbfb-bc24f450b185-kube-api-access-7rbbt\" (UniqueName: \"kubernetes.io/projected/5aa6bbf7-bc9e-4c67-bbfb-bc24f450b185-kube-api-access-7rbbt\") pod \"5aa6bbf7-bc9e-4c67-bbfb-bc24f450b185\" (UID: \"5aa6bbf7-bc9e-4c67-bbfb-bc24f450b185\") " Nov 02 06:48:38.689088 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:38.689026 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5aa6bbf7-bc9e-4c67-bbfb-bc24f450b185-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5aa6bbf7-bc9e-4c67-bbfb-bc24f450b185-maas-api-openshift-ca\") pod \"5aa6bbf7-bc9e-4c67-bbfb-bc24f450b185\" (UID: \"5aa6bbf7-bc9e-4c67-bbfb-bc24f450b185\") " Nov 02 06:48:38.689450 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:38.689421 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5aa6bbf7-bc9e-4c67-bbfb-bc24f450b185-maas-api-openshift-ca" pod "5aa6bbf7-bc9e-4c67-bbfb-bc24f450b185" (UID: "5aa6bbf7-bc9e-4c67-bbfb-bc24f450b185"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:48:38.689564 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:38.689543 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5aa6bbf7-bc9e-4c67-bbfb-bc24f450b185-maas-api-openshift-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:48:38.691115 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:38.691089 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5aa6bbf7-bc9e-4c67-bbfb-bc24f450b185-maas-api-xks-ca" pod "5aa6bbf7-bc9e-4c67-bbfb-bc24f450b185" (UID: "5aa6bbf7-bc9e-4c67-bbfb-bc24f450b185"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:48:38.691215 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:38.691140 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5aa6bbf7-bc9e-4c67-bbfb-bc24f450b185-kube-api-access-7rbbt" pod "5aa6bbf7-bc9e-4c67-bbfb-bc24f450b185" (UID: "5aa6bbf7-bc9e-4c67-bbfb-bc24f450b185"). InnerVolumeSpecName "kube-api-access-7rbbt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:48:38.790734 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:38.790699 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5aa6bbf7-bc9e-4c67-bbfb-bc24f450b185-maas-api-xks-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:48:38.790734 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:38.790731 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rbbt\" (UniqueName: \"kubernetes.io/projected/5aa6bbf7-bc9e-4c67-bbfb-bc24f450b185-kube-api-access-7rbbt\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:48:39.410007 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:39.409973 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f83fbf8-fhcqz" event={"ID":"5aa6bbf7-bc9e-4c67-bbfb-bc24f450b185","Type":"ContainerDied","Data":"120ba976bef93b884e4d0868cac9ac36ec7ecb87e56ff889b1c6cdca2652ee6d"} Nov 02 06:48:39.410007 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:39.410004 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f83fbf8-fhcqz" Nov 02 06:48:39.410262 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:39.410022 5916 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="120ba976bef93b884e4d0868cac9ac36ec7ecb87e56ff889b1c6cdca2652ee6d" Nov 02 06:48:41.447479 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:41.447440 5916 generic.go:356] "Generic (PLEG): container finished" podID="3818427d-4da9-4347-8a90-83e34f7ed19d" containerID="6fb33bb629b56824d71e353bda7d0af5f016efc68d498ac0ce4736efda682b08" exitCode=0 Nov 02 06:48:41.448017 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:41.447516 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-25spl" event={"ID":"3818427d-4da9-4347-8a90-83e34f7ed19d","Type":"ContainerDied","Data":"6fb33bb629b56824d71e353bda7d0af5f016efc68d498ac0ce4736efda682b08"} Nov 02 06:48:41.463541 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:41.463516 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-25spl" Nov 02 06:48:41.547948 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:41.547901 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3818427d-4da9-4347-8a90-83e34f7ed19d-kube-api-access-cwjbv\" (UniqueName: \"kubernetes.io/projected/3818427d-4da9-4347-8a90-83e34f7ed19d-kube-api-access-cwjbv\") pod \"3818427d-4da9-4347-8a90-83e34f7ed19d\" (UID: \"3818427d-4da9-4347-8a90-83e34f7ed19d\") " Nov 02 06:48:41.548168 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:41.548004 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3818427d-4da9-4347-8a90-83e34f7ed19d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3818427d-4da9-4347-8a90-83e34f7ed19d-istio-podinfo\") pod \"3818427d-4da9-4347-8a90-83e34f7ed19d\" (UID: \"3818427d-4da9-4347-8a90-83e34f7ed19d\") " Nov 02 06:48:41.548168 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:41.548113 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3818427d-4da9-4347-8a90-83e34f7ed19d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3818427d-4da9-4347-8a90-83e34f7ed19d-istiod-ca-cert\") pod \"3818427d-4da9-4347-8a90-83e34f7ed19d\" (UID: \"3818427d-4da9-4347-8a90-83e34f7ed19d\") " Nov 02 06:48:41.548293 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:41.548224 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3818427d-4da9-4347-8a90-83e34f7ed19d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3818427d-4da9-4347-8a90-83e34f7ed19d-workload-certs\") pod \"3818427d-4da9-4347-8a90-83e34f7ed19d\" (UID: \"3818427d-4da9-4347-8a90-83e34f7ed19d\") " Nov 02 06:48:41.548354 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:41.548325 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3818427d-4da9-4347-8a90-83e34f7ed19d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3818427d-4da9-4347-8a90-83e34f7ed19d-istio-data\") pod \"3818427d-4da9-4347-8a90-83e34f7ed19d\" (UID: \"3818427d-4da9-4347-8a90-83e34f7ed19d\") " Nov 02 06:48:41.548409 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:41.548368 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3818427d-4da9-4347-8a90-83e34f7ed19d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3818427d-4da9-4347-8a90-83e34f7ed19d-istio-envoy\") pod \"3818427d-4da9-4347-8a90-83e34f7ed19d\" (UID: \"3818427d-4da9-4347-8a90-83e34f7ed19d\") " Nov 02 06:48:41.548466 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:41.548411 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3818427d-4da9-4347-8a90-83e34f7ed19d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3818427d-4da9-4347-8a90-83e34f7ed19d-credential-socket\") pod \"3818427d-4da9-4347-8a90-83e34f7ed19d\" (UID: \"3818427d-4da9-4347-8a90-83e34f7ed19d\") " Nov 02 06:48:41.548466 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:41.548441 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3818427d-4da9-4347-8a90-83e34f7ed19d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3818427d-4da9-4347-8a90-83e34f7ed19d-workload-socket\") pod \"3818427d-4da9-4347-8a90-83e34f7ed19d\" (UID: \"3818427d-4da9-4347-8a90-83e34f7ed19d\") " Nov 02 06:48:41.548574 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:41.548484 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3818427d-4da9-4347-8a90-83e34f7ed19d-istio-token\" (UniqueName: \"kubernetes.io/projected/3818427d-4da9-4347-8a90-83e34f7ed19d-istio-token\") pod \"3818427d-4da9-4347-8a90-83e34f7ed19d\" (UID: \"3818427d-4da9-4347-8a90-83e34f7ed19d\") " Nov 02 06:48:41.548628 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:41.548595 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3818427d-4da9-4347-8a90-83e34f7ed19d-istio-data" pod "3818427d-4da9-4347-8a90-83e34f7ed19d" (UID: "3818427d-4da9-4347-8a90-83e34f7ed19d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:48:41.548628 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:41.548527 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3818427d-4da9-4347-8a90-83e34f7ed19d-istiod-ca-cert" pod "3818427d-4da9-4347-8a90-83e34f7ed19d" (UID: "3818427d-4da9-4347-8a90-83e34f7ed19d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:48:41.548800 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:41.548781 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3818427d-4da9-4347-8a90-83e34f7ed19d-workload-certs" pod "3818427d-4da9-4347-8a90-83e34f7ed19d" (UID: "3818427d-4da9-4347-8a90-83e34f7ed19d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:48:41.548858 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:41.548814 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3818427d-4da9-4347-8a90-83e34f7ed19d-credential-socket" pod "3818427d-4da9-4347-8a90-83e34f7ed19d" (UID: "3818427d-4da9-4347-8a90-83e34f7ed19d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:48:41.549189 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:41.549138 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3818427d-4da9-4347-8a90-83e34f7ed19d-workload-socket" pod "3818427d-4da9-4347-8a90-83e34f7ed19d" (UID: "3818427d-4da9-4347-8a90-83e34f7ed19d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:48:41.549279 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:41.549235 5916 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3818427d-4da9-4347-8a90-83e34f7ed19d-istiod-ca-cert\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:48:41.549279 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:41.549251 5916 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3818427d-4da9-4347-8a90-83e34f7ed19d-workload-certs\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:48:41.549279 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:41.549265 5916 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3818427d-4da9-4347-8a90-83e34f7ed19d-istio-data\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:48:41.549279 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:41.549279 5916 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3818427d-4da9-4347-8a90-83e34f7ed19d-credential-socket\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:48:41.550772 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:41.550737 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3818427d-4da9-4347-8a90-83e34f7ed19d-istio-podinfo" pod "3818427d-4da9-4347-8a90-83e34f7ed19d" (UID: "3818427d-4da9-4347-8a90-83e34f7ed19d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 06:48:41.551274 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:41.551242 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3818427d-4da9-4347-8a90-83e34f7ed19d-kube-api-access-cwjbv" pod "3818427d-4da9-4347-8a90-83e34f7ed19d" (UID: "3818427d-4da9-4347-8a90-83e34f7ed19d"). InnerVolumeSpecName "kube-api-access-cwjbv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:48:41.551391 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:41.551343 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3818427d-4da9-4347-8a90-83e34f7ed19d-istio-envoy" pod "3818427d-4da9-4347-8a90-83e34f7ed19d" (UID: "3818427d-4da9-4347-8a90-83e34f7ed19d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:48:41.551762 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:41.551740 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3818427d-4da9-4347-8a90-83e34f7ed19d-istio-token" pod "3818427d-4da9-4347-8a90-83e34f7ed19d" (UID: "3818427d-4da9-4347-8a90-83e34f7ed19d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:48:41.651479 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:41.651440 5916 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3818427d-4da9-4347-8a90-83e34f7ed19d-istio-podinfo\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:48:41.651479 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:41.651481 5916 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3818427d-4da9-4347-8a90-83e34f7ed19d-istio-envoy\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:48:41.651721 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:41.651496 5916 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3818427d-4da9-4347-8a90-83e34f7ed19d-workload-socket\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:48:41.651721 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:41.651510 5916 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3818427d-4da9-4347-8a90-83e34f7ed19d-istio-token\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:48:41.651721 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:41.651524 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwjbv\" (UniqueName: \"kubernetes.io/projected/3818427d-4da9-4347-8a90-83e34f7ed19d-kube-api-access-cwjbv\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:48:42.458260 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:42.458223 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-25spl" Nov 02 06:48:42.458260 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:42.458237 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-25spl" event={"ID":"3818427d-4da9-4347-8a90-83e34f7ed19d","Type":"ContainerDied","Data":"bcbf8a12ca30e35cc71f078c86a7460044efa08161cdd00ac67ab1ec61d6090d"} Nov 02 06:48:42.458828 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:42.458297 5916 scope.go:122] "RemoveContainer" containerID="6fb33bb629b56824d71e353bda7d0af5f016efc68d498ac0ce4736efda682b08" Nov 02 06:48:42.660076 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:42.660043 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-25spl"] Nov 02 06:48:42.709866 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:42.709790 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-25spl"] Nov 02 06:48:43.847921 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:43.847870 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3818427d-4da9-4347-8a90-83e34f7ed19d" path="/var/lib/kubelet/pods/3818427d-4da9-4347-8a90-83e34f7ed19d/volumes" Nov 02 06:48:49.226090 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:49.226057 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8ecb23-rzfnk"] Nov 02 06:48:49.226764 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:49.226748 5916 cpu_manager.go:414] "containerMap: removing container" podUID="3818427d-4da9-4347-8a90-83e34f7ed19d" containerName="istio-proxy" Nov 02 06:48:49.226814 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:49.226765 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3818427d-4da9-4347-8a90-83e34f7ed19d" containerName="istio-proxy" Nov 02 06:48:49.226814 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:49.226781 5916 cpu_manager.go:414] "containerMap: removing container" podUID="5aa6bbf7-bc9e-4c67-bbfb-bc24f450b185" containerName="revoke-keys" Nov 02 06:48:49.226814 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:49.226786 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5aa6bbf7-bc9e-4c67-bbfb-bc24f450b185" containerName="revoke-keys" Nov 02 06:48:49.226928 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:49.226881 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="3818427d-4da9-4347-8a90-83e34f7ed19d" containerName="istio-proxy" Nov 02 06:48:49.226928 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:49.226892 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="5aa6bbf7-bc9e-4c67-bbfb-bc24f450b185" containerName="revoke-keys" Nov 02 06:48:49.231937 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:49.231900 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8ecb23-rzfnk" Nov 02 06:48:49.289357 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:49.289322 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8ecb23-rzfnk"] Nov 02 06:48:49.334347 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:49.334306 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 06:48:49.334505 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:49.334387 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-68l4r\"" Nov 02 06:48:49.367245 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:49.367208 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb070200-cf13-49f5-94e4-20d0afdfd322-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-8ecb23-rzfnk\" (UID: \"eb070200-cf13-49f5-94e4-20d0afdfd322\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8ecb23-rzfnk" Nov 02 06:48:49.367396 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:49.367365 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb070200-cf13-49f5-94e4-20d0afdfd322-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-8ecb23-rzfnk\" (UID: \"eb070200-cf13-49f5-94e4-20d0afdfd322\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8ecb23-rzfnk" Nov 02 06:48:49.367453 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:49.367395 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dnp5\" (UniqueName: \"kubernetes.io/projected/eb070200-cf13-49f5-94e4-20d0afdfd322-kube-api-access-8dnp5\") pod \"maas-api-revoke-keys-e2e-api-a-8ecb23-rzfnk\" (UID: \"eb070200-cf13-49f5-94e4-20d0afdfd322\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8ecb23-rzfnk" Nov 02 06:48:49.469132 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:49.469102 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb070200-cf13-49f5-94e4-20d0afdfd322-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-8ecb23-rzfnk\" (UID: \"eb070200-cf13-49f5-94e4-20d0afdfd322\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8ecb23-rzfnk" Nov 02 06:48:49.469322 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:49.469140 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dnp5\" (UniqueName: \"kubernetes.io/projected/eb070200-cf13-49f5-94e4-20d0afdfd322-kube-api-access-8dnp5\") pod \"maas-api-revoke-keys-e2e-api-a-8ecb23-rzfnk\" (UID: \"eb070200-cf13-49f5-94e4-20d0afdfd322\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8ecb23-rzfnk" Nov 02 06:48:49.469322 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:49.469305 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb070200-cf13-49f5-94e4-20d0afdfd322-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-8ecb23-rzfnk\" (UID: \"eb070200-cf13-49f5-94e4-20d0afdfd322\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8ecb23-rzfnk" Nov 02 06:48:49.469777 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:49.469755 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb070200-cf13-49f5-94e4-20d0afdfd322-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-8ecb23-rzfnk\" (UID: \"eb070200-cf13-49f5-94e4-20d0afdfd322\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8ecb23-rzfnk" Nov 02 06:48:49.471598 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:49.471575 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb070200-cf13-49f5-94e4-20d0afdfd322-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-8ecb23-rzfnk\" (UID: \"eb070200-cf13-49f5-94e4-20d0afdfd322\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8ecb23-rzfnk" Nov 02 06:48:49.532969 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:49.532099 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dnp5\" (UniqueName: \"kubernetes.io/projected/eb070200-cf13-49f5-94e4-20d0afdfd322-kube-api-access-8dnp5\") pod \"maas-api-revoke-keys-e2e-api-a-8ecb23-rzfnk\" (UID: \"eb070200-cf13-49f5-94e4-20d0afdfd322\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8ecb23-rzfnk" Nov 02 06:48:49.546444 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:49.546408 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8ecb23-rzfnk" Nov 02 06:48:49.897429 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:48:49.897389 5916 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p98cq" containerName="extractor" Nov 02 06:48:49.934046 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:49.934021 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8ecb23-rzfnk"] Nov 02 06:48:49.934837 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:48:49.934812 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb070200_cf13_49f5_94e4_20d0afdfd322.slice/crio-678e89a545a7dbfd431685fcbd35deb180cf588a3f647d59a35850c28c5b3a96 WatchSource:0}: Error finding container 678e89a545a7dbfd431685fcbd35deb180cf588a3f647d59a35850c28c5b3a96: Status 404 returned error can't find the container with id 678e89a545a7dbfd431685fcbd35deb180cf588a3f647d59a35850c28c5b3a96 Nov 02 06:48:50.548330 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:50.548236 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8ecb23-rzfnk" event={"ID":"eb070200-cf13-49f5-94e4-20d0afdfd322","Type":"ContainerStarted","Data":"325eadb393dfd468183817f1c953d320b4e869eedc627c65c983b305687cef09"} Nov 02 06:48:50.548330 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:50.548293 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8ecb23-rzfnk" event={"ID":"eb070200-cf13-49f5-94e4-20d0afdfd322","Type":"ContainerStarted","Data":"678e89a545a7dbfd431685fcbd35deb180cf588a3f647d59a35850c28c5b3a96"} Nov 02 06:48:50.671108 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:50.671054 5916 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8ecb23-rzfnk" podStartSLOduration=1.6710397179999998 podStartE2EDuration="1.671039718s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:48:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:48:50.666678724 +0000 UTC m=+1918.938499096" watchObservedRunningTime="2026-11-02 06:48:50.671039718 +0000 UTC m=+1918.942860164" Nov 02 06:48:51.092630 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:51.092552 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-c955m"] Nov 02 06:48:51.092779 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:51.092757 5916 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-c955m" podUID="12eeae94-c1bf-4f75-bc8d-2170fc882d8e" containerName="istio-proxy" containerID="cri-o://f5a5d7460a73766554bf55fd4276489711bbea9edda84644c823835082903343" gracePeriod=30 Nov 02 06:48:51.143073 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:51.143036 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-w4bhs"] Nov 02 06:48:51.143338 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:51.143291 5916 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-w4bhs" podUID="eb1d132e-7237-4c50-b70b-8f97862d20c3" containerName="istio-proxy" containerID="cri-o://43c8d7a3bf910eefad89dc39e2172de86c4222c34f2651eca5f8f11f9ab43cf6" gracePeriod=30 Nov 02 06:48:51.556890 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:51.556852 5916 generic.go:356] "Generic (PLEG): container finished" podID="eb070200-cf13-49f5-94e4-20d0afdfd322" containerID="325eadb393dfd468183817f1c953d320b4e869eedc627c65c983b305687cef09" exitCode=0 Nov 02 06:48:51.557328 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:51.556951 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8ecb23-rzfnk" event={"ID":"eb070200-cf13-49f5-94e4-20d0afdfd322","Type":"ContainerDied","Data":"325eadb393dfd468183817f1c953d320b4e869eedc627c65c983b305687cef09"} Nov 02 06:48:51.862406 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:51.862308 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gbxhl"] Nov 02 06:48:51.862643 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:51.862611 5916 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gbxhl" podUID="807aa567-b901-4c00-a31d-198216fe94d8" containerName="istio-proxy" containerID="cri-o://1e11a229f24f64cce7469d4adb32c5b0140d66d494966db2cee5a02ad524b41f" gracePeriod=30 Nov 02 06:48:52.714076 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:52.714045 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-v2lws"] Nov 02 06:48:52.714530 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:52.714246 5916 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-v2lws" podUID="9e0634b1-6e2e-480b-9e74-27cc8525ff6b" containerName="istio-proxy" containerID="cri-o://28d667e56b297c4ba9d0b92fc1e6681613cec18fe69ff61c733c726a468bb970" gracePeriod=30 Nov 02 06:48:52.744355 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:52.744320 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8ecb23-rzfnk" Nov 02 06:48:52.824251 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:52.824013 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb070200-cf13-49f5-94e4-20d0afdfd322-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb070200-cf13-49f5-94e4-20d0afdfd322-maas-api-openshift-ca\") pod \"eb070200-cf13-49f5-94e4-20d0afdfd322\" (UID: \"eb070200-cf13-49f5-94e4-20d0afdfd322\") " Nov 02 06:48:52.824809 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:52.824724 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb070200-cf13-49f5-94e4-20d0afdfd322-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb070200-cf13-49f5-94e4-20d0afdfd322-maas-api-xks-ca\") pod \"eb070200-cf13-49f5-94e4-20d0afdfd322\" (UID: \"eb070200-cf13-49f5-94e4-20d0afdfd322\") " Nov 02 06:48:52.825177 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:52.825104 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb070200-cf13-49f5-94e4-20d0afdfd322-kube-api-access-8dnp5\" (UniqueName: \"kubernetes.io/projected/eb070200-cf13-49f5-94e4-20d0afdfd322-kube-api-access-8dnp5\") pod \"eb070200-cf13-49f5-94e4-20d0afdfd322\" (UID: \"eb070200-cf13-49f5-94e4-20d0afdfd322\") " Nov 02 06:48:52.826647 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:52.826616 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb070200-cf13-49f5-94e4-20d0afdfd322-maas-api-openshift-ca" pod "eb070200-cf13-49f5-94e4-20d0afdfd322" (UID: "eb070200-cf13-49f5-94e4-20d0afdfd322"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:48:52.828690 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:52.828666 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb070200-cf13-49f5-94e4-20d0afdfd322-maas-api-xks-ca" pod "eb070200-cf13-49f5-94e4-20d0afdfd322" (UID: "eb070200-cf13-49f5-94e4-20d0afdfd322"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:48:52.828957 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:52.828873 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb070200-cf13-49f5-94e4-20d0afdfd322-kube-api-access-8dnp5" pod "eb070200-cf13-49f5-94e4-20d0afdfd322" (UID: "eb070200-cf13-49f5-94e4-20d0afdfd322"). InnerVolumeSpecName "kube-api-access-8dnp5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:48:52.885940 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:52.885875 5916 patch_prober.go:29] interesting pod/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gbxhl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 06:48:52.886093 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:52.885966 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gbxhl" podUID="807aa567-b901-4c00-a31d-198216fe94d8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 06:48:52.928189 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:52.928150 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb070200-cf13-49f5-94e4-20d0afdfd322-maas-api-openshift-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:48:52.928189 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:52.928188 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb070200-cf13-49f5-94e4-20d0afdfd322-maas-api-xks-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:48:52.928382 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:52.928203 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8dnp5\" (UniqueName: \"kubernetes.io/projected/eb070200-cf13-49f5-94e4-20d0afdfd322-kube-api-access-8dnp5\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:48:53.574594 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:53.574556 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8ecb23-rzfnk" Nov 02 06:48:53.574594 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:53.574587 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8ecb23-rzfnk" event={"ID":"eb070200-cf13-49f5-94e4-20d0afdfd322","Type":"ContainerDied","Data":"678e89a545a7dbfd431685fcbd35deb180cf588a3f647d59a35850c28c5b3a96"} Nov 02 06:48:53.574802 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:53.574635 5916 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="678e89a545a7dbfd431685fcbd35deb180cf588a3f647d59a35850c28c5b3a96" Nov 02 06:48:53.972134 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:53.972098 5916 patch_prober.go:29] interesting pod/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-c955m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 06:48:53.972521 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:53.972158 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-c955m" podUID="12eeae94-c1bf-4f75-bc8d-2170fc882d8e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 06:48:55.634460 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:55.634427 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a2ba11-5c66894c8c-p472q"] Nov 02 06:48:55.635108 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:55.635091 5916 cpu_manager.go:414] "containerMap: removing container" podUID="eb070200-cf13-49f5-94e4-20d0afdfd322" containerName="revoke-keys" Nov 02 06:48:55.635108 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:55.635108 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb070200-cf13-49f5-94e4-20d0afdfd322" containerName="revoke-keys" Nov 02 06:48:55.635240 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:55.635231 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb070200-cf13-49f5-94e4-20d0afdfd322" containerName="revoke-keys" Nov 02 06:48:55.640006 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:55.639987 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a2ba11-5c66894c8c-p472q" Nov 02 06:48:55.775955 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:55.767472 5916 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-a2ba11\"" Nov 02 06:48:55.775955 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:55.768981 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a2ba11-5c66894c8c-p472q"] Nov 02 06:48:55.776464 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:55.776429 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-a2ba11-dockercfg-r2bnj\"" Nov 02 06:48:55.871683 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:55.864683 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1936eac5-6027-48a9-86ff-12ad457b94d9-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-a2ba11-5c66894c8c-p472q\" (UID: \"1936eac5-6027-48a9-86ff-12ad457b94d9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a2ba11-5c66894c8c-p472q" Nov 02 06:48:55.872081 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:55.872036 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4czx\" (UniqueName: \"kubernetes.io/projected/1936eac5-6027-48a9-86ff-12ad457b94d9-kube-api-access-r4czx\") pod \"payload-pre-processing-e2e-inf-a-a2ba11-5c66894c8c-p472q\" (UID: \"1936eac5-6027-48a9-86ff-12ad457b94d9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a2ba11-5c66894c8c-p472q" Nov 02 06:48:55.919339 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:55.911226 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a2ba11-58967fcd4f-qpggq"] Nov 02 06:48:55.929060 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:55.929030 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a2ba11-58967fcd4f-qpggq"] Nov 02 06:48:55.929266 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:55.929168 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a2ba11-58967fcd4f-qpggq" Nov 02 06:48:55.973256 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:55.973213 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1936eac5-6027-48a9-86ff-12ad457b94d9-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-a2ba11-5c66894c8c-p472q\" (UID: \"1936eac5-6027-48a9-86ff-12ad457b94d9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a2ba11-5c66894c8c-p472q" Nov 02 06:48:55.973398 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:55.973318 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9l8m\" (UniqueName: \"kubernetes.io/projected/610f1b35-d018-4931-815c-98bfc24ebfc5-kube-api-access-g9l8m\") pod \"payload-processing-e2e-inf-a-a2ba11-58967fcd4f-qpggq\" (UID: \"610f1b35-d018-4931-815c-98bfc24ebfc5\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a2ba11-58967fcd4f-qpggq" Nov 02 06:48:55.973398 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:55.973354 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4czx\" (UniqueName: \"kubernetes.io/projected/1936eac5-6027-48a9-86ff-12ad457b94d9-kube-api-access-r4czx\") pod \"payload-pre-processing-e2e-inf-a-a2ba11-5c66894c8c-p472q\" (UID: \"1936eac5-6027-48a9-86ff-12ad457b94d9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a2ba11-5c66894c8c-p472q" Nov 02 06:48:55.973542 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:55.973514 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/610f1b35-d018-4931-815c-98bfc24ebfc5-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-a2ba11-58967fcd4f-qpggq\" (UID: \"610f1b35-d018-4931-815c-98bfc24ebfc5\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a2ba11-58967fcd4f-qpggq" Nov 02 06:48:55.973975 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:55.973952 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1936eac5-6027-48a9-86ff-12ad457b94d9-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-a2ba11-5c66894c8c-p472q\" (UID: \"1936eac5-6027-48a9-86ff-12ad457b94d9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a2ba11-5c66894c8c-p472q" Nov 02 06:48:56.032341 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:56.032312 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4czx\" (UniqueName: \"kubernetes.io/projected/1936eac5-6027-48a9-86ff-12ad457b94d9-kube-api-access-r4czx\") pod \"payload-pre-processing-e2e-inf-a-a2ba11-5c66894c8c-p472q\" (UID: \"1936eac5-6027-48a9-86ff-12ad457b94d9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a2ba11-5c66894c8c-p472q" Nov 02 06:48:56.074876 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:56.074840 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9l8m\" (UniqueName: \"kubernetes.io/projected/610f1b35-d018-4931-815c-98bfc24ebfc5-kube-api-access-g9l8m\") pod \"payload-processing-e2e-inf-a-a2ba11-58967fcd4f-qpggq\" (UID: \"610f1b35-d018-4931-815c-98bfc24ebfc5\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a2ba11-58967fcd4f-qpggq" Nov 02 06:48:56.075037 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:56.075016 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/610f1b35-d018-4931-815c-98bfc24ebfc5-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-a2ba11-58967fcd4f-qpggq\" (UID: \"610f1b35-d018-4931-815c-98bfc24ebfc5\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a2ba11-58967fcd4f-qpggq" Nov 02 06:48:56.075699 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:56.075673 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/610f1b35-d018-4931-815c-98bfc24ebfc5-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-a2ba11-58967fcd4f-qpggq\" (UID: \"610f1b35-d018-4931-815c-98bfc24ebfc5\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a2ba11-58967fcd4f-qpggq" Nov 02 06:48:56.132986 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:56.132953 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9l8m\" (UniqueName: \"kubernetes.io/projected/610f1b35-d018-4931-815c-98bfc24ebfc5-kube-api-access-g9l8m\") pod \"payload-processing-e2e-inf-a-a2ba11-58967fcd4f-qpggq\" (UID: \"610f1b35-d018-4931-815c-98bfc24ebfc5\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a2ba11-58967fcd4f-qpggq" Nov 02 06:48:56.245372 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:56.245341 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a2ba11-58967fcd4f-qpggq" Nov 02 06:48:56.254078 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:56.254050 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a2ba11-5c66894c8c-p472q" Nov 02 06:48:56.494018 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:56.493988 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-c955m" Nov 02 06:48:56.497785 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:56.497760 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-w4bhs" Nov 02 06:48:56.583498 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:56.583463 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb1d132e-7237-4c50-b70b-8f97862d20c3-istio-token\" (UniqueName: \"kubernetes.io/projected/eb1d132e-7237-4c50-b70b-8f97862d20c3-istio-token\") pod \"eb1d132e-7237-4c50-b70b-8f97862d20c3\" (UID: \"eb1d132e-7237-4c50-b70b-8f97862d20c3\") " Nov 02 06:48:56.583697 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:56.583516 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb1d132e-7237-4c50-b70b-8f97862d20c3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb1d132e-7237-4c50-b70b-8f97862d20c3-workload-certs\") pod \"eb1d132e-7237-4c50-b70b-8f97862d20c3\" (UID: \"eb1d132e-7237-4c50-b70b-8f97862d20c3\") " Nov 02 06:48:56.583697 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:56.583551 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb1d132e-7237-4c50-b70b-8f97862d20c3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb1d132e-7237-4c50-b70b-8f97862d20c3-credential-socket\") pod \"eb1d132e-7237-4c50-b70b-8f97862d20c3\" (UID: \"eb1d132e-7237-4c50-b70b-8f97862d20c3\") " Nov 02 06:48:56.583697 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:56.583577 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12eeae94-c1bf-4f75-bc8d-2170fc882d8e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12eeae94-c1bf-4f75-bc8d-2170fc882d8e-istio-data\") pod \"12eeae94-c1bf-4f75-bc8d-2170fc882d8e\" (UID: \"12eeae94-c1bf-4f75-bc8d-2170fc882d8e\") " Nov 02 06:48:56.583697 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:56.583615 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12eeae94-c1bf-4f75-bc8d-2170fc882d8e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12eeae94-c1bf-4f75-bc8d-2170fc882d8e-workload-socket\") pod \"12eeae94-c1bf-4f75-bc8d-2170fc882d8e\" (UID: \"12eeae94-c1bf-4f75-bc8d-2170fc882d8e\") " Nov 02 06:48:56.583697 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:56.583643 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12eeae94-c1bf-4f75-bc8d-2170fc882d8e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12eeae94-c1bf-4f75-bc8d-2170fc882d8e-istio-envoy\") pod \"12eeae94-c1bf-4f75-bc8d-2170fc882d8e\" (UID: \"12eeae94-c1bf-4f75-bc8d-2170fc882d8e\") " Nov 02 06:48:56.583697 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:56.583673 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/12eeae94-c1bf-4f75-bc8d-2170fc882d8e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12eeae94-c1bf-4f75-bc8d-2170fc882d8e-istio-podinfo\") pod \"12eeae94-c1bf-4f75-bc8d-2170fc882d8e\" (UID: \"12eeae94-c1bf-4f75-bc8d-2170fc882d8e\") " Nov 02 06:48:56.584028 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:56.583766 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb1d132e-7237-4c50-b70b-8f97862d20c3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb1d132e-7237-4c50-b70b-8f97862d20c3-workload-socket\") pod \"eb1d132e-7237-4c50-b70b-8f97862d20c3\" (UID: \"eb1d132e-7237-4c50-b70b-8f97862d20c3\") " Nov 02 06:48:56.584028 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:56.583822 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12eeae94-c1bf-4f75-bc8d-2170fc882d8e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12eeae94-c1bf-4f75-bc8d-2170fc882d8e-credential-socket\") pod \"12eeae94-c1bf-4f75-bc8d-2170fc882d8e\" (UID: \"12eeae94-c1bf-4f75-bc8d-2170fc882d8e\") " Nov 02 06:48:56.584028 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:56.583819 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12eeae94-c1bf-4f75-bc8d-2170fc882d8e-workload-socket" pod "12eeae94-c1bf-4f75-bc8d-2170fc882d8e" (UID: "12eeae94-c1bf-4f75-bc8d-2170fc882d8e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:48:56.584028 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:56.583859 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb1d132e-7237-4c50-b70b-8f97862d20c3-workload-certs" pod "eb1d132e-7237-4c50-b70b-8f97862d20c3" (UID: "eb1d132e-7237-4c50-b70b-8f97862d20c3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:48:56.584028 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:56.583864 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb1d132e-7237-4c50-b70b-8f97862d20c3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb1d132e-7237-4c50-b70b-8f97862d20c3-istio-data\") pod \"eb1d132e-7237-4c50-b70b-8f97862d20c3\" (UID: \"eb1d132e-7237-4c50-b70b-8f97862d20c3\") " Nov 02 06:48:56.584028 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:56.583948 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12eeae94-c1bf-4f75-bc8d-2170fc882d8e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12eeae94-c1bf-4f75-bc8d-2170fc882d8e-workload-certs\") pod \"12eeae94-c1bf-4f75-bc8d-2170fc882d8e\" (UID: \"12eeae94-c1bf-4f75-bc8d-2170fc882d8e\") " Nov 02 06:48:56.584028 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:56.583978 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12eeae94-c1bf-4f75-bc8d-2170fc882d8e-istio-token\" (UniqueName: \"kubernetes.io/projected/12eeae94-c1bf-4f75-bc8d-2170fc882d8e-istio-token\") pod \"12eeae94-c1bf-4f75-bc8d-2170fc882d8e\" (UID: \"12eeae94-c1bf-4f75-bc8d-2170fc882d8e\") " Nov 02 06:48:56.584028 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:56.584012 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12eeae94-c1bf-4f75-bc8d-2170fc882d8e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12eeae94-c1bf-4f75-bc8d-2170fc882d8e-istiod-ca-cert\") pod \"12eeae94-c1bf-4f75-bc8d-2170fc882d8e\" (UID: \"12eeae94-c1bf-4f75-bc8d-2170fc882d8e\") " Nov 02 06:48:56.584437 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:56.584045 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12eeae94-c1bf-4f75-bc8d-2170fc882d8e-kube-api-access-9tkr7\" (UniqueName: \"kubernetes.io/projected/12eeae94-c1bf-4f75-bc8d-2170fc882d8e-kube-api-access-9tkr7\") pod \"12eeae94-c1bf-4f75-bc8d-2170fc882d8e\" (UID: \"12eeae94-c1bf-4f75-bc8d-2170fc882d8e\") " Nov 02 06:48:56.584437 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:56.584184 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/eb1d132e-7237-4c50-b70b-8f97862d20c3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb1d132e-7237-4c50-b70b-8f97862d20c3-istio-podinfo\") pod \"eb1d132e-7237-4c50-b70b-8f97862d20c3\" (UID: \"eb1d132e-7237-4c50-b70b-8f97862d20c3\") " Nov 02 06:48:56.584437 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:56.584222 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb1d132e-7237-4c50-b70b-8f97862d20c3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb1d132e-7237-4c50-b70b-8f97862d20c3-istiod-ca-cert\") pod \"eb1d132e-7237-4c50-b70b-8f97862d20c3\" (UID: \"eb1d132e-7237-4c50-b70b-8f97862d20c3\") " Nov 02 06:48:56.584437 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:56.584254 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb1d132e-7237-4c50-b70b-8f97862d20c3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb1d132e-7237-4c50-b70b-8f97862d20c3-istio-envoy\") pod \"eb1d132e-7237-4c50-b70b-8f97862d20c3\" (UID: \"eb1d132e-7237-4c50-b70b-8f97862d20c3\") " Nov 02 06:48:56.584437 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:56.584288 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb1d132e-7237-4c50-b70b-8f97862d20c3-kube-api-access-pqplr\" (UniqueName: \"kubernetes.io/projected/eb1d132e-7237-4c50-b70b-8f97862d20c3-kube-api-access-pqplr\") pod \"eb1d132e-7237-4c50-b70b-8f97862d20c3\" (UID: \"eb1d132e-7237-4c50-b70b-8f97862d20c3\") " Nov 02 06:48:56.584437 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:56.583865 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12eeae94-c1bf-4f75-bc8d-2170fc882d8e-istio-data" pod "12eeae94-c1bf-4f75-bc8d-2170fc882d8e" (UID: "12eeae94-c1bf-4f75-bc8d-2170fc882d8e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:48:56.584437 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:56.584353 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12eeae94-c1bf-4f75-bc8d-2170fc882d8e-workload-certs" pod "12eeae94-c1bf-4f75-bc8d-2170fc882d8e" (UID: "12eeae94-c1bf-4f75-bc8d-2170fc882d8e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:48:56.584437 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:56.583900 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb1d132e-7237-4c50-b70b-8f97862d20c3-credential-socket" pod "eb1d132e-7237-4c50-b70b-8f97862d20c3" (UID: "eb1d132e-7237-4c50-b70b-8f97862d20c3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:48:56.584437 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:56.584138 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb1d132e-7237-4c50-b70b-8f97862d20c3-istio-data" pod "eb1d132e-7237-4c50-b70b-8f97862d20c3" (UID: "eb1d132e-7237-4c50-b70b-8f97862d20c3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:48:56.584437 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:56.584151 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb1d132e-7237-4c50-b70b-8f97862d20c3-workload-socket" pod "eb1d132e-7237-4c50-b70b-8f97862d20c3" (UID: "eb1d132e-7237-4c50-b70b-8f97862d20c3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:48:56.584437 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:56.584340 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12eeae94-c1bf-4f75-bc8d-2170fc882d8e-credential-socket" pod "12eeae94-c1bf-4f75-bc8d-2170fc882d8e" (UID: "12eeae94-c1bf-4f75-bc8d-2170fc882d8e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:48:56.585029 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:56.584685 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb1d132e-7237-4c50-b70b-8f97862d20c3-istiod-ca-cert" pod "eb1d132e-7237-4c50-b70b-8f97862d20c3" (UID: "eb1d132e-7237-4c50-b70b-8f97862d20c3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:48:56.585398 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:56.585370 5916 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb1d132e-7237-4c50-b70b-8f97862d20c3-workload-certs\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:48:56.585511 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:56.585400 5916 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb1d132e-7237-4c50-b70b-8f97862d20c3-credential-socket\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:48:56.585511 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:56.585416 5916 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12eeae94-c1bf-4f75-bc8d-2170fc882d8e-istio-data\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:48:56.585511 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:56.585430 5916 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12eeae94-c1bf-4f75-bc8d-2170fc882d8e-workload-socket\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:48:56.585511 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:56.585444 5916 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb1d132e-7237-4c50-b70b-8f97862d20c3-workload-socket\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:48:56.585511 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:56.585459 5916 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12eeae94-c1bf-4f75-bc8d-2170fc882d8e-credential-socket\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:48:56.585511 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:56.585474 5916 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb1d132e-7237-4c50-b70b-8f97862d20c3-istio-data\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:48:56.585511 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:56.585486 5916 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12eeae94-c1bf-4f75-bc8d-2170fc882d8e-workload-certs\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:48:56.585511 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:56.585502 5916 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb1d132e-7237-4c50-b70b-8f97862d20c3-istiod-ca-cert\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:48:56.586115 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:56.586008 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12eeae94-c1bf-4f75-bc8d-2170fc882d8e-istiod-ca-cert" pod "12eeae94-c1bf-4f75-bc8d-2170fc882d8e" (UID: "12eeae94-c1bf-4f75-bc8d-2170fc882d8e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:48:56.586292 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:56.586264 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12eeae94-c1bf-4f75-bc8d-2170fc882d8e-istio-envoy" pod "12eeae94-c1bf-4f75-bc8d-2170fc882d8e" (UID: "12eeae94-c1bf-4f75-bc8d-2170fc882d8e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:48:56.586292 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:56.586288 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb1d132e-7237-4c50-b70b-8f97862d20c3-istio-token" pod "eb1d132e-7237-4c50-b70b-8f97862d20c3" (UID: "eb1d132e-7237-4c50-b70b-8f97862d20c3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:48:56.586898 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:56.586871 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb1d132e-7237-4c50-b70b-8f97862d20c3-kube-api-access-pqplr" pod "eb1d132e-7237-4c50-b70b-8f97862d20c3" (UID: "eb1d132e-7237-4c50-b70b-8f97862d20c3"). InnerVolumeSpecName "kube-api-access-pqplr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:48:56.586898 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:56.586896 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/12eeae94-c1bf-4f75-bc8d-2170fc882d8e-istio-podinfo" pod "12eeae94-c1bf-4f75-bc8d-2170fc882d8e" (UID: "12eeae94-c1bf-4f75-bc8d-2170fc882d8e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 06:48:56.587159 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:56.587138 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eb1d132e-7237-4c50-b70b-8f97862d20c3-istio-podinfo" pod "eb1d132e-7237-4c50-b70b-8f97862d20c3" (UID: "eb1d132e-7237-4c50-b70b-8f97862d20c3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 06:48:56.587674 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:56.587650 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb1d132e-7237-4c50-b70b-8f97862d20c3-istio-envoy" pod "eb1d132e-7237-4c50-b70b-8f97862d20c3" (UID: "eb1d132e-7237-4c50-b70b-8f97862d20c3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:48:56.588050 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:56.588025 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12eeae94-c1bf-4f75-bc8d-2170fc882d8e-istio-token" pod "12eeae94-c1bf-4f75-bc8d-2170fc882d8e" (UID: "12eeae94-c1bf-4f75-bc8d-2170fc882d8e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:48:56.588133 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:56.588105 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12eeae94-c1bf-4f75-bc8d-2170fc882d8e-kube-api-access-9tkr7" pod "12eeae94-c1bf-4f75-bc8d-2170fc882d8e" (UID: "12eeae94-c1bf-4f75-bc8d-2170fc882d8e"). InnerVolumeSpecName "kube-api-access-9tkr7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:48:56.601267 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:56.601236 5916 generic.go:356] "Generic (PLEG): container finished" podID="12eeae94-c1bf-4f75-bc8d-2170fc882d8e" containerID="f5a5d7460a73766554bf55fd4276489711bbea9edda84644c823835082903343" exitCode=0 Nov 02 06:48:56.601520 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:56.601498 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-c955m" event={"ID":"12eeae94-c1bf-4f75-bc8d-2170fc882d8e","Type":"ContainerDied","Data":"f5a5d7460a73766554bf55fd4276489711bbea9edda84644c823835082903343"} Nov 02 06:48:56.601583 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:56.601556 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-c955m" Nov 02 06:48:56.601583 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:56.601567 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-c955m" event={"ID":"12eeae94-c1bf-4f75-bc8d-2170fc882d8e","Type":"ContainerDied","Data":"76504f59f711f2dac6a5d3d28a88dc430029508e43a65936cac525f4e961136d"} Nov 02 06:48:56.601688 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:56.601586 5916 scope.go:122] "RemoveContainer" containerID="f5a5d7460a73766554bf55fd4276489711bbea9edda84644c823835082903343" Nov 02 06:48:56.603023 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:56.603000 5916 generic.go:356] "Generic (PLEG): container finished" podID="eb1d132e-7237-4c50-b70b-8f97862d20c3" containerID="43c8d7a3bf910eefad89dc39e2172de86c4222c34f2651eca5f8f11f9ab43cf6" exitCode=0 Nov 02 06:48:56.603139 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:56.603086 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-w4bhs" event={"ID":"eb1d132e-7237-4c50-b70b-8f97862d20c3","Type":"ContainerDied","Data":"43c8d7a3bf910eefad89dc39e2172de86c4222c34f2651eca5f8f11f9ab43cf6"} Nov 02 06:48:56.603139 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:56.603126 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-w4bhs" event={"ID":"eb1d132e-7237-4c50-b70b-8f97862d20c3","Type":"ContainerDied","Data":"d59ee75a833e6f8afb3dd1db766f9e47e04c9b59bd5ffb02c5fe9f9d88ce111d"} Nov 02 06:48:56.603263 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:56.603190 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-w4bhs" Nov 02 06:48:56.620221 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:56.620103 5916 scope.go:122] "RemoveContainer" containerID="f5a5d7460a73766554bf55fd4276489711bbea9edda84644c823835082903343" Nov 02 06:48:56.620501 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:48:56.620461 5916 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5a5d7460a73766554bf55fd4276489711bbea9edda84644c823835082903343\": container with ID starting with f5a5d7460a73766554bf55fd4276489711bbea9edda84644c823835082903343 not found: ID does not exist" containerID="f5a5d7460a73766554bf55fd4276489711bbea9edda84644c823835082903343" Nov 02 06:48:56.620631 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:56.620501 5916 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5a5d7460a73766554bf55fd4276489711bbea9edda84644c823835082903343"} err="failed to get container status \"f5a5d7460a73766554bf55fd4276489711bbea9edda84644c823835082903343\": rpc error: code = NotFound desc = could not find container \"f5a5d7460a73766554bf55fd4276489711bbea9edda84644c823835082903343\": container with ID starting with f5a5d7460a73766554bf55fd4276489711bbea9edda84644c823835082903343 not found: ID does not exist" Nov 02 06:48:56.620631 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:56.620527 5916 scope.go:122] "RemoveContainer" containerID="43c8d7a3bf910eefad89dc39e2172de86c4222c34f2651eca5f8f11f9ab43cf6" Nov 02 06:48:56.639147 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:56.639116 5916 scope.go:122] "RemoveContainer" containerID="43c8d7a3bf910eefad89dc39e2172de86c4222c34f2651eca5f8f11f9ab43cf6" Nov 02 06:48:56.639581 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:48:56.639515 5916 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43c8d7a3bf910eefad89dc39e2172de86c4222c34f2651eca5f8f11f9ab43cf6\": container with ID starting with 43c8d7a3bf910eefad89dc39e2172de86c4222c34f2651eca5f8f11f9ab43cf6 not found: ID does not exist" containerID="43c8d7a3bf910eefad89dc39e2172de86c4222c34f2651eca5f8f11f9ab43cf6" Nov 02 06:48:56.639581 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:56.639555 5916 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43c8d7a3bf910eefad89dc39e2172de86c4222c34f2651eca5f8f11f9ab43cf6"} err="failed to get container status \"43c8d7a3bf910eefad89dc39e2172de86c4222c34f2651eca5f8f11f9ab43cf6\": rpc error: code = NotFound desc = could not find container \"43c8d7a3bf910eefad89dc39e2172de86c4222c34f2651eca5f8f11f9ab43cf6\": container with ID starting with 43c8d7a3bf910eefad89dc39e2172de86c4222c34f2651eca5f8f11f9ab43cf6 not found: ID does not exist" Nov 02 06:48:56.645710 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:48:56.645679 5916 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb1d132e_7237_4c50_b70b_8f97862d20c3.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12eeae94_c1bf_4f75_bc8d_2170fc882d8e.slice\": RecentStats: unable to find data in memory cache]" Nov 02 06:48:56.647079 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:56.647059 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a2ba11-58967fcd4f-qpggq"] Nov 02 06:48:56.683654 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:48:56.683625 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod610f1b35_d018_4931_815c_98bfc24ebfc5.slice/crio-ba9f00a31b5bf9ecbddc49ec78ddd8994a281b85795ee2bcf0a369f057fab194 WatchSource:0}: Error finding container ba9f00a31b5bf9ecbddc49ec78ddd8994a281b85795ee2bcf0a369f057fab194: Status 404 returned error can't find the container with id ba9f00a31b5bf9ecbddc49ec78ddd8994a281b85795ee2bcf0a369f057fab194 Nov 02 06:48:56.686270 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:56.686244 5916 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12eeae94-c1bf-4f75-bc8d-2170fc882d8e-istio-token\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:48:56.686413 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:56.686274 5916 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12eeae94-c1bf-4f75-bc8d-2170fc882d8e-istiod-ca-cert\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:48:56.686413 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:56.686297 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9tkr7\" (UniqueName: \"kubernetes.io/projected/12eeae94-c1bf-4f75-bc8d-2170fc882d8e-kube-api-access-9tkr7\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:48:56.686413 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:56.686312 5916 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb1d132e-7237-4c50-b70b-8f97862d20c3-istio-podinfo\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:48:56.686413 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:56.686326 5916 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb1d132e-7237-4c50-b70b-8f97862d20c3-istio-envoy\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:48:56.686413 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:56.686344 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqplr\" (UniqueName: \"kubernetes.io/projected/eb1d132e-7237-4c50-b70b-8f97862d20c3-kube-api-access-pqplr\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:48:56.686413 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:56.686369 5916 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb1d132e-7237-4c50-b70b-8f97862d20c3-istio-token\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:48:56.686413 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:56.686385 5916 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12eeae94-c1bf-4f75-bc8d-2170fc882d8e-istio-envoy\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:48:56.686413 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:56.686398 5916 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12eeae94-c1bf-4f75-bc8d-2170fc882d8e-istio-podinfo\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:48:56.694515 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:56.694493 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a2ba11-5c66894c8c-p472q"] Nov 02 06:48:56.697135 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:48:56.697111 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1936eac5_6027_48a9_86ff_12ad457b94d9.slice/crio-160ef2ef732422f6c10567ec93ae13def8a623c4b67a016a9fbaf55ac31f2b0b WatchSource:0}: Error finding container 160ef2ef732422f6c10567ec93ae13def8a623c4b67a016a9fbaf55ac31f2b0b: Status 404 returned error can't find the container with id 160ef2ef732422f6c10567ec93ae13def8a623c4b67a016a9fbaf55ac31f2b0b Nov 02 06:48:56.703671 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:56.703580 5916 patch_prober.go:29] interesting pod/e2e-worker-w6-6ff85e-openshift-default-698585659c-v2lws container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 06:48:56.703671 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:56.703630 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-v2lws" podUID="9e0634b1-6e2e-480b-9e74-27cc8525ff6b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 06:48:56.909564 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:56.909534 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-c955m"] Nov 02 06:48:56.915393 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:56.915367 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-c955m"] Nov 02 06:48:57.023113 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:57.023021 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-w4bhs"] Nov 02 06:48:57.029987 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:57.027930 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-w4bhs"] Nov 02 06:48:57.220373 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:57.220339 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gbxhl" Nov 02 06:48:57.345974 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:57.344892 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/807aa567-b901-4c00-a31d-198216fe94d8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/807aa567-b901-4c00-a31d-198216fe94d8-istio-podinfo\") pod \"807aa567-b901-4c00-a31d-198216fe94d8\" (UID: \"807aa567-b901-4c00-a31d-198216fe94d8\") " Nov 02 06:48:57.345974 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:57.344960 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/807aa567-b901-4c00-a31d-198216fe94d8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/807aa567-b901-4c00-a31d-198216fe94d8-istio-envoy\") pod \"807aa567-b901-4c00-a31d-198216fe94d8\" (UID: \"807aa567-b901-4c00-a31d-198216fe94d8\") " Nov 02 06:48:57.345974 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:57.344996 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/807aa567-b901-4c00-a31d-198216fe94d8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/807aa567-b901-4c00-a31d-198216fe94d8-workload-certs\") pod \"807aa567-b901-4c00-a31d-198216fe94d8\" (UID: \"807aa567-b901-4c00-a31d-198216fe94d8\") " Nov 02 06:48:57.345974 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:57.345108 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/807aa567-b901-4c00-a31d-198216fe94d8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/807aa567-b901-4c00-a31d-198216fe94d8-istio-data\") pod \"807aa567-b901-4c00-a31d-198216fe94d8\" (UID: \"807aa567-b901-4c00-a31d-198216fe94d8\") " Nov 02 06:48:57.347703 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:57.347641 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/807aa567-b901-4c00-a31d-198216fe94d8-istio-podinfo" pod "807aa567-b901-4c00-a31d-198216fe94d8" (UID: "807aa567-b901-4c00-a31d-198216fe94d8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 06:48:57.348226 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:57.348067 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/807aa567-b901-4c00-a31d-198216fe94d8-istio-data" pod "807aa567-b901-4c00-a31d-198216fe94d8" (UID: "807aa567-b901-4c00-a31d-198216fe94d8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:48:57.356794 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:57.356764 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/807aa567-b901-4c00-a31d-198216fe94d8-workload-certs" pod "807aa567-b901-4c00-a31d-198216fe94d8" (UID: "807aa567-b901-4c00-a31d-198216fe94d8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:48:57.356960 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:57.356895 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/807aa567-b901-4c00-a31d-198216fe94d8-kube-api-access-s972w\" (UniqueName: \"kubernetes.io/projected/807aa567-b901-4c00-a31d-198216fe94d8-kube-api-access-s972w\") pod \"807aa567-b901-4c00-a31d-198216fe94d8\" (UID: \"807aa567-b901-4c00-a31d-198216fe94d8\") " Nov 02 06:48:57.357024 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:57.356955 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/807aa567-b901-4c00-a31d-198216fe94d8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/807aa567-b901-4c00-a31d-198216fe94d8-workload-socket\") pod \"807aa567-b901-4c00-a31d-198216fe94d8\" (UID: \"807aa567-b901-4c00-a31d-198216fe94d8\") " Nov 02 06:48:57.357082 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:57.357037 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/807aa567-b901-4c00-a31d-198216fe94d8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/807aa567-b901-4c00-a31d-198216fe94d8-credential-socket\") pod \"807aa567-b901-4c00-a31d-198216fe94d8\" (UID: \"807aa567-b901-4c00-a31d-198216fe94d8\") " Nov 02 06:48:57.357139 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:57.357098 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/807aa567-b901-4c00-a31d-198216fe94d8-istio-token\" (UniqueName: \"kubernetes.io/projected/807aa567-b901-4c00-a31d-198216fe94d8-istio-token\") pod \"807aa567-b901-4c00-a31d-198216fe94d8\" (UID: \"807aa567-b901-4c00-a31d-198216fe94d8\") " Nov 02 06:48:57.357139 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:57.357130 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/807aa567-b901-4c00-a31d-198216fe94d8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/807aa567-b901-4c00-a31d-198216fe94d8-istiod-ca-cert\") pod \"807aa567-b901-4c00-a31d-198216fe94d8\" (UID: \"807aa567-b901-4c00-a31d-198216fe94d8\") " Nov 02 06:48:57.357982 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:57.357957 5916 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/807aa567-b901-4c00-a31d-198216fe94d8-istio-podinfo\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:48:57.357982 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:57.357988 5916 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/807aa567-b901-4c00-a31d-198216fe94d8-workload-certs\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:48:57.358133 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:57.358003 5916 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/807aa567-b901-4c00-a31d-198216fe94d8-istio-data\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:48:57.358327 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:57.358306 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/807aa567-b901-4c00-a31d-198216fe94d8-credential-socket" pod "807aa567-b901-4c00-a31d-198216fe94d8" (UID: "807aa567-b901-4c00-a31d-198216fe94d8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:48:57.359367 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:57.359151 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/807aa567-b901-4c00-a31d-198216fe94d8-istiod-ca-cert" pod "807aa567-b901-4c00-a31d-198216fe94d8" (UID: "807aa567-b901-4c00-a31d-198216fe94d8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:48:57.359367 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:57.359211 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/807aa567-b901-4c00-a31d-198216fe94d8-workload-socket" pod "807aa567-b901-4c00-a31d-198216fe94d8" (UID: "807aa567-b901-4c00-a31d-198216fe94d8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:48:57.381063 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:57.379470 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/807aa567-b901-4c00-a31d-198216fe94d8-istio-envoy" pod "807aa567-b901-4c00-a31d-198216fe94d8" (UID: "807aa567-b901-4c00-a31d-198216fe94d8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:48:57.381063 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:57.380253 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/807aa567-b901-4c00-a31d-198216fe94d8-istio-token" pod "807aa567-b901-4c00-a31d-198216fe94d8" (UID: "807aa567-b901-4c00-a31d-198216fe94d8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:48:57.381506 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:57.381482 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/807aa567-b901-4c00-a31d-198216fe94d8-kube-api-access-s972w" pod "807aa567-b901-4c00-a31d-198216fe94d8" (UID: "807aa567-b901-4c00-a31d-198216fe94d8"). InnerVolumeSpecName "kube-api-access-s972w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:48:57.459752 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:57.459713 5916 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/807aa567-b901-4c00-a31d-198216fe94d8-istio-envoy\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:48:57.459752 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:57.459755 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s972w\" (UniqueName: \"kubernetes.io/projected/807aa567-b901-4c00-a31d-198216fe94d8-kube-api-access-s972w\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:48:57.459976 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:57.459771 5916 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/807aa567-b901-4c00-a31d-198216fe94d8-workload-socket\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:48:57.459976 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:57.459785 5916 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/807aa567-b901-4c00-a31d-198216fe94d8-credential-socket\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:48:57.459976 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:57.459799 5916 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/807aa567-b901-4c00-a31d-198216fe94d8-istio-token\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:48:57.459976 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:57.459813 5916 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/807aa567-b901-4c00-a31d-198216fe94d8-istiod-ca-cert\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:48:57.616039 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:57.615945 5916 generic.go:356] "Generic (PLEG): container finished" podID="807aa567-b901-4c00-a31d-198216fe94d8" containerID="1e11a229f24f64cce7469d4adb32c5b0140d66d494966db2cee5a02ad524b41f" exitCode=0 Nov 02 06:48:57.616292 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:57.615981 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gbxhl" Nov 02 06:48:57.616774 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:57.616001 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gbxhl" event={"ID":"807aa567-b901-4c00-a31d-198216fe94d8","Type":"ContainerDied","Data":"1e11a229f24f64cce7469d4adb32c5b0140d66d494966db2cee5a02ad524b41f"} Nov 02 06:48:57.616870 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:57.616818 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gbxhl" event={"ID":"807aa567-b901-4c00-a31d-198216fe94d8","Type":"ContainerDied","Data":"dea9ea43a33cb838faa16bb8567468463743868754bceedd02ce8b667fff0903"} Nov 02 06:48:57.616870 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:57.616853 5916 scope.go:122] "RemoveContainer" containerID="1e11a229f24f64cce7469d4adb32c5b0140d66d494966db2cee5a02ad524b41f" Nov 02 06:48:57.622875 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:57.621938 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a2ba11-58967fcd4f-qpggq" event={"ID":"610f1b35-d018-4931-815c-98bfc24ebfc5","Type":"ContainerStarted","Data":"55dc76fa41f5d3c108cb7aeedd7f5e4862e80c3b1daac12d38b4ccf6e8738080"} Nov 02 06:48:57.622875 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:57.622033 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a2ba11-58967fcd4f-qpggq" event={"ID":"610f1b35-d018-4931-815c-98bfc24ebfc5","Type":"ContainerStarted","Data":"ba9f00a31b5bf9ecbddc49ec78ddd8994a281b85795ee2bcf0a369f057fab194"} Nov 02 06:48:57.623432 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:57.622827 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-a2ba11-58967fcd4f-qpggq" Nov 02 06:48:57.627440 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:57.626479 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a2ba11-5c66894c8c-p472q" event={"ID":"1936eac5-6027-48a9-86ff-12ad457b94d9","Type":"ContainerStarted","Data":"fd52bf8f16f3bd30c336e3412703766a5dcbf0e40f02ae13ddf2c1b564aec501"} Nov 02 06:48:57.627440 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:57.626522 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a2ba11-5c66894c8c-p472q" event={"ID":"1936eac5-6027-48a9-86ff-12ad457b94d9","Type":"ContainerStarted","Data":"160ef2ef732422f6c10567ec93ae13def8a623c4b67a016a9fbaf55ac31f2b0b"} Nov 02 06:48:57.627440 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:57.627233 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a2ba11-5c66894c8c-p472q" Nov 02 06:48:57.632307 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:57.632287 5916 scope.go:122] "RemoveContainer" containerID="1e11a229f24f64cce7469d4adb32c5b0140d66d494966db2cee5a02ad524b41f" Nov 02 06:48:57.632626 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:48:57.632566 5916 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e11a229f24f64cce7469d4adb32c5b0140d66d494966db2cee5a02ad524b41f\": container with ID starting with 1e11a229f24f64cce7469d4adb32c5b0140d66d494966db2cee5a02ad524b41f not found: ID does not exist" containerID="1e11a229f24f64cce7469d4adb32c5b0140d66d494966db2cee5a02ad524b41f" Nov 02 06:48:57.632626 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:57.632600 5916 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e11a229f24f64cce7469d4adb32c5b0140d66d494966db2cee5a02ad524b41f"} err="failed to get container status \"1e11a229f24f64cce7469d4adb32c5b0140d66d494966db2cee5a02ad524b41f\": rpc error: code = NotFound desc = could not find container \"1e11a229f24f64cce7469d4adb32c5b0140d66d494966db2cee5a02ad524b41f\": container with ID starting with 1e11a229f24f64cce7469d4adb32c5b0140d66d494966db2cee5a02ad524b41f not found: ID does not exist" Nov 02 06:48:57.791310 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:57.791280 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gbxhl"] Nov 02 06:48:57.796941 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:57.796884 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-gbxhl"] Nov 02 06:48:57.848944 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:57.848891 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12eeae94-c1bf-4f75-bc8d-2170fc882d8e" path="/var/lib/kubelet/pods/12eeae94-c1bf-4f75-bc8d-2170fc882d8e/volumes" Nov 02 06:48:57.849655 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:57.849630 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="807aa567-b901-4c00-a31d-198216fe94d8" path="/var/lib/kubelet/pods/807aa567-b901-4c00-a31d-198216fe94d8/volumes" Nov 02 06:48:57.850288 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:57.850270 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb1d132e-7237-4c50-b70b-8f97862d20c3" path="/var/lib/kubelet/pods/eb1d132e-7237-4c50-b70b-8f97862d20c3/volumes" Nov 02 06:48:57.914078 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:57.914010 5916 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-a2ba11-58967fcd4f-qpggq" podStartSLOduration=2.913987932 podStartE2EDuration="2.913987932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:48:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:48:57.903396158 +0000 UTC m=+1926.175216553" watchObservedRunningTime="2026-11-02 06:48:57.913987932 +0000 UTC m=+1926.185808330" Nov 02 06:48:58.019722 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:58.019659 5916 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a2ba11-5c66894c8c-p472q" podStartSLOduration=3.019641134 podStartE2EDuration="3.019641134s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:48:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:48:58.017259482 +0000 UTC m=+1926.289079877" watchObservedRunningTime="2026-11-02 06:48:58.019641134 +0000 UTC m=+1926.291461529" Nov 02 06:48:58.030667 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:58.030634 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-v2lws" Nov 02 06:48:58.182429 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:58.182338 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e0634b1-6e2e-480b-9e74-27cc8525ff6b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e0634b1-6e2e-480b-9e74-27cc8525ff6b-workload-certs\") pod \"9e0634b1-6e2e-480b-9e74-27cc8525ff6b\" (UID: \"9e0634b1-6e2e-480b-9e74-27cc8525ff6b\") " Nov 02 06:48:58.182624 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:58.182467 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e0634b1-6e2e-480b-9e74-27cc8525ff6b-kube-api-access-vss9x\" (UniqueName: \"kubernetes.io/projected/9e0634b1-6e2e-480b-9e74-27cc8525ff6b-kube-api-access-vss9x\") pod \"9e0634b1-6e2e-480b-9e74-27cc8525ff6b\" (UID: \"9e0634b1-6e2e-480b-9e74-27cc8525ff6b\") " Nov 02 06:48:58.182624 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:58.182537 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e0634b1-6e2e-480b-9e74-27cc8525ff6b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e0634b1-6e2e-480b-9e74-27cc8525ff6b-istio-envoy\") pod \"9e0634b1-6e2e-480b-9e74-27cc8525ff6b\" (UID: \"9e0634b1-6e2e-480b-9e74-27cc8525ff6b\") " Nov 02 06:48:58.182624 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:58.182602 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e0634b1-6e2e-480b-9e74-27cc8525ff6b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e0634b1-6e2e-480b-9e74-27cc8525ff6b-istiod-ca-cert\") pod \"9e0634b1-6e2e-480b-9e74-27cc8525ff6b\" (UID: \"9e0634b1-6e2e-480b-9e74-27cc8525ff6b\") " Nov 02 06:48:58.182813 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:58.182664 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9e0634b1-6e2e-480b-9e74-27cc8525ff6b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e0634b1-6e2e-480b-9e74-27cc8525ff6b-istio-podinfo\") pod \"9e0634b1-6e2e-480b-9e74-27cc8525ff6b\" (UID: \"9e0634b1-6e2e-480b-9e74-27cc8525ff6b\") " Nov 02 06:48:58.182813 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:58.182790 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e0634b1-6e2e-480b-9e74-27cc8525ff6b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e0634b1-6e2e-480b-9e74-27cc8525ff6b-istio-data\") pod \"9e0634b1-6e2e-480b-9e74-27cc8525ff6b\" (UID: \"9e0634b1-6e2e-480b-9e74-27cc8525ff6b\") " Nov 02 06:48:58.187987 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:58.182824 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e0634b1-6e2e-480b-9e74-27cc8525ff6b-istio-token\" (UniqueName: \"kubernetes.io/projected/9e0634b1-6e2e-480b-9e74-27cc8525ff6b-istio-token\") pod \"9e0634b1-6e2e-480b-9e74-27cc8525ff6b\" (UID: \"9e0634b1-6e2e-480b-9e74-27cc8525ff6b\") " Nov 02 06:48:58.187987 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:58.182888 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e0634b1-6e2e-480b-9e74-27cc8525ff6b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e0634b1-6e2e-480b-9e74-27cc8525ff6b-credential-socket\") pod \"9e0634b1-6e2e-480b-9e74-27cc8525ff6b\" (UID: \"9e0634b1-6e2e-480b-9e74-27cc8525ff6b\") " Nov 02 06:48:58.187987 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:58.183042 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e0634b1-6e2e-480b-9e74-27cc8525ff6b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e0634b1-6e2e-480b-9e74-27cc8525ff6b-workload-socket\") pod \"9e0634b1-6e2e-480b-9e74-27cc8525ff6b\" (UID: \"9e0634b1-6e2e-480b-9e74-27cc8525ff6b\") " Nov 02 06:48:58.187987 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:58.184070 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e0634b1-6e2e-480b-9e74-27cc8525ff6b-workload-socket" pod "9e0634b1-6e2e-480b-9e74-27cc8525ff6b" (UID: "9e0634b1-6e2e-480b-9e74-27cc8525ff6b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:48:58.187987 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:58.184305 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e0634b1-6e2e-480b-9e74-27cc8525ff6b-workload-certs" pod "9e0634b1-6e2e-480b-9e74-27cc8525ff6b" (UID: "9e0634b1-6e2e-480b-9e74-27cc8525ff6b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:48:58.195186 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:58.194830 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e0634b1-6e2e-480b-9e74-27cc8525ff6b-istiod-ca-cert" pod "9e0634b1-6e2e-480b-9e74-27cc8525ff6b" (UID: "9e0634b1-6e2e-480b-9e74-27cc8525ff6b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:48:58.195186 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:58.195111 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e0634b1-6e2e-480b-9e74-27cc8525ff6b-istio-data" pod "9e0634b1-6e2e-480b-9e74-27cc8525ff6b" (UID: "9e0634b1-6e2e-480b-9e74-27cc8525ff6b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:48:58.195860 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:58.195423 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e0634b1-6e2e-480b-9e74-27cc8525ff6b-credential-socket" pod "9e0634b1-6e2e-480b-9e74-27cc8525ff6b" (UID: "9e0634b1-6e2e-480b-9e74-27cc8525ff6b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:48:58.216150 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:58.216114 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e0634b1-6e2e-480b-9e74-27cc8525ff6b-istio-envoy" pod "9e0634b1-6e2e-480b-9e74-27cc8525ff6b" (UID: "9e0634b1-6e2e-480b-9e74-27cc8525ff6b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:48:58.226052 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:58.226020 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9e0634b1-6e2e-480b-9e74-27cc8525ff6b-istio-podinfo" pod "9e0634b1-6e2e-480b-9e74-27cc8525ff6b" (UID: "9e0634b1-6e2e-480b-9e74-27cc8525ff6b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 06:48:58.232255 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:58.232205 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e0634b1-6e2e-480b-9e74-27cc8525ff6b-kube-api-access-vss9x" pod "9e0634b1-6e2e-480b-9e74-27cc8525ff6b" (UID: "9e0634b1-6e2e-480b-9e74-27cc8525ff6b"). InnerVolumeSpecName "kube-api-access-vss9x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:48:58.235117 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:58.235091 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e0634b1-6e2e-480b-9e74-27cc8525ff6b-istio-token" pod "9e0634b1-6e2e-480b-9e74-27cc8525ff6b" (UID: "9e0634b1-6e2e-480b-9e74-27cc8525ff6b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:48:58.285089 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:58.285052 5916 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e0634b1-6e2e-480b-9e74-27cc8525ff6b-credential-socket\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:48:58.285089 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:58.285085 5916 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e0634b1-6e2e-480b-9e74-27cc8525ff6b-workload-socket\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:48:58.285089 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:58.285096 5916 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e0634b1-6e2e-480b-9e74-27cc8525ff6b-workload-certs\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:48:58.285403 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:58.285107 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vss9x\" (UniqueName: \"kubernetes.io/projected/9e0634b1-6e2e-480b-9e74-27cc8525ff6b-kube-api-access-vss9x\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:48:58.285403 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:58.285118 5916 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e0634b1-6e2e-480b-9e74-27cc8525ff6b-istio-envoy\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:48:58.285403 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:58.285126 5916 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e0634b1-6e2e-480b-9e74-27cc8525ff6b-istiod-ca-cert\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:48:58.285403 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:58.285133 5916 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e0634b1-6e2e-480b-9e74-27cc8525ff6b-istio-podinfo\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:48:58.285403 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:58.285144 5916 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e0634b1-6e2e-480b-9e74-27cc8525ff6b-istio-data\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:48:58.285403 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:58.285155 5916 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e0634b1-6e2e-480b-9e74-27cc8525ff6b-istio-token\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:48:58.640106 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:58.640067 5916 generic.go:356] "Generic (PLEG): container finished" podID="9e0634b1-6e2e-480b-9e74-27cc8525ff6b" containerID="28d667e56b297c4ba9d0b92fc1e6681613cec18fe69ff61c733c726a468bb970" exitCode=0 Nov 02 06:48:58.640319 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:58.640126 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-v2lws" Nov 02 06:48:58.640319 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:58.640164 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-v2lws" event={"ID":"9e0634b1-6e2e-480b-9e74-27cc8525ff6b","Type":"ContainerDied","Data":"28d667e56b297c4ba9d0b92fc1e6681613cec18fe69ff61c733c726a468bb970"} Nov 02 06:48:58.640319 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:58.640261 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-v2lws" event={"ID":"9e0634b1-6e2e-480b-9e74-27cc8525ff6b","Type":"ContainerDied","Data":"9816c54ef13dde72b930a74fa054e9b58d73f34506cd6a80808500eeec82415a"} Nov 02 06:48:58.640319 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:58.640292 5916 scope.go:122] "RemoveContainer" containerID="28d667e56b297c4ba9d0b92fc1e6681613cec18fe69ff61c733c726a468bb970" Nov 02 06:48:58.663763 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:58.663733 5916 scope.go:122] "RemoveContainer" containerID="28d667e56b297c4ba9d0b92fc1e6681613cec18fe69ff61c733c726a468bb970" Nov 02 06:48:58.664184 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:48:58.664160 5916 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28d667e56b297c4ba9d0b92fc1e6681613cec18fe69ff61c733c726a468bb970\": container with ID starting with 28d667e56b297c4ba9d0b92fc1e6681613cec18fe69ff61c733c726a468bb970 not found: ID does not exist" containerID="28d667e56b297c4ba9d0b92fc1e6681613cec18fe69ff61c733c726a468bb970" Nov 02 06:48:58.664282 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:58.664191 5916 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28d667e56b297c4ba9d0b92fc1e6681613cec18fe69ff61c733c726a468bb970"} err="failed to get container status \"28d667e56b297c4ba9d0b92fc1e6681613cec18fe69ff61c733c726a468bb970\": rpc error: code = NotFound desc = could not find container \"28d667e56b297c4ba9d0b92fc1e6681613cec18fe69ff61c733c726a468bb970\": container with ID starting with 28d667e56b297c4ba9d0b92fc1e6681613cec18fe69ff61c733c726a468bb970 not found: ID does not exist" Nov 02 06:48:58.807702 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:58.807659 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-v2lws"] Nov 02 06:48:58.813391 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:58.813364 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-v2lws"] Nov 02 06:48:59.846663 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:48:59.846624 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e0634b1-6e2e-480b-9e74-27cc8525ff6b" path="/var/lib/kubelet/pods/9e0634b1-6e2e-480b-9e74-27cc8525ff6b/volumes" Nov 02 06:49:09.266526 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:49:09.266488 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-08bbf9-xmssn"] Nov 02 06:49:09.267421 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:49:09.267399 5916 cpu_manager.go:414] "containerMap: removing container" podUID="12eeae94-c1bf-4f75-bc8d-2170fc882d8e" containerName="istio-proxy" Nov 02 06:49:09.267463 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:49:09.267423 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12eeae94-c1bf-4f75-bc8d-2170fc882d8e" containerName="istio-proxy" Nov 02 06:49:09.267463 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:49:09.267443 5916 cpu_manager.go:414] "containerMap: removing container" podUID="eb1d132e-7237-4c50-b70b-8f97862d20c3" containerName="istio-proxy" Nov 02 06:49:09.267463 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:49:09.267450 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb1d132e-7237-4c50-b70b-8f97862d20c3" containerName="istio-proxy" Nov 02 06:49:09.267608 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:49:09.267510 5916 cpu_manager.go:414] "containerMap: removing container" podUID="9e0634b1-6e2e-480b-9e74-27cc8525ff6b" containerName="istio-proxy" Nov 02 06:49:09.267608 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:49:09.267520 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e0634b1-6e2e-480b-9e74-27cc8525ff6b" containerName="istio-proxy" Nov 02 06:49:09.267608 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:49:09.267551 5916 cpu_manager.go:414] "containerMap: removing container" podUID="807aa567-b901-4c00-a31d-198216fe94d8" containerName="istio-proxy" Nov 02 06:49:09.267608 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:49:09.267560 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="807aa567-b901-4c00-a31d-198216fe94d8" containerName="istio-proxy" Nov 02 06:49:09.267730 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:49:09.267722 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="12eeae94-c1bf-4f75-bc8d-2170fc882d8e" containerName="istio-proxy" Nov 02 06:49:09.267763 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:49:09.267737 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb1d132e-7237-4c50-b70b-8f97862d20c3" containerName="istio-proxy" Nov 02 06:49:09.267763 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:49:09.267755 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e0634b1-6e2e-480b-9e74-27cc8525ff6b" containerName="istio-proxy" Nov 02 06:49:09.267823 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:49:09.267770 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="807aa567-b901-4c00-a31d-198216fe94d8" containerName="istio-proxy" Nov 02 06:49:09.273034 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:49:09.273014 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-08bbf9-xmssn" Nov 02 06:49:09.325600 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:49:09.325570 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-08bbf9-xmssn"] Nov 02 06:49:09.349477 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:49:09.349437 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37f32b77-9b37-4736-8885-88eb18a7c6d9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-08bbf9-xmssn\" (UID: \"37f32b77-9b37-4736-8885-88eb18a7c6d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-08bbf9-xmssn" Nov 02 06:49:09.349659 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:49:09.349587 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtd8m\" (UniqueName: \"kubernetes.io/projected/37f32b77-9b37-4736-8885-88eb18a7c6d9-kube-api-access-qtd8m\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-08bbf9-xmssn\" (UID: \"37f32b77-9b37-4736-8885-88eb18a7c6d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-08bbf9-xmssn" Nov 02 06:49:09.349718 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:49:09.349707 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37f32b77-9b37-4736-8885-88eb18a7c6d9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-08bbf9-xmssn\" (UID: \"37f32b77-9b37-4736-8885-88eb18a7c6d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-08bbf9-xmssn" Nov 02 06:49:09.370182 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:49:09.370151 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 06:49:09.370319 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:49:09.370205 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-68l4r\"" Nov 02 06:49:09.450936 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:49:09.450887 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37f32b77-9b37-4736-8885-88eb18a7c6d9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-08bbf9-xmssn\" (UID: \"37f32b77-9b37-4736-8885-88eb18a7c6d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-08bbf9-xmssn" Nov 02 06:49:09.451130 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:49:09.450981 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtd8m\" (UniqueName: \"kubernetes.io/projected/37f32b77-9b37-4736-8885-88eb18a7c6d9-kube-api-access-qtd8m\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-08bbf9-xmssn\" (UID: \"37f32b77-9b37-4736-8885-88eb18a7c6d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-08bbf9-xmssn" Nov 02 06:49:09.451130 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:49:09.451072 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37f32b77-9b37-4736-8885-88eb18a7c6d9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-08bbf9-xmssn\" (UID: \"37f32b77-9b37-4736-8885-88eb18a7c6d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-08bbf9-xmssn" Nov 02 06:49:09.451589 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:49:09.451564 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37f32b77-9b37-4736-8885-88eb18a7c6d9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-08bbf9-xmssn\" (UID: \"37f32b77-9b37-4736-8885-88eb18a7c6d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-08bbf9-xmssn" Nov 02 06:49:09.453498 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:49:09.453474 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37f32b77-9b37-4736-8885-88eb18a7c6d9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-08bbf9-xmssn\" (UID: \"37f32b77-9b37-4736-8885-88eb18a7c6d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-08bbf9-xmssn" Nov 02 06:49:09.509467 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:49:09.509440 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtd8m\" (UniqueName: \"kubernetes.io/projected/37f32b77-9b37-4736-8885-88eb18a7c6d9-kube-api-access-qtd8m\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-08bbf9-xmssn\" (UID: \"37f32b77-9b37-4736-8885-88eb18a7c6d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-08bbf9-xmssn" Nov 02 06:49:09.587494 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:49:09.587412 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-08bbf9-xmssn" Nov 02 06:49:09.661161 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:49:09.660492 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a2ba11-5c66894c8c-p472q" Nov 02 06:49:09.661967 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:49:09.661424 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-a2ba11-58967fcd4f-qpggq" Nov 02 06:49:10.012857 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:49:10.012822 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-08bbf9-xmssn"] Nov 02 06:49:10.014070 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:49:10.014041 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37f32b77_9b37_4736_8885_88eb18a7c6d9.slice/crio-2c1c472b6c141c6e28aac3c4898270769d821839025f5ca3913747ae55d2e9a9 WatchSource:0}: Error finding container 2c1c472b6c141c6e28aac3c4898270769d821839025f5ca3913747ae55d2e9a9: Status 404 returned error can't find the container with id 2c1c472b6c141c6e28aac3c4898270769d821839025f5ca3913747ae55d2e9a9 Nov 02 06:49:10.738007 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:49:10.737970 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-08bbf9-xmssn" event={"ID":"37f32b77-9b37-4736-8885-88eb18a7c6d9","Type":"ContainerStarted","Data":"7000134fb2522a45ba6f403832aa45064ed396ffb88e88492e88148e5737d2d5"} Nov 02 06:49:10.738436 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:49:10.738054 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-08bbf9-xmssn" event={"ID":"37f32b77-9b37-4736-8885-88eb18a7c6d9","Type":"ContainerStarted","Data":"2c1c472b6c141c6e28aac3c4898270769d821839025f5ca3913747ae55d2e9a9"} Nov 02 06:49:10.852749 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:49:10.852689 5916 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-08bbf9-xmssn" podStartSLOduration=1.852671921 podStartE2EDuration="1.852671921s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:49:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:49:10.851674515 +0000 UTC m=+1939.123494988" watchObservedRunningTime="2026-11-02 06:49:10.852671921 +0000 UTC m=+1939.124492316" Nov 02 06:49:11.807958 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:49:11.807921 5916 generic.go:356] "Generic (PLEG): container finished" podID="37f32b77-9b37-4736-8885-88eb18a7c6d9" containerID="7000134fb2522a45ba6f403832aa45064ed396ffb88e88492e88148e5737d2d5" exitCode=0 Nov 02 06:49:11.808654 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:49:11.808628 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-08bbf9-xmssn" event={"ID":"37f32b77-9b37-4736-8885-88eb18a7c6d9","Type":"ContainerDied","Data":"7000134fb2522a45ba6f403832aa45064ed396ffb88e88492e88148e5737d2d5"} Nov 02 06:49:13.170161 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:49:13.170119 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-08bbf9-xmssn" Nov 02 06:49:13.185571 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:49:13.185536 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/37f32b77-9b37-4736-8885-88eb18a7c6d9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37f32b77-9b37-4736-8885-88eb18a7c6d9-maas-api-xks-ca\") pod \"37f32b77-9b37-4736-8885-88eb18a7c6d9\" (UID: \"37f32b77-9b37-4736-8885-88eb18a7c6d9\") " Nov 02 06:49:13.185742 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:49:13.185586 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37f32b77-9b37-4736-8885-88eb18a7c6d9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37f32b77-9b37-4736-8885-88eb18a7c6d9-maas-api-openshift-ca\") pod \"37f32b77-9b37-4736-8885-88eb18a7c6d9\" (UID: \"37f32b77-9b37-4736-8885-88eb18a7c6d9\") " Nov 02 06:49:13.185742 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:49:13.185626 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37f32b77-9b37-4736-8885-88eb18a7c6d9-kube-api-access-qtd8m\" (UniqueName: \"kubernetes.io/projected/37f32b77-9b37-4736-8885-88eb18a7c6d9-kube-api-access-qtd8m\") pod \"37f32b77-9b37-4736-8885-88eb18a7c6d9\" (UID: \"37f32b77-9b37-4736-8885-88eb18a7c6d9\") " Nov 02 06:49:13.186077 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:49:13.186040 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37f32b77-9b37-4736-8885-88eb18a7c6d9-maas-api-openshift-ca" pod "37f32b77-9b37-4736-8885-88eb18a7c6d9" (UID: "37f32b77-9b37-4736-8885-88eb18a7c6d9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:49:13.186460 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:49:13.186434 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37f32b77-9b37-4736-8885-88eb18a7c6d9-maas-api-openshift-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:49:13.187988 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:49:13.187958 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37f32b77-9b37-4736-8885-88eb18a7c6d9-kube-api-access-qtd8m" pod "37f32b77-9b37-4736-8885-88eb18a7c6d9" (UID: "37f32b77-9b37-4736-8885-88eb18a7c6d9"). InnerVolumeSpecName "kube-api-access-qtd8m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:49:13.187988 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:49:13.187972 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37f32b77-9b37-4736-8885-88eb18a7c6d9-maas-api-xks-ca" pod "37f32b77-9b37-4736-8885-88eb18a7c6d9" (UID: "37f32b77-9b37-4736-8885-88eb18a7c6d9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:49:13.288008 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:49:13.287968 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37f32b77-9b37-4736-8885-88eb18a7c6d9-maas-api-xks-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:49:13.288008 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:49:13.288007 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtd8m\" (UniqueName: \"kubernetes.io/projected/37f32b77-9b37-4736-8885-88eb18a7c6d9-kube-api-access-qtd8m\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:49:13.849778 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:49:13.849748 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-08bbf9-xmssn" Nov 02 06:49:13.856178 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:49:13.856147 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-08bbf9-xmssn" event={"ID":"37f32b77-9b37-4736-8885-88eb18a7c6d9","Type":"ContainerDied","Data":"2c1c472b6c141c6e28aac3c4898270769d821839025f5ca3913747ae55d2e9a9"} Nov 02 06:49:13.856344 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:49:13.856198 5916 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2c1c472b6c141c6e28aac3c4898270769d821839025f5ca3913747ae55d2e9a9" Nov 02 06:49:19.895901 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:49:19.895867 5916 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p98cq" containerName="extractor" Nov 02 06:49:26.811827 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:49:26.811789 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-70898f-hphsf"] Nov 02 06:49:26.812430 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:49:26.812415 5916 cpu_manager.go:414] "containerMap: removing container" podUID="37f32b77-9b37-4736-8885-88eb18a7c6d9" containerName="revoke-keys" Nov 02 06:49:26.812477 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:49:26.812431 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37f32b77-9b37-4736-8885-88eb18a7c6d9" containerName="revoke-keys" Nov 02 06:49:26.812563 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:49:26.812554 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="37f32b77-9b37-4736-8885-88eb18a7c6d9" containerName="revoke-keys" Nov 02 06:49:26.816406 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:49:26.816388 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-70898f-hphsf" Nov 02 06:49:26.820966 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:49:26.820945 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b24eac55-a02b-4114-b6b5-0bcfe43c203b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-70898f-hphsf\" (UID: \"b24eac55-a02b-4114-b6b5-0bcfe43c203b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-70898f-hphsf" Nov 02 06:49:26.821058 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:49:26.820973 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b24eac55-a02b-4114-b6b5-0bcfe43c203b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-70898f-hphsf\" (UID: \"b24eac55-a02b-4114-b6b5-0bcfe43c203b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-70898f-hphsf" Nov 02 06:49:26.821183 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:49:26.821166 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-927z6\" (UniqueName: \"kubernetes.io/projected/b24eac55-a02b-4114-b6b5-0bcfe43c203b-kube-api-access-927z6\") pod \"maas-api-revoke-keys-e2e-api-a-70898f-hphsf\" (UID: \"b24eac55-a02b-4114-b6b5-0bcfe43c203b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-70898f-hphsf" Nov 02 06:49:26.871940 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:49:26.871891 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-70898f-hphsf"] Nov 02 06:49:26.917747 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:49:26.917716 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 06:49:26.917958 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:49:26.917785 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-68l4r\"" Nov 02 06:49:26.922424 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:49:26.922397 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b24eac55-a02b-4114-b6b5-0bcfe43c203b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-70898f-hphsf\" (UID: \"b24eac55-a02b-4114-b6b5-0bcfe43c203b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-70898f-hphsf" Nov 02 06:49:26.922424 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:49:26.922426 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b24eac55-a02b-4114-b6b5-0bcfe43c203b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-70898f-hphsf\" (UID: \"b24eac55-a02b-4114-b6b5-0bcfe43c203b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-70898f-hphsf" Nov 02 06:49:26.922605 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:49:26.922559 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-927z6\" (UniqueName: \"kubernetes.io/projected/b24eac55-a02b-4114-b6b5-0bcfe43c203b-kube-api-access-927z6\") pod \"maas-api-revoke-keys-e2e-api-a-70898f-hphsf\" (UID: \"b24eac55-a02b-4114-b6b5-0bcfe43c203b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-70898f-hphsf" Nov 02 06:49:26.923160 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:49:26.923137 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b24eac55-a02b-4114-b6b5-0bcfe43c203b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-70898f-hphsf\" (UID: \"b24eac55-a02b-4114-b6b5-0bcfe43c203b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-70898f-hphsf" Nov 02 06:49:26.925007 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:49:26.924983 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b24eac55-a02b-4114-b6b5-0bcfe43c203b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-70898f-hphsf\" (UID: \"b24eac55-a02b-4114-b6b5-0bcfe43c203b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-70898f-hphsf" Nov 02 06:49:26.983149 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:49:26.983121 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-927z6\" (UniqueName: \"kubernetes.io/projected/b24eac55-a02b-4114-b6b5-0bcfe43c203b-kube-api-access-927z6\") pod \"maas-api-revoke-keys-e2e-api-a-70898f-hphsf\" (UID: \"b24eac55-a02b-4114-b6b5-0bcfe43c203b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-70898f-hphsf" Nov 02 06:49:27.132276 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:49:27.132187 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-70898f-hphsf" Nov 02 06:49:27.540284 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:49:27.539999 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-70898f-hphsf"] Nov 02 06:49:27.540284 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:49:27.540049 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb24eac55_a02b_4114_b6b5_0bcfe43c203b.slice/crio-9ef39dd45ff7831331791ec5415a7d197861ebe0af2160e4c14506881fcb901a WatchSource:0}: Error finding container 9ef39dd45ff7831331791ec5415a7d197861ebe0af2160e4c14506881fcb901a: Status 404 returned error can't find the container with id 9ef39dd45ff7831331791ec5415a7d197861ebe0af2160e4c14506881fcb901a Nov 02 06:49:27.973402 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:49:27.973357 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-70898f-hphsf" event={"ID":"b24eac55-a02b-4114-b6b5-0bcfe43c203b","Type":"ContainerStarted","Data":"da58e7dc8cfbfb14d03adbd957ca48b64ee92c0b0f5c3c798cb736b73e866541"} Nov 02 06:49:27.973800 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:49:27.973414 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-70898f-hphsf" event={"ID":"b24eac55-a02b-4114-b6b5-0bcfe43c203b","Type":"ContainerStarted","Data":"9ef39dd45ff7831331791ec5415a7d197861ebe0af2160e4c14506881fcb901a"} Nov 02 06:49:28.088652 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:49:28.088532 5916 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-70898f-hphsf" podStartSLOduration=2.088517647 podStartE2EDuration="2.088517647s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:49:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:49:28.086822932 +0000 UTC m=+1956.358643328" watchObservedRunningTime="2026-11-02 06:49:28.088517647 +0000 UTC m=+1956.360338040" Nov 02 06:49:28.982353 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:49:28.982320 5916 generic.go:356] "Generic (PLEG): container finished" podID="b24eac55-a02b-4114-b6b5-0bcfe43c203b" containerID="da58e7dc8cfbfb14d03adbd957ca48b64ee92c0b0f5c3c798cb736b73e866541" exitCode=0 Nov 02 06:49:28.982784 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:49:28.982363 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-70898f-hphsf" event={"ID":"b24eac55-a02b-4114-b6b5-0bcfe43c203b","Type":"ContainerDied","Data":"da58e7dc8cfbfb14d03adbd957ca48b64ee92c0b0f5c3c798cb736b73e866541"} Nov 02 06:49:30.159398 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:49:30.159374 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-70898f-hphsf" Nov 02 06:49:30.270406 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:49:30.270325 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b24eac55-a02b-4114-b6b5-0bcfe43c203b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b24eac55-a02b-4114-b6b5-0bcfe43c203b-maas-api-xks-ca\") pod \"b24eac55-a02b-4114-b6b5-0bcfe43c203b\" (UID: \"b24eac55-a02b-4114-b6b5-0bcfe43c203b\") " Nov 02 06:49:30.270574 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:49:30.270503 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b24eac55-a02b-4114-b6b5-0bcfe43c203b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b24eac55-a02b-4114-b6b5-0bcfe43c203b-maas-api-openshift-ca\") pod \"b24eac55-a02b-4114-b6b5-0bcfe43c203b\" (UID: \"b24eac55-a02b-4114-b6b5-0bcfe43c203b\") " Nov 02 06:49:30.270574 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:49:30.270556 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b24eac55-a02b-4114-b6b5-0bcfe43c203b-kube-api-access-927z6\" (UniqueName: \"kubernetes.io/projected/b24eac55-a02b-4114-b6b5-0bcfe43c203b-kube-api-access-927z6\") pod \"b24eac55-a02b-4114-b6b5-0bcfe43c203b\" (UID: \"b24eac55-a02b-4114-b6b5-0bcfe43c203b\") " Nov 02 06:49:30.270923 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:49:30.270876 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b24eac55-a02b-4114-b6b5-0bcfe43c203b-maas-api-openshift-ca" pod "b24eac55-a02b-4114-b6b5-0bcfe43c203b" (UID: "b24eac55-a02b-4114-b6b5-0bcfe43c203b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:49:30.271125 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:49:30.271105 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b24eac55-a02b-4114-b6b5-0bcfe43c203b-maas-api-openshift-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:49:30.272656 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:49:30.272631 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b24eac55-a02b-4114-b6b5-0bcfe43c203b-maas-api-xks-ca" pod "b24eac55-a02b-4114-b6b5-0bcfe43c203b" (UID: "b24eac55-a02b-4114-b6b5-0bcfe43c203b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:49:30.274392 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:49:30.274359 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b24eac55-a02b-4114-b6b5-0bcfe43c203b-kube-api-access-927z6" pod "b24eac55-a02b-4114-b6b5-0bcfe43c203b" (UID: "b24eac55-a02b-4114-b6b5-0bcfe43c203b"). InnerVolumeSpecName "kube-api-access-927z6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:49:30.372154 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:49:30.372123 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b24eac55-a02b-4114-b6b5-0bcfe43c203b-maas-api-xks-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:49:30.372154 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:49:30.372151 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-927z6\" (UniqueName: \"kubernetes.io/projected/b24eac55-a02b-4114-b6b5-0bcfe43c203b-kube-api-access-927z6\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:49:31.001387 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:49:31.001358 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-70898f-hphsf" Nov 02 06:49:31.001387 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:49:31.001382 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-70898f-hphsf" event={"ID":"b24eac55-a02b-4114-b6b5-0bcfe43c203b","Type":"ContainerDied","Data":"9ef39dd45ff7831331791ec5415a7d197861ebe0af2160e4c14506881fcb901a"} Nov 02 06:49:31.001600 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:49:31.001426 5916 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9ef39dd45ff7831331791ec5415a7d197861ebe0af2160e4c14506881fcb901a" Nov 02 06:49:38.135579 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:49:38.135540 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-12bfb5-zhcsf"] Nov 02 06:49:38.184762 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:49:38.184727 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-12bfb5-zhcsf"] Nov 02 06:49:39.844118 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:49:39.844059 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c0da92a-5e4c-4e80-842a-da1642aabeb6" path="/var/lib/kubelet/pods/9c0da92a-5e4c-4e80-842a-da1642aabeb6/volumes" Nov 02 06:49:47.443036 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:49:47.442996 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-a25b83-qjk2w"] Nov 02 06:49:47.443899 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:49:47.443881 5916 cpu_manager.go:414] "containerMap: removing container" podUID="b24eac55-a02b-4114-b6b5-0bcfe43c203b" containerName="revoke-keys" Nov 02 06:49:47.443899 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:49:47.443915 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b24eac55-a02b-4114-b6b5-0bcfe43c203b" containerName="revoke-keys" Nov 02 06:49:47.444116 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:49:47.444104 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="b24eac55-a02b-4114-b6b5-0bcfe43c203b" containerName="revoke-keys" Nov 02 06:49:47.448302 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:49:47.448279 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-a25b83-qjk2w" Nov 02 06:49:47.502755 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:49:47.502726 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-a25b83-qjk2w"] Nov 02 06:49:47.547577 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:49:47.547547 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 06:49:47.547824 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:49:47.547804 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-68l4r\"" Nov 02 06:49:47.554869 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:49:47.554835 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e9622d0-be90-400f-a2fa-9dd997e323ac-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-a25b83-qjk2w\" (UID: \"5e9622d0-be90-400f-a2fa-9dd997e323ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-a25b83-qjk2w" Nov 02 06:49:47.555047 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:49:47.554875 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e9622d0-be90-400f-a2fa-9dd997e323ac-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-a25b83-qjk2w\" (UID: \"5e9622d0-be90-400f-a2fa-9dd997e323ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-a25b83-qjk2w" Nov 02 06:49:47.555148 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:49:47.555131 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxf44\" (UniqueName: \"kubernetes.io/projected/5e9622d0-be90-400f-a2fa-9dd997e323ac-kube-api-access-jxf44\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-a25b83-qjk2w\" (UID: \"5e9622d0-be90-400f-a2fa-9dd997e323ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-a25b83-qjk2w" Nov 02 06:49:47.656285 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:49:47.656248 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e9622d0-be90-400f-a2fa-9dd997e323ac-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-a25b83-qjk2w\" (UID: \"5e9622d0-be90-400f-a2fa-9dd997e323ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-a25b83-qjk2w" Nov 02 06:49:47.656479 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:49:47.656299 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e9622d0-be90-400f-a2fa-9dd997e323ac-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-a25b83-qjk2w\" (UID: \"5e9622d0-be90-400f-a2fa-9dd997e323ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-a25b83-qjk2w" Nov 02 06:49:47.656479 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:49:47.656466 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxf44\" (UniqueName: \"kubernetes.io/projected/5e9622d0-be90-400f-a2fa-9dd997e323ac-kube-api-access-jxf44\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-a25b83-qjk2w\" (UID: \"5e9622d0-be90-400f-a2fa-9dd997e323ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-a25b83-qjk2w" Nov 02 06:49:47.657187 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:49:47.657157 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e9622d0-be90-400f-a2fa-9dd997e323ac-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-a25b83-qjk2w\" (UID: \"5e9622d0-be90-400f-a2fa-9dd997e323ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-a25b83-qjk2w" Nov 02 06:49:47.659145 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:49:47.659121 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e9622d0-be90-400f-a2fa-9dd997e323ac-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-a25b83-qjk2w\" (UID: \"5e9622d0-be90-400f-a2fa-9dd997e323ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-a25b83-qjk2w" Nov 02 06:49:47.714646 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:49:47.714576 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxf44\" (UniqueName: \"kubernetes.io/projected/5e9622d0-be90-400f-a2fa-9dd997e323ac-kube-api-access-jxf44\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-a25b83-qjk2w\" (UID: \"5e9622d0-be90-400f-a2fa-9dd997e323ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-a25b83-qjk2w" Nov 02 06:49:47.762864 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:49:47.762833 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-a25b83-qjk2w" Nov 02 06:49:48.146096 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:49:48.146054 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e9622d0_be90_400f_a2fa_9dd997e323ac.slice/crio-2d4107a03b6cd2af6c1203f63ab3f18c889332eeb2fb55087239f69957df240e WatchSource:0}: Error finding container 2d4107a03b6cd2af6c1203f63ab3f18c889332eeb2fb55087239f69957df240e: Status 404 returned error can't find the container with id 2d4107a03b6cd2af6c1203f63ab3f18c889332eeb2fb55087239f69957df240e Nov 02 06:49:48.146292 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:49:48.146269 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-a25b83-qjk2w"] Nov 02 06:49:49.157265 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:49:49.157228 5916 generic.go:356] "Generic (PLEG): container finished" podID="5e9622d0-be90-400f-a2fa-9dd997e323ac" containerID="f16ff7aeb9afbffd78e1c7a8441621d241a0489fd22c6b750664ff560215bfe9" exitCode=0 Nov 02 06:49:49.157953 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:49:49.157920 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-a25b83-qjk2w" event={"ID":"5e9622d0-be90-400f-a2fa-9dd997e323ac","Type":"ContainerDied","Data":"f16ff7aeb9afbffd78e1c7a8441621d241a0489fd22c6b750664ff560215bfe9"} Nov 02 06:49:49.158171 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:49:49.158154 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-a25b83-qjk2w" event={"ID":"5e9622d0-be90-400f-a2fa-9dd997e323ac","Type":"ContainerStarted","Data":"2d4107a03b6cd2af6c1203f63ab3f18c889332eeb2fb55087239f69957df240e"} Nov 02 06:49:49.895598 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:49:49.895558 5916 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p98cq" containerName="extractor" Nov 02 06:49:50.335319 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:49:50.335293 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-a25b83-qjk2w" Nov 02 06:49:50.395030 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:49:50.395001 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e9622d0-be90-400f-a2fa-9dd997e323ac-kube-api-access-jxf44\" (UniqueName: \"kubernetes.io/projected/5e9622d0-be90-400f-a2fa-9dd997e323ac-kube-api-access-jxf44\") pod \"5e9622d0-be90-400f-a2fa-9dd997e323ac\" (UID: \"5e9622d0-be90-400f-a2fa-9dd997e323ac\") " Nov 02 06:49:50.395170 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:49:50.395122 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e9622d0-be90-400f-a2fa-9dd997e323ac-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e9622d0-be90-400f-a2fa-9dd997e323ac-maas-api-xks-ca\") pod \"5e9622d0-be90-400f-a2fa-9dd997e323ac\" (UID: \"5e9622d0-be90-400f-a2fa-9dd997e323ac\") " Nov 02 06:49:50.395241 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:49:50.395226 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e9622d0-be90-400f-a2fa-9dd997e323ac-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e9622d0-be90-400f-a2fa-9dd997e323ac-maas-api-openshift-ca\") pod \"5e9622d0-be90-400f-a2fa-9dd997e323ac\" (UID: \"5e9622d0-be90-400f-a2fa-9dd997e323ac\") " Nov 02 06:49:50.395621 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:49:50.395590 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e9622d0-be90-400f-a2fa-9dd997e323ac-maas-api-openshift-ca" pod "5e9622d0-be90-400f-a2fa-9dd997e323ac" (UID: "5e9622d0-be90-400f-a2fa-9dd997e323ac"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:49:50.395813 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:49:50.395795 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e9622d0-be90-400f-a2fa-9dd997e323ac-maas-api-openshift-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:49:50.397130 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:49:50.397107 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e9622d0-be90-400f-a2fa-9dd997e323ac-kube-api-access-jxf44" pod "5e9622d0-be90-400f-a2fa-9dd997e323ac" (UID: "5e9622d0-be90-400f-a2fa-9dd997e323ac"). InnerVolumeSpecName "kube-api-access-jxf44". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:49:50.397220 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:49:50.397186 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e9622d0-be90-400f-a2fa-9dd997e323ac-maas-api-xks-ca" pod "5e9622d0-be90-400f-a2fa-9dd997e323ac" (UID: "5e9622d0-be90-400f-a2fa-9dd997e323ac"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:49:50.496813 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:49:50.496732 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jxf44\" (UniqueName: \"kubernetes.io/projected/5e9622d0-be90-400f-a2fa-9dd997e323ac-kube-api-access-jxf44\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:49:50.496813 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:49:50.496773 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e9622d0-be90-400f-a2fa-9dd997e323ac-maas-api-xks-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:49:51.182776 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:49:51.182748 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-a25b83-qjk2w" Nov 02 06:49:51.182971 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:49:51.182772 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-a25b83-qjk2w" event={"ID":"5e9622d0-be90-400f-a2fa-9dd997e323ac","Type":"ContainerDied","Data":"2d4107a03b6cd2af6c1203f63ab3f18c889332eeb2fb55087239f69957df240e"} Nov 02 06:49:51.182971 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:49:51.182824 5916 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2d4107a03b6cd2af6c1203f63ab3f18c889332eeb2fb55087239f69957df240e" Nov 02 06:50:15.873627 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:15.873551 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-tzzcm"] Nov 02 06:50:15.874193 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:15.874175 5916 cpu_manager.go:414] "containerMap: removing container" podUID="5e9622d0-be90-400f-a2fa-9dd997e323ac" containerName="revoke-keys" Nov 02 06:50:15.874245 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:15.874195 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e9622d0-be90-400f-a2fa-9dd997e323ac" containerName="revoke-keys" Nov 02 06:50:15.874337 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:15.874327 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e9622d0-be90-400f-a2fa-9dd997e323ac" containerName="revoke-keys" Nov 02 06:50:15.877837 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:15.877813 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-tzzcm" Nov 02 06:50:15.921738 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:15.921708 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-5c92q"] Nov 02 06:50:15.925829 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:15.925802 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-5c92q" Nov 02 06:50:15.968125 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:15.968094 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-3bf02ebf-openshift-default-dockercfg-njbqs\"" Nov 02 06:50:15.976138 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:15.976107 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-tzzcm"] Nov 02 06:50:15.990945 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:15.990888 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b64d8a67-82cc-4714-87b8-2181f9388f07-workload-certs\") pod \"e2e-mt-3bf02ebf-openshift-default-6b7794758f-tzzcm\" (UID: \"b64d8a67-82cc-4714-87b8-2181f9388f07\") " pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-tzzcm" Nov 02 06:50:15.990945 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:15.990935 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b64d8a67-82cc-4714-87b8-2181f9388f07-istio-token\") pod \"e2e-mt-3bf02ebf-openshift-default-6b7794758f-tzzcm\" (UID: \"b64d8a67-82cc-4714-87b8-2181f9388f07\") " pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-tzzcm" Nov 02 06:50:15.991136 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:15.990970 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b64d8a67-82cc-4714-87b8-2181f9388f07-istiod-ca-cert\") pod \"e2e-mt-3bf02ebf-openshift-default-6b7794758f-tzzcm\" (UID: \"b64d8a67-82cc-4714-87b8-2181f9388f07\") " pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-tzzcm" Nov 02 06:50:15.991136 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:15.991048 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b64d8a67-82cc-4714-87b8-2181f9388f07-istio-envoy\") pod \"e2e-mt-3bf02ebf-openshift-default-6b7794758f-tzzcm\" (UID: \"b64d8a67-82cc-4714-87b8-2181f9388f07\") " pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-tzzcm" Nov 02 06:50:15.991136 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:15.991084 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5srz\" (UniqueName: \"kubernetes.io/projected/b64d8a67-82cc-4714-87b8-2181f9388f07-kube-api-access-h5srz\") pod \"e2e-mt-3bf02ebf-openshift-default-6b7794758f-tzzcm\" (UID: \"b64d8a67-82cc-4714-87b8-2181f9388f07\") " pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-tzzcm" Nov 02 06:50:15.991245 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:15.991225 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b64d8a67-82cc-4714-87b8-2181f9388f07-istio-data\") pod \"e2e-mt-3bf02ebf-openshift-default-6b7794758f-tzzcm\" (UID: \"b64d8a67-82cc-4714-87b8-2181f9388f07\") " pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-tzzcm" Nov 02 06:50:15.991361 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:15.991341 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b64d8a67-82cc-4714-87b8-2181f9388f07-istio-podinfo\") pod \"e2e-mt-3bf02ebf-openshift-default-6b7794758f-tzzcm\" (UID: \"b64d8a67-82cc-4714-87b8-2181f9388f07\") " pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-tzzcm" Nov 02 06:50:15.991443 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:15.991427 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b64d8a67-82cc-4714-87b8-2181f9388f07-workload-socket\") pod \"e2e-mt-3bf02ebf-openshift-default-6b7794758f-tzzcm\" (UID: \"b64d8a67-82cc-4714-87b8-2181f9388f07\") " pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-tzzcm" Nov 02 06:50:15.991523 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:15.991504 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b64d8a67-82cc-4714-87b8-2181f9388f07-credential-socket\") pod \"e2e-mt-3bf02ebf-openshift-default-6b7794758f-tzzcm\" (UID: \"b64d8a67-82cc-4714-87b8-2181f9388f07\") " pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-tzzcm" Nov 02 06:50:16.002670 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:16.002635 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-5c92q"] Nov 02 06:50:16.093062 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:16.093030 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15eca023-182b-433b-a2ba-38c0b4e8a1ec-istio-podinfo\") pod \"e2e-mt-3bf02ebf-openshift-default-6b7794758f-5c92q\" (UID: \"15eca023-182b-433b-a2ba-38c0b4e8a1ec\") " pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-5c92q" Nov 02 06:50:16.093247 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:16.093073 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15eca023-182b-433b-a2ba-38c0b4e8a1ec-credential-socket\") pod \"e2e-mt-3bf02ebf-openshift-default-6b7794758f-5c92q\" (UID: \"15eca023-182b-433b-a2ba-38c0b4e8a1ec\") " pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-5c92q" Nov 02 06:50:16.093247 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:16.093094 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15eca023-182b-433b-a2ba-38c0b4e8a1ec-workload-certs\") pod \"e2e-mt-3bf02ebf-openshift-default-6b7794758f-5c92q\" (UID: \"15eca023-182b-433b-a2ba-38c0b4e8a1ec\") " pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-5c92q" Nov 02 06:50:16.093247 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:16.093216 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b64d8a67-82cc-4714-87b8-2181f9388f07-workload-certs\") pod \"e2e-mt-3bf02ebf-openshift-default-6b7794758f-tzzcm\" (UID: \"b64d8a67-82cc-4714-87b8-2181f9388f07\") " pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-tzzcm" Nov 02 06:50:16.093433 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:16.093247 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b64d8a67-82cc-4714-87b8-2181f9388f07-istio-token\") pod \"e2e-mt-3bf02ebf-openshift-default-6b7794758f-tzzcm\" (UID: \"b64d8a67-82cc-4714-87b8-2181f9388f07\") " pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-tzzcm" Nov 02 06:50:16.093433 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:16.093306 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b64d8a67-82cc-4714-87b8-2181f9388f07-istiod-ca-cert\") pod \"e2e-mt-3bf02ebf-openshift-default-6b7794758f-tzzcm\" (UID: \"b64d8a67-82cc-4714-87b8-2181f9388f07\") " pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-tzzcm" Nov 02 06:50:16.093433 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:16.093335 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b64d8a67-82cc-4714-87b8-2181f9388f07-istio-envoy\") pod \"e2e-mt-3bf02ebf-openshift-default-6b7794758f-tzzcm\" (UID: \"b64d8a67-82cc-4714-87b8-2181f9388f07\") " pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-tzzcm" Nov 02 06:50:16.093433 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:16.093360 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5srz\" (UniqueName: \"kubernetes.io/projected/b64d8a67-82cc-4714-87b8-2181f9388f07-kube-api-access-h5srz\") pod \"e2e-mt-3bf02ebf-openshift-default-6b7794758f-tzzcm\" (UID: \"b64d8a67-82cc-4714-87b8-2181f9388f07\") " pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-tzzcm" Nov 02 06:50:16.093433 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:16.093417 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15eca023-182b-433b-a2ba-38c0b4e8a1ec-istiod-ca-cert\") pod \"e2e-mt-3bf02ebf-openshift-default-6b7794758f-5c92q\" (UID: \"15eca023-182b-433b-a2ba-38c0b4e8a1ec\") " pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-5c92q" Nov 02 06:50:16.093674 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:16.093476 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b64d8a67-82cc-4714-87b8-2181f9388f07-istio-data\") pod \"e2e-mt-3bf02ebf-openshift-default-6b7794758f-tzzcm\" (UID: \"b64d8a67-82cc-4714-87b8-2181f9388f07\") " pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-tzzcm" Nov 02 06:50:16.093674 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:16.093594 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15eca023-182b-433b-a2ba-38c0b4e8a1ec-istio-envoy\") pod \"e2e-mt-3bf02ebf-openshift-default-6b7794758f-5c92q\" (UID: \"15eca023-182b-433b-a2ba-38c0b4e8a1ec\") " pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-5c92q" Nov 02 06:50:16.093674 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:16.093659 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b64d8a67-82cc-4714-87b8-2181f9388f07-workload-certs\") pod \"e2e-mt-3bf02ebf-openshift-default-6b7794758f-tzzcm\" (UID: \"b64d8a67-82cc-4714-87b8-2181f9388f07\") " pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-tzzcm" Nov 02 06:50:16.093853 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:16.093728 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15eca023-182b-433b-a2ba-38c0b4e8a1ec-workload-socket\") pod \"e2e-mt-3bf02ebf-openshift-default-6b7794758f-5c92q\" (UID: \"15eca023-182b-433b-a2ba-38c0b4e8a1ec\") " pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-5c92q" Nov 02 06:50:16.093853 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:16.093783 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b64d8a67-82cc-4714-87b8-2181f9388f07-istio-podinfo\") pod \"e2e-mt-3bf02ebf-openshift-default-6b7794758f-tzzcm\" (UID: \"b64d8a67-82cc-4714-87b8-2181f9388f07\") " pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-tzzcm" Nov 02 06:50:16.093853 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:16.093815 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnks7\" (UniqueName: \"kubernetes.io/projected/15eca023-182b-433b-a2ba-38c0b4e8a1ec-kube-api-access-dnks7\") pod \"e2e-mt-3bf02ebf-openshift-default-6b7794758f-5c92q\" (UID: \"15eca023-182b-433b-a2ba-38c0b4e8a1ec\") " pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-5c92q" Nov 02 06:50:16.094060 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:16.093865 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b64d8a67-82cc-4714-87b8-2181f9388f07-workload-socket\") pod \"e2e-mt-3bf02ebf-openshift-default-6b7794758f-tzzcm\" (UID: \"b64d8a67-82cc-4714-87b8-2181f9388f07\") " pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-tzzcm" Nov 02 06:50:16.094060 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:16.093871 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b64d8a67-82cc-4714-87b8-2181f9388f07-istio-data\") pod \"e2e-mt-3bf02ebf-openshift-default-6b7794758f-tzzcm\" (UID: \"b64d8a67-82cc-4714-87b8-2181f9388f07\") " pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-tzzcm" Nov 02 06:50:16.094060 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:16.093984 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15eca023-182b-433b-a2ba-38c0b4e8a1ec-istio-data\") pod \"e2e-mt-3bf02ebf-openshift-default-6b7794758f-5c92q\" (UID: \"15eca023-182b-433b-a2ba-38c0b4e8a1ec\") " pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-5c92q" Nov 02 06:50:16.094060 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:16.094033 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b64d8a67-82cc-4714-87b8-2181f9388f07-credential-socket\") pod \"e2e-mt-3bf02ebf-openshift-default-6b7794758f-tzzcm\" (UID: \"b64d8a67-82cc-4714-87b8-2181f9388f07\") " pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-tzzcm" Nov 02 06:50:16.094060 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:16.094054 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b64d8a67-82cc-4714-87b8-2181f9388f07-istiod-ca-cert\") pod \"e2e-mt-3bf02ebf-openshift-default-6b7794758f-tzzcm\" (UID: \"b64d8a67-82cc-4714-87b8-2181f9388f07\") " pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-tzzcm" Nov 02 06:50:16.094319 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:16.094204 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15eca023-182b-433b-a2ba-38c0b4e8a1ec-istio-token\") pod \"e2e-mt-3bf02ebf-openshift-default-6b7794758f-5c92q\" (UID: \"15eca023-182b-433b-a2ba-38c0b4e8a1ec\") " pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-5c92q" Nov 02 06:50:16.094319 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:16.094307 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b64d8a67-82cc-4714-87b8-2181f9388f07-credential-socket\") pod \"e2e-mt-3bf02ebf-openshift-default-6b7794758f-tzzcm\" (UID: \"b64d8a67-82cc-4714-87b8-2181f9388f07\") " pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-tzzcm" Nov 02 06:50:16.094415 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:16.094343 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b64d8a67-82cc-4714-87b8-2181f9388f07-workload-socket\") pod \"e2e-mt-3bf02ebf-openshift-default-6b7794758f-tzzcm\" (UID: \"b64d8a67-82cc-4714-87b8-2181f9388f07\") " pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-tzzcm" Nov 02 06:50:16.095634 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:16.095616 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b64d8a67-82cc-4714-87b8-2181f9388f07-istio-envoy\") pod \"e2e-mt-3bf02ebf-openshift-default-6b7794758f-tzzcm\" (UID: \"b64d8a67-82cc-4714-87b8-2181f9388f07\") " pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-tzzcm" Nov 02 06:50:16.096100 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:16.096077 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b64d8a67-82cc-4714-87b8-2181f9388f07-istio-podinfo\") pod \"e2e-mt-3bf02ebf-openshift-default-6b7794758f-tzzcm\" (UID: \"b64d8a67-82cc-4714-87b8-2181f9388f07\") " pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-tzzcm" Nov 02 06:50:16.198149 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:16.197134 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15eca023-182b-433b-a2ba-38c0b4e8a1ec-workload-socket\") pod \"e2e-mt-3bf02ebf-openshift-default-6b7794758f-5c92q\" (UID: \"15eca023-182b-433b-a2ba-38c0b4e8a1ec\") " pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-5c92q" Nov 02 06:50:16.198149 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:16.197220 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnks7\" (UniqueName: \"kubernetes.io/projected/15eca023-182b-433b-a2ba-38c0b4e8a1ec-kube-api-access-dnks7\") pod \"e2e-mt-3bf02ebf-openshift-default-6b7794758f-5c92q\" (UID: \"15eca023-182b-433b-a2ba-38c0b4e8a1ec\") " pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-5c92q" Nov 02 06:50:16.198149 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:16.197298 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15eca023-182b-433b-a2ba-38c0b4e8a1ec-istio-data\") pod \"e2e-mt-3bf02ebf-openshift-default-6b7794758f-5c92q\" (UID: \"15eca023-182b-433b-a2ba-38c0b4e8a1ec\") " pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-5c92q" Nov 02 06:50:16.198149 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:16.197352 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15eca023-182b-433b-a2ba-38c0b4e8a1ec-istio-token\") pod \"e2e-mt-3bf02ebf-openshift-default-6b7794758f-5c92q\" (UID: \"15eca023-182b-433b-a2ba-38c0b4e8a1ec\") " pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-5c92q" Nov 02 06:50:16.198149 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:16.197400 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15eca023-182b-433b-a2ba-38c0b4e8a1ec-istio-podinfo\") pod \"e2e-mt-3bf02ebf-openshift-default-6b7794758f-5c92q\" (UID: \"15eca023-182b-433b-a2ba-38c0b4e8a1ec\") " pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-5c92q" Nov 02 06:50:16.198149 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:16.197433 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15eca023-182b-433b-a2ba-38c0b4e8a1ec-credential-socket\") pod \"e2e-mt-3bf02ebf-openshift-default-6b7794758f-5c92q\" (UID: \"15eca023-182b-433b-a2ba-38c0b4e8a1ec\") " pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-5c92q" Nov 02 06:50:16.198149 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:16.197459 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15eca023-182b-433b-a2ba-38c0b4e8a1ec-workload-certs\") pod \"e2e-mt-3bf02ebf-openshift-default-6b7794758f-5c92q\" (UID: \"15eca023-182b-433b-a2ba-38c0b4e8a1ec\") " pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-5c92q" Nov 02 06:50:16.198149 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:16.197563 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15eca023-182b-433b-a2ba-38c0b4e8a1ec-istiod-ca-cert\") pod \"e2e-mt-3bf02ebf-openshift-default-6b7794758f-5c92q\" (UID: \"15eca023-182b-433b-a2ba-38c0b4e8a1ec\") " pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-5c92q" Nov 02 06:50:16.198149 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:16.197655 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15eca023-182b-433b-a2ba-38c0b4e8a1ec-istio-envoy\") pod \"e2e-mt-3bf02ebf-openshift-default-6b7794758f-5c92q\" (UID: \"15eca023-182b-433b-a2ba-38c0b4e8a1ec\") " pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-5c92q" Nov 02 06:50:16.199515 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:16.199484 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15eca023-182b-433b-a2ba-38c0b4e8a1ec-workload-socket\") pod \"e2e-mt-3bf02ebf-openshift-default-6b7794758f-5c92q\" (UID: \"15eca023-182b-433b-a2ba-38c0b4e8a1ec\") " pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-5c92q" Nov 02 06:50:16.199747 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:16.199654 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15eca023-182b-433b-a2ba-38c0b4e8a1ec-istio-data\") pod \"e2e-mt-3bf02ebf-openshift-default-6b7794758f-5c92q\" (UID: \"15eca023-182b-433b-a2ba-38c0b4e8a1ec\") " pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-5c92q" Nov 02 06:50:16.200036 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:16.200006 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15eca023-182b-433b-a2ba-38c0b4e8a1ec-credential-socket\") pod \"e2e-mt-3bf02ebf-openshift-default-6b7794758f-5c92q\" (UID: \"15eca023-182b-433b-a2ba-38c0b4e8a1ec\") " pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-5c92q" Nov 02 06:50:16.200535 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:16.200503 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15eca023-182b-433b-a2ba-38c0b4e8a1ec-workload-certs\") pod \"e2e-mt-3bf02ebf-openshift-default-6b7794758f-5c92q\" (UID: \"15eca023-182b-433b-a2ba-38c0b4e8a1ec\") " pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-5c92q" Nov 02 06:50:16.214926 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:16.211585 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15eca023-182b-433b-a2ba-38c0b4e8a1ec-istiod-ca-cert\") pod \"e2e-mt-3bf02ebf-openshift-default-6b7794758f-5c92q\" (UID: \"15eca023-182b-433b-a2ba-38c0b4e8a1ec\") " pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-5c92q" Nov 02 06:50:16.214926 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:16.212249 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b64d8a67-82cc-4714-87b8-2181f9388f07-istio-token\") pod \"e2e-mt-3bf02ebf-openshift-default-6b7794758f-tzzcm\" (UID: \"b64d8a67-82cc-4714-87b8-2181f9388f07\") " pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-tzzcm" Nov 02 06:50:16.258522 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:16.258457 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnks7\" (UniqueName: \"kubernetes.io/projected/15eca023-182b-433b-a2ba-38c0b4e8a1ec-kube-api-access-dnks7\") pod \"e2e-mt-3bf02ebf-openshift-default-6b7794758f-5c92q\" (UID: \"15eca023-182b-433b-a2ba-38c0b4e8a1ec\") " pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-5c92q" Nov 02 06:50:16.262762 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:16.262734 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15eca023-182b-433b-a2ba-38c0b4e8a1ec-istio-envoy\") pod \"e2e-mt-3bf02ebf-openshift-default-6b7794758f-5c92q\" (UID: \"15eca023-182b-433b-a2ba-38c0b4e8a1ec\") " pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-5c92q" Nov 02 06:50:16.263138 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:16.263111 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15eca023-182b-433b-a2ba-38c0b4e8a1ec-istio-podinfo\") pod \"e2e-mt-3bf02ebf-openshift-default-6b7794758f-5c92q\" (UID: \"15eca023-182b-433b-a2ba-38c0b4e8a1ec\") " pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-5c92q" Nov 02 06:50:16.263644 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:16.263623 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5srz\" (UniqueName: \"kubernetes.io/projected/b64d8a67-82cc-4714-87b8-2181f9388f07-kube-api-access-h5srz\") pod \"e2e-mt-3bf02ebf-openshift-default-6b7794758f-tzzcm\" (UID: \"b64d8a67-82cc-4714-87b8-2181f9388f07\") " pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-tzzcm" Nov 02 06:50:16.267446 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:16.267406 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15eca023-182b-433b-a2ba-38c0b4e8a1ec-istio-token\") pod \"e2e-mt-3bf02ebf-openshift-default-6b7794758f-5c92q\" (UID: \"15eca023-182b-433b-a2ba-38c0b4e8a1ec\") " pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-5c92q" Nov 02 06:50:16.492384 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:16.492303 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-tzzcm" Nov 02 06:50:16.542929 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:16.542858 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-5c92q" Nov 02 06:50:16.964753 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:16.964710 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-tzzcm"] Nov 02 06:50:16.966101 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:50:16.966061 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb64d8a67_82cc_4714_87b8_2181f9388f07.slice/crio-d53139037d7e83c2390d25bfbc4565c1784eac48832d5cab06a438ba53d0b761 WatchSource:0}: Error finding container d53139037d7e83c2390d25bfbc4565c1784eac48832d5cab06a438ba53d0b761: Status 404 returned error can't find the container with id d53139037d7e83c2390d25bfbc4565c1784eac48832d5cab06a438ba53d0b761 Nov 02 06:50:16.969482 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:16.969438 5916 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 06:50:16.969591 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:16.969529 5916 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 06:50:16.969591 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:16.969572 5916 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 06:50:16.992536 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:16.992506 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-5c92q"] Nov 02 06:50:17.017809 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:17.017761 5916 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 06:50:17.017982 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:17.017849 5916 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 06:50:17.017982 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:17.017882 5916 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 06:50:17.479952 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:17.477672 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-a25b83-qjk2w"] Nov 02 06:50:17.483898 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:17.483554 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-a25b83-qjk2w"] Nov 02 06:50:17.576021 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:17.574497 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-tzzcm" event={"ID":"b64d8a67-82cc-4714-87b8-2181f9388f07","Type":"ContainerStarted","Data":"d27cea362f78aa9bdaeee118c6f93dce1368069453c17c2f286e136c4231f442"} Nov 02 06:50:17.576021 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:17.574566 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-tzzcm" event={"ID":"b64d8a67-82cc-4714-87b8-2181f9388f07","Type":"ContainerStarted","Data":"d53139037d7e83c2390d25bfbc4565c1784eac48832d5cab06a438ba53d0b761"} Nov 02 06:50:17.580478 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:17.579635 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-5c92q" event={"ID":"15eca023-182b-433b-a2ba-38c0b4e8a1ec","Type":"ContainerStarted","Data":"2e631eb9f7e93acbef72ef7d30ef7417c936f9f3f96386eabbab20d7deb3f2b3"} Nov 02 06:50:17.580478 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:17.579692 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-5c92q" event={"ID":"15eca023-182b-433b-a2ba-38c0b4e8a1ec","Type":"ContainerStarted","Data":"8aa6d8dc8af8b036297686a421554f116118df1cde24d673c34fa74c41e12015"} Nov 02 06:50:17.714328 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:17.714253 5916 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-tzzcm" podStartSLOduration=2.714229128 podStartE2EDuration="2.714229128s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:50:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:50:17.696307542 +0000 UTC m=+2005.968127937" watchObservedRunningTime="2026-11-02 06:50:17.714229128 +0000 UTC m=+2005.986049523" Nov 02 06:50:17.858936 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:17.853150 5916 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-5c92q" podStartSLOduration=2.8531298290000002 podStartE2EDuration="2.853129829s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:50:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:50:17.832328897 +0000 UTC m=+2006.104149291" watchObservedRunningTime="2026-11-02 06:50:17.853129829 +0000 UTC m=+2006.124950223" Nov 02 06:50:17.881547 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:17.881511 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e9622d0-be90-400f-a2fa-9dd997e323ac" path="/var/lib/kubelet/pods/5e9622d0-be90-400f-a2fa-9dd997e323ac/volumes" Nov 02 06:50:18.494923 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:18.493197 5916 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-tzzcm" Nov 02 06:50:18.505627 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:18.505373 5916 patch_prober.go:29] interesting pod/e2e-mt-3bf02ebf-openshift-default-6b7794758f-tzzcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body= Nov 02 06:50:18.505627 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:18.505438 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-tzzcm" podUID="b64d8a67-82cc-4714-87b8-2181f9388f07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" Nov 02 06:50:18.547932 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:18.547076 5916 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-5c92q" Nov 02 06:50:18.566928 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:18.566020 5916 patch_prober.go:29] interesting pod/e2e-mt-3bf02ebf-openshift-default-6b7794758f-5c92q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body= Nov 02 06:50:18.566928 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:18.566075 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-5c92q" podUID="15eca023-182b-433b-a2ba-38c0b4e8a1ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" Nov 02 06:50:19.496930 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:19.493504 5916 patch_prober.go:29] interesting pod/e2e-mt-3bf02ebf-openshift-default-6b7794758f-tzzcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body= Nov 02 06:50:19.496930 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:19.493580 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-tzzcm" podUID="b64d8a67-82cc-4714-87b8-2181f9388f07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" Nov 02 06:50:19.545931 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:19.544295 5916 patch_prober.go:29] interesting pod/e2e-mt-3bf02ebf-openshift-default-6b7794758f-5c92q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body= Nov 02 06:50:19.545931 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:19.544358 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-5c92q" podUID="15eca023-182b-433b-a2ba-38c0b4e8a1ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" Nov 02 06:50:19.989946 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:50:19.989891 5916 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p98cq" containerName="extractor" Nov 02 06:50:20.494924 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:20.492657 5916 patch_prober.go:29] interesting pod/e2e-mt-3bf02ebf-openshift-default-6b7794758f-tzzcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body= Nov 02 06:50:20.494924 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:20.492718 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-tzzcm" podUID="b64d8a67-82cc-4714-87b8-2181f9388f07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" Nov 02 06:50:20.557930 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:20.557002 5916 patch_prober.go:29] interesting pod/e2e-mt-3bf02ebf-openshift-default-6b7794758f-5c92q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body= Nov 02 06:50:20.557930 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:20.557060 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-5c92q" podUID="15eca023-182b-433b-a2ba-38c0b4e8a1ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" Nov 02 06:50:21.493128 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:21.493096 5916 patch_prober.go:29] interesting pod/e2e-mt-3bf02ebf-openshift-default-6b7794758f-tzzcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body= Nov 02 06:50:21.493370 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:21.493349 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-tzzcm" podUID="b64d8a67-82cc-4714-87b8-2181f9388f07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" Nov 02 06:50:21.545491 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:21.545328 5916 patch_prober.go:29] interesting pod/e2e-mt-3bf02ebf-openshift-default-6b7794758f-5c92q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body= Nov 02 06:50:21.545491 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:21.545391 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-5c92q" podUID="15eca023-182b-433b-a2ba-38c0b4e8a1ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" Nov 02 06:50:22.530074 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:22.530031 5916 patch_prober.go:29] interesting pod/e2e-mt-3bf02ebf-openshift-default-6b7794758f-tzzcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body= Nov 02 06:50:22.530707 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:22.530672 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-tzzcm" podUID="b64d8a67-82cc-4714-87b8-2181f9388f07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" Nov 02 06:50:22.544926 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:22.543230 5916 patch_prober.go:29] interesting pod/e2e-mt-3bf02ebf-openshift-default-6b7794758f-5c92q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body= Nov 02 06:50:22.544926 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:22.543289 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-5c92q" podUID="15eca023-182b-433b-a2ba-38c0b4e8a1ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" Nov 02 06:50:23.492932 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:23.492721 5916 patch_prober.go:29] interesting pod/e2e-mt-3bf02ebf-openshift-default-6b7794758f-tzzcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body= Nov 02 06:50:23.492932 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:23.492826 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-tzzcm" podUID="b64d8a67-82cc-4714-87b8-2181f9388f07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" Nov 02 06:50:23.564941 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:23.553814 5916 patch_prober.go:29] interesting pod/e2e-mt-3bf02ebf-openshift-default-6b7794758f-5c92q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body= Nov 02 06:50:23.564941 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:23.557028 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-5c92q" podUID="15eca023-182b-433b-a2ba-38c0b4e8a1ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" Nov 02 06:50:24.544320 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:24.544281 5916 patch_prober.go:29] interesting pod/e2e-mt-3bf02ebf-openshift-default-6b7794758f-5c92q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body= Nov 02 06:50:24.544600 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:24.544569 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-5c92q" podUID="15eca023-182b-433b-a2ba-38c0b4e8a1ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" Nov 02 06:50:24.797978 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:24.797869 5916 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-tzzcm" Nov 02 06:50:24.799525 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:24.799501 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-tzzcm" Nov 02 06:50:24.802620 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:24.802595 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-tzzcm" Nov 02 06:50:25.103073 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:25.102982 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-08bbf9-xmssn"] Nov 02 06:50:25.110844 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:25.110819 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-08bbf9-xmssn"] Nov 02 06:50:25.548824 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:25.548792 5916 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-5c92q" Nov 02 06:50:25.549246 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:25.549219 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-5c92q" Nov 02 06:50:25.550120 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:25.550101 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-5c92q" Nov 02 06:50:25.874097 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:25.870292 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37f32b77-9b37-4736-8885-88eb18a7c6d9" path="/var/lib/kubelet/pods/37f32b77-9b37-4736-8885-88eb18a7c6d9/volumes" Nov 02 06:50:26.351045 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:26.351007 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-29abe7-f28qb"] Nov 02 06:50:26.356497 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:26.356475 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-29abe7-f28qb" Nov 02 06:50:26.412013 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:26.411977 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-29abe7-f28qb"] Nov 02 06:50:26.412615 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:26.412593 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f8fbc11-84db-4998-9f70-acc2ea9ccef7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-29abe7-f28qb\" (UID: \"0f8fbc11-84db-4998-9f70-acc2ea9ccef7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-29abe7-f28qb" Nov 02 06:50:26.412804 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:26.412789 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f8fbc11-84db-4998-9f70-acc2ea9ccef7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-29abe7-f28qb\" (UID: \"0f8fbc11-84db-4998-9f70-acc2ea9ccef7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-29abe7-f28qb" Nov 02 06:50:26.412871 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:26.412857 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdd6d\" (UniqueName: \"kubernetes.io/projected/0f8fbc11-84db-4998-9f70-acc2ea9ccef7-kube-api-access-rdd6d\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-29abe7-f28qb\" (UID: \"0f8fbc11-84db-4998-9f70-acc2ea9ccef7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-29abe7-f28qb" Nov 02 06:50:26.463882 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:26.463847 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 06:50:26.464087 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:26.464047 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-68l4r\"" Nov 02 06:50:26.514289 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:26.514259 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f8fbc11-84db-4998-9f70-acc2ea9ccef7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-29abe7-f28qb\" (UID: \"0f8fbc11-84db-4998-9f70-acc2ea9ccef7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-29abe7-f28qb" Nov 02 06:50:26.514439 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:26.514327 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdd6d\" (UniqueName: \"kubernetes.io/projected/0f8fbc11-84db-4998-9f70-acc2ea9ccef7-kube-api-access-rdd6d\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-29abe7-f28qb\" (UID: \"0f8fbc11-84db-4998-9f70-acc2ea9ccef7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-29abe7-f28qb" Nov 02 06:50:26.514439 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:26.514366 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f8fbc11-84db-4998-9f70-acc2ea9ccef7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-29abe7-f28qb\" (UID: \"0f8fbc11-84db-4998-9f70-acc2ea9ccef7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-29abe7-f28qb" Nov 02 06:50:26.514962 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:26.514901 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f8fbc11-84db-4998-9f70-acc2ea9ccef7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-29abe7-f28qb\" (UID: \"0f8fbc11-84db-4998-9f70-acc2ea9ccef7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-29abe7-f28qb" Nov 02 06:50:26.516720 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:26.516702 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f8fbc11-84db-4998-9f70-acc2ea9ccef7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-29abe7-f28qb\" (UID: \"0f8fbc11-84db-4998-9f70-acc2ea9ccef7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-29abe7-f28qb" Nov 02 06:50:26.585284 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:26.585248 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdd6d\" (UniqueName: \"kubernetes.io/projected/0f8fbc11-84db-4998-9f70-acc2ea9ccef7-kube-api-access-rdd6d\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-29abe7-f28qb\" (UID: \"0f8fbc11-84db-4998-9f70-acc2ea9ccef7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-29abe7-f28qb" Nov 02 06:50:26.675557 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:26.675514 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-29abe7-f28qb" Nov 02 06:50:27.060564 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:27.060532 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-29abe7-f28qb"] Nov 02 06:50:27.061791 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:50:27.061767 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f8fbc11_84db_4998_9f70_acc2ea9ccef7.slice/crio-262df11d6a74d87945b5b89d3006b5bc4f227d74b572fc27913c8591145e29f4 WatchSource:0}: Error finding container 262df11d6a74d87945b5b89d3006b5bc4f227d74b572fc27913c8591145e29f4: Status 404 returned error can't find the container with id 262df11d6a74d87945b5b89d3006b5bc4f227d74b572fc27913c8591145e29f4 Nov 02 06:50:27.862185 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:27.862100 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-29abe7-f28qb" event={"ID":"0f8fbc11-84db-4998-9f70-acc2ea9ccef7","Type":"ContainerStarted","Data":"bf05711a7cfc850bdfaea5762e9789763e72bc5a0f6aac3302d539ee7f1c40d6"} Nov 02 06:50:27.862185 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:27.862158 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-29abe7-f28qb" event={"ID":"0f8fbc11-84db-4998-9f70-acc2ea9ccef7","Type":"ContainerStarted","Data":"262df11d6a74d87945b5b89d3006b5bc4f227d74b572fc27913c8591145e29f4"} Nov 02 06:50:27.976538 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:27.976492 5916 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-29abe7-f28qb" podStartSLOduration=1.9764746739999999 podStartE2EDuration="1.976474674s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:50:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:50:27.975449066 +0000 UTC m=+2016.247269461" watchObservedRunningTime="2026-11-02 06:50:27.976474674 +0000 UTC m=+2016.248295067" Nov 02 06:50:28.871708 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:28.871670 5916 generic.go:356] "Generic (PLEG): container finished" podID="0f8fbc11-84db-4998-9f70-acc2ea9ccef7" containerID="bf05711a7cfc850bdfaea5762e9789763e72bc5a0f6aac3302d539ee7f1c40d6" exitCode=0 Nov 02 06:50:28.872169 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:28.871760 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-29abe7-f28qb" event={"ID":"0f8fbc11-84db-4998-9f70-acc2ea9ccef7","Type":"ContainerDied","Data":"bf05711a7cfc850bdfaea5762e9789763e72bc5a0f6aac3302d539ee7f1c40d6"} Nov 02 06:50:30.048776 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:30.048747 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-29abe7-f28qb" Nov 02 06:50:30.169192 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:30.169156 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f8fbc11-84db-4998-9f70-acc2ea9ccef7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f8fbc11-84db-4998-9f70-acc2ea9ccef7-maas-api-openshift-ca\") pod \"0f8fbc11-84db-4998-9f70-acc2ea9ccef7\" (UID: \"0f8fbc11-84db-4998-9f70-acc2ea9ccef7\") " Nov 02 06:50:30.169374 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:30.169271 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f8fbc11-84db-4998-9f70-acc2ea9ccef7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f8fbc11-84db-4998-9f70-acc2ea9ccef7-maas-api-xks-ca\") pod \"0f8fbc11-84db-4998-9f70-acc2ea9ccef7\" (UID: \"0f8fbc11-84db-4998-9f70-acc2ea9ccef7\") " Nov 02 06:50:30.169450 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:30.169421 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f8fbc11-84db-4998-9f70-acc2ea9ccef7-kube-api-access-rdd6d\" (UniqueName: \"kubernetes.io/projected/0f8fbc11-84db-4998-9f70-acc2ea9ccef7-kube-api-access-rdd6d\") pod \"0f8fbc11-84db-4998-9f70-acc2ea9ccef7\" (UID: \"0f8fbc11-84db-4998-9f70-acc2ea9ccef7\") " Nov 02 06:50:30.169560 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:30.169534 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f8fbc11-84db-4998-9f70-acc2ea9ccef7-maas-api-openshift-ca" pod "0f8fbc11-84db-4998-9f70-acc2ea9ccef7" (UID: "0f8fbc11-84db-4998-9f70-acc2ea9ccef7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:50:30.169982 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:30.169960 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f8fbc11-84db-4998-9f70-acc2ea9ccef7-maas-api-openshift-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:50:30.171415 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:30.171388 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f8fbc11-84db-4998-9f70-acc2ea9ccef7-maas-api-xks-ca" pod "0f8fbc11-84db-4998-9f70-acc2ea9ccef7" (UID: "0f8fbc11-84db-4998-9f70-acc2ea9ccef7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:50:30.171532 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:30.171512 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f8fbc11-84db-4998-9f70-acc2ea9ccef7-kube-api-access-rdd6d" pod "0f8fbc11-84db-4998-9f70-acc2ea9ccef7" (UID: "0f8fbc11-84db-4998-9f70-acc2ea9ccef7"). InnerVolumeSpecName "kube-api-access-rdd6d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:50:30.271334 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:30.271301 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdd6d\" (UniqueName: \"kubernetes.io/projected/0f8fbc11-84db-4998-9f70-acc2ea9ccef7-kube-api-access-rdd6d\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:50:30.271334 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:30.271330 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f8fbc11-84db-4998-9f70-acc2ea9ccef7-maas-api-xks-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:50:30.889389 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:30.889357 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-29abe7-f28qb" Nov 02 06:50:30.889587 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:30.889386 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-29abe7-f28qb" event={"ID":"0f8fbc11-84db-4998-9f70-acc2ea9ccef7","Type":"ContainerDied","Data":"262df11d6a74d87945b5b89d3006b5bc4f227d74b572fc27913c8591145e29f4"} Nov 02 06:50:30.889587 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:30.889447 5916 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="262df11d6a74d87945b5b89d3006b5bc4f227d74b572fc27913c8591145e29f4" Nov 02 06:50:32.237433 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:32.237402 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-08bbf9-openshift-default-5479f6b7c-d2hk2"] Nov 02 06:50:32.237968 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:32.237653 5916 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-08bbf9-openshift-default-5479f6b7c-d2hk2" podUID="1b46fdb7-008c-45d7-92cb-8e392c54547f" containerName="istio-proxy" containerID="cri-o://99f1370b5029fee6fc8ee76ecc8c970f413ecd3fd0cda3c3accce9ca2f81f183" gracePeriod=30 Nov 02 06:50:32.980394 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:32.980354 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6d13c3-b7rlm"] Nov 02 06:50:32.981082 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:32.981053 5916 cpu_manager.go:414] "containerMap: removing container" podUID="0f8fbc11-84db-4998-9f70-acc2ea9ccef7" containerName="revoke-keys" Nov 02 06:50:32.981082 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:32.981071 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f8fbc11-84db-4998-9f70-acc2ea9ccef7" containerName="revoke-keys" Nov 02 06:50:32.981289 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:32.981253 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f8fbc11-84db-4998-9f70-acc2ea9ccef7" containerName="revoke-keys" Nov 02 06:50:32.985393 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:32.985374 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6d13c3-b7rlm" Nov 02 06:50:33.011917 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:33.011881 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5647086-76cc-485c-bff3-1c520fef97ff-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-6d13c3-b7rlm\" (UID: \"f5647086-76cc-485c-bff3-1c520fef97ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6d13c3-b7rlm" Nov 02 06:50:33.012170 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:33.012149 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5647086-76cc-485c-bff3-1c520fef97ff-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-6d13c3-b7rlm\" (UID: \"f5647086-76cc-485c-bff3-1c520fef97ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6d13c3-b7rlm" Nov 02 06:50:33.012246 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:33.012208 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pk6wl\" (UniqueName: \"kubernetes.io/projected/f5647086-76cc-485c-bff3-1c520fef97ff-kube-api-access-pk6wl\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-6d13c3-b7rlm\" (UID: \"f5647086-76cc-485c-bff3-1c520fef97ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6d13c3-b7rlm" Nov 02 06:50:33.042641 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:33.042607 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6d13c3-b7rlm"] Nov 02 06:50:33.086591 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:33.086554 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 06:50:33.086743 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:33.086646 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-68l4r\"" Nov 02 06:50:33.113555 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:33.113525 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5647086-76cc-485c-bff3-1c520fef97ff-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-6d13c3-b7rlm\" (UID: \"f5647086-76cc-485c-bff3-1c520fef97ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6d13c3-b7rlm" Nov 02 06:50:33.113727 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:33.113569 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pk6wl\" (UniqueName: \"kubernetes.io/projected/f5647086-76cc-485c-bff3-1c520fef97ff-kube-api-access-pk6wl\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-6d13c3-b7rlm\" (UID: \"f5647086-76cc-485c-bff3-1c520fef97ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6d13c3-b7rlm" Nov 02 06:50:33.113727 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:33.113639 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5647086-76cc-485c-bff3-1c520fef97ff-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-6d13c3-b7rlm\" (UID: \"f5647086-76cc-485c-bff3-1c520fef97ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6d13c3-b7rlm" Nov 02 06:50:33.114329 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:33.114308 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5647086-76cc-485c-bff3-1c520fef97ff-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-6d13c3-b7rlm\" (UID: \"f5647086-76cc-485c-bff3-1c520fef97ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6d13c3-b7rlm" Nov 02 06:50:33.115898 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:33.115875 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5647086-76cc-485c-bff3-1c520fef97ff-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-6d13c3-b7rlm\" (UID: \"f5647086-76cc-485c-bff3-1c520fef97ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6d13c3-b7rlm" Nov 02 06:50:33.171279 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:33.171248 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pk6wl\" (UniqueName: \"kubernetes.io/projected/f5647086-76cc-485c-bff3-1c520fef97ff-kube-api-access-pk6wl\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-6d13c3-b7rlm\" (UID: \"f5647086-76cc-485c-bff3-1c520fef97ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6d13c3-b7rlm" Nov 02 06:50:33.300619 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:33.300552 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6d13c3-b7rlm" Nov 02 06:50:33.684945 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:50:33.684914 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5647086_76cc_485c_bff3_1c520fef97ff.slice/crio-1e48e42877cc6536f4dba9a52d148c2546fa4f48801b681fc0f36f81d27ec2eb WatchSource:0}: Error finding container 1e48e42877cc6536f4dba9a52d148c2546fa4f48801b681fc0f36f81d27ec2eb: Status 404 returned error can't find the container with id 1e48e42877cc6536f4dba9a52d148c2546fa4f48801b681fc0f36f81d27ec2eb Nov 02 06:50:33.685149 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:33.685123 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6d13c3-b7rlm"] Nov 02 06:50:33.913717 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:33.913680 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6d13c3-b7rlm" event={"ID":"f5647086-76cc-485c-bff3-1c520fef97ff","Type":"ContainerStarted","Data":"1e48e42877cc6536f4dba9a52d148c2546fa4f48801b681fc0f36f81d27ec2eb"} Nov 02 06:50:34.922661 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:34.922626 5916 generic.go:356] "Generic (PLEG): container finished" podID="f5647086-76cc-485c-bff3-1c520fef97ff" containerID="a27965f3572e4513d105e04cd8f356867f4769ca651b831d1ac47067c26f1e48" exitCode=0 Nov 02 06:50:34.923176 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:34.922712 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6d13c3-b7rlm" event={"ID":"f5647086-76cc-485c-bff3-1c520fef97ff","Type":"ContainerDied","Data":"a27965f3572e4513d105e04cd8f356867f4769ca651b831d1ac47067c26f1e48"} Nov 02 06:50:36.087065 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:36.087039 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6d13c3-b7rlm" Nov 02 06:50:36.157420 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:36.157390 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f5647086-76cc-485c-bff3-1c520fef97ff-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5647086-76cc-485c-bff3-1c520fef97ff-maas-api-xks-ca\") pod \"f5647086-76cc-485c-bff3-1c520fef97ff\" (UID: \"f5647086-76cc-485c-bff3-1c520fef97ff\") " Nov 02 06:50:36.157578 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:36.157454 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5647086-76cc-485c-bff3-1c520fef97ff-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5647086-76cc-485c-bff3-1c520fef97ff-maas-api-openshift-ca\") pod \"f5647086-76cc-485c-bff3-1c520fef97ff\" (UID: \"f5647086-76cc-485c-bff3-1c520fef97ff\") " Nov 02 06:50:36.157797 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:36.157775 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5647086-76cc-485c-bff3-1c520fef97ff-maas-api-openshift-ca" pod "f5647086-76cc-485c-bff3-1c520fef97ff" (UID: "f5647086-76cc-485c-bff3-1c520fef97ff"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:50:36.157844 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:36.157783 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5647086-76cc-485c-bff3-1c520fef97ff-kube-api-access-pk6wl\" (UniqueName: \"kubernetes.io/projected/f5647086-76cc-485c-bff3-1c520fef97ff-kube-api-access-pk6wl\") pod \"f5647086-76cc-485c-bff3-1c520fef97ff\" (UID: \"f5647086-76cc-485c-bff3-1c520fef97ff\") " Nov 02 06:50:36.158343 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:36.158321 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5647086-76cc-485c-bff3-1c520fef97ff-maas-api-openshift-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:50:36.159677 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:36.159643 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5647086-76cc-485c-bff3-1c520fef97ff-maas-api-xks-ca" pod "f5647086-76cc-485c-bff3-1c520fef97ff" (UID: "f5647086-76cc-485c-bff3-1c520fef97ff"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:50:36.159781 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:36.159692 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5647086-76cc-485c-bff3-1c520fef97ff-kube-api-access-pk6wl" pod "f5647086-76cc-485c-bff3-1c520fef97ff" (UID: "f5647086-76cc-485c-bff3-1c520fef97ff"). InnerVolumeSpecName "kube-api-access-pk6wl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:50:36.260148 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:36.260072 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pk6wl\" (UniqueName: \"kubernetes.io/projected/f5647086-76cc-485c-bff3-1c520fef97ff-kube-api-access-pk6wl\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:50:36.260148 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:36.260101 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5647086-76cc-485c-bff3-1c520fef97ff-maas-api-xks-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:50:36.940058 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:36.940027 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6d13c3-b7rlm" Nov 02 06:50:36.940215 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:36.940022 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6d13c3-b7rlm" event={"ID":"f5647086-76cc-485c-bff3-1c520fef97ff","Type":"ContainerDied","Data":"1e48e42877cc6536f4dba9a52d148c2546fa4f48801b681fc0f36f81d27ec2eb"} Nov 02 06:50:36.940215 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:36.940156 5916 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1e48e42877cc6536f4dba9a52d148c2546fa4f48801b681fc0f36f81d27ec2eb" Nov 02 06:50:37.548695 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:37.548671 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-08bbf9-openshift-default-5479f6b7c-d2hk2" Nov 02 06:50:37.682746 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:37.682712 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b46fdb7-008c-45d7-92cb-8e392c54547f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b46fdb7-008c-45d7-92cb-8e392c54547f-workload-socket\") pod \"1b46fdb7-008c-45d7-92cb-8e392c54547f\" (UID: \"1b46fdb7-008c-45d7-92cb-8e392c54547f\") " Nov 02 06:50:37.682746 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:37.682747 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b46fdb7-008c-45d7-92cb-8e392c54547f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b46fdb7-008c-45d7-92cb-8e392c54547f-workload-certs\") pod \"1b46fdb7-008c-45d7-92cb-8e392c54547f\" (UID: \"1b46fdb7-008c-45d7-92cb-8e392c54547f\") " Nov 02 06:50:37.683026 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:37.682786 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b46fdb7-008c-45d7-92cb-8e392c54547f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b46fdb7-008c-45d7-92cb-8e392c54547f-istiod-ca-cert\") pod \"1b46fdb7-008c-45d7-92cb-8e392c54547f\" (UID: \"1b46fdb7-008c-45d7-92cb-8e392c54547f\") " Nov 02 06:50:37.683026 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:37.682871 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b46fdb7-008c-45d7-92cb-8e392c54547f-kube-api-access-mhgzk\" (UniqueName: \"kubernetes.io/projected/1b46fdb7-008c-45d7-92cb-8e392c54547f-kube-api-access-mhgzk\") pod \"1b46fdb7-008c-45d7-92cb-8e392c54547f\" (UID: \"1b46fdb7-008c-45d7-92cb-8e392c54547f\") " Nov 02 06:50:37.683026 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:37.682915 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b46fdb7-008c-45d7-92cb-8e392c54547f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b46fdb7-008c-45d7-92cb-8e392c54547f-credential-socket\") pod \"1b46fdb7-008c-45d7-92cb-8e392c54547f\" (UID: \"1b46fdb7-008c-45d7-92cb-8e392c54547f\") " Nov 02 06:50:37.683026 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:37.682933 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1b46fdb7-008c-45d7-92cb-8e392c54547f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b46fdb7-008c-45d7-92cb-8e392c54547f-istio-podinfo\") pod \"1b46fdb7-008c-45d7-92cb-8e392c54547f\" (UID: \"1b46fdb7-008c-45d7-92cb-8e392c54547f\") " Nov 02 06:50:37.683026 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:37.683015 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b46fdb7-008c-45d7-92cb-8e392c54547f-workload-socket" pod "1b46fdb7-008c-45d7-92cb-8e392c54547f" (UID: "1b46fdb7-008c-45d7-92cb-8e392c54547f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:50:37.683304 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:37.683030 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b46fdb7-008c-45d7-92cb-8e392c54547f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b46fdb7-008c-45d7-92cb-8e392c54547f-istio-envoy\") pod \"1b46fdb7-008c-45d7-92cb-8e392c54547f\" (UID: \"1b46fdb7-008c-45d7-92cb-8e392c54547f\") " Nov 02 06:50:37.683304 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:37.683078 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b46fdb7-008c-45d7-92cb-8e392c54547f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b46fdb7-008c-45d7-92cb-8e392c54547f-istio-data\") pod \"1b46fdb7-008c-45d7-92cb-8e392c54547f\" (UID: \"1b46fdb7-008c-45d7-92cb-8e392c54547f\") " Nov 02 06:50:37.683304 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:37.683145 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b46fdb7-008c-45d7-92cb-8e392c54547f-istio-token\" (UniqueName: \"kubernetes.io/projected/1b46fdb7-008c-45d7-92cb-8e392c54547f-istio-token\") pod \"1b46fdb7-008c-45d7-92cb-8e392c54547f\" (UID: \"1b46fdb7-008c-45d7-92cb-8e392c54547f\") " Nov 02 06:50:37.683304 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:37.683160 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b46fdb7-008c-45d7-92cb-8e392c54547f-workload-certs" pod "1b46fdb7-008c-45d7-92cb-8e392c54547f" (UID: "1b46fdb7-008c-45d7-92cb-8e392c54547f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:50:37.683304 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:37.683222 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b46fdb7-008c-45d7-92cb-8e392c54547f-credential-socket" pod "1b46fdb7-008c-45d7-92cb-8e392c54547f" (UID: "1b46fdb7-008c-45d7-92cb-8e392c54547f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:50:37.683545 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:37.683402 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b46fdb7-008c-45d7-92cb-8e392c54547f-istio-data" pod "1b46fdb7-008c-45d7-92cb-8e392c54547f" (UID: "1b46fdb7-008c-45d7-92cb-8e392c54547f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:50:37.683545 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:37.683428 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b46fdb7-008c-45d7-92cb-8e392c54547f-istiod-ca-cert" pod "1b46fdb7-008c-45d7-92cb-8e392c54547f" (UID: "1b46fdb7-008c-45d7-92cb-8e392c54547f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:50:37.683789 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:37.683763 5916 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b46fdb7-008c-45d7-92cb-8e392c54547f-credential-socket\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:50:37.683789 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:37.683786 5916 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b46fdb7-008c-45d7-92cb-8e392c54547f-istio-data\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:50:37.684069 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:37.683796 5916 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b46fdb7-008c-45d7-92cb-8e392c54547f-workload-socket\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:50:37.684069 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:37.683805 5916 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b46fdb7-008c-45d7-92cb-8e392c54547f-workload-certs\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:50:37.684069 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:37.683814 5916 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b46fdb7-008c-45d7-92cb-8e392c54547f-istiod-ca-cert\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:50:37.685271 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:37.685240 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b46fdb7-008c-45d7-92cb-8e392c54547f-kube-api-access-mhgzk" pod "1b46fdb7-008c-45d7-92cb-8e392c54547f" (UID: "1b46fdb7-008c-45d7-92cb-8e392c54547f"). InnerVolumeSpecName "kube-api-access-mhgzk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:50:37.685372 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:37.685320 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b46fdb7-008c-45d7-92cb-8e392c54547f-istio-envoy" pod "1b46fdb7-008c-45d7-92cb-8e392c54547f" (UID: "1b46fdb7-008c-45d7-92cb-8e392c54547f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:50:37.685521 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:37.685507 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b46fdb7-008c-45d7-92cb-8e392c54547f-istio-token" pod "1b46fdb7-008c-45d7-92cb-8e392c54547f" (UID: "1b46fdb7-008c-45d7-92cb-8e392c54547f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:50:37.685569 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:37.685514 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1b46fdb7-008c-45d7-92cb-8e392c54547f-istio-podinfo" pod "1b46fdb7-008c-45d7-92cb-8e392c54547f" (UID: "1b46fdb7-008c-45d7-92cb-8e392c54547f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 06:50:37.785550 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:37.785469 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhgzk\" (UniqueName: \"kubernetes.io/projected/1b46fdb7-008c-45d7-92cb-8e392c54547f-kube-api-access-mhgzk\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:50:37.785550 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:37.785496 5916 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b46fdb7-008c-45d7-92cb-8e392c54547f-istio-podinfo\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:50:37.785550 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:37.785509 5916 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b46fdb7-008c-45d7-92cb-8e392c54547f-istio-envoy\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:50:37.785550 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:37.785517 5916 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b46fdb7-008c-45d7-92cb-8e392c54547f-istio-token\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:50:37.951796 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:37.951762 5916 generic.go:356] "Generic (PLEG): container finished" podID="1b46fdb7-008c-45d7-92cb-8e392c54547f" containerID="99f1370b5029fee6fc8ee76ecc8c970f413ecd3fd0cda3c3accce9ca2f81f183" exitCode=0 Nov 02 06:50:37.952003 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:37.951810 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-08bbf9-openshift-default-5479f6b7c-d2hk2" Nov 02 06:50:37.952003 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:37.951844 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-08bbf9-openshift-default-5479f6b7c-d2hk2" event={"ID":"1b46fdb7-008c-45d7-92cb-8e392c54547f","Type":"ContainerDied","Data":"99f1370b5029fee6fc8ee76ecc8c970f413ecd3fd0cda3c3accce9ca2f81f183"} Nov 02 06:50:37.952003 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:37.951901 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-08bbf9-openshift-default-5479f6b7c-d2hk2" event={"ID":"1b46fdb7-008c-45d7-92cb-8e392c54547f","Type":"ContainerDied","Data":"a3259bcc6ce1bd28e3ac14a8fed8ab0b4368c10b1f94ad750a5eb2339221ca61"} Nov 02 06:50:37.952003 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:37.951933 5916 scope.go:122] "RemoveContainer" containerID="99f1370b5029fee6fc8ee76ecc8c970f413ecd3fd0cda3c3accce9ca2f81f183" Nov 02 06:50:37.963101 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:37.963082 5916 scope.go:122] "RemoveContainer" containerID="ef905e16b3a5fd3ea80cf5a33b5275a7425d95738fb1e82d4afbd99a49491dfd" Nov 02 06:50:37.972621 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:37.972604 5916 scope.go:122] "RemoveContainer" containerID="99f1370b5029fee6fc8ee76ecc8c970f413ecd3fd0cda3c3accce9ca2f81f183" Nov 02 06:50:37.972883 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:50:37.972862 5916 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99f1370b5029fee6fc8ee76ecc8c970f413ecd3fd0cda3c3accce9ca2f81f183\": container with ID starting with 99f1370b5029fee6fc8ee76ecc8c970f413ecd3fd0cda3c3accce9ca2f81f183 not found: ID does not exist" containerID="99f1370b5029fee6fc8ee76ecc8c970f413ecd3fd0cda3c3accce9ca2f81f183" Nov 02 06:50:37.973004 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:37.972889 5916 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99f1370b5029fee6fc8ee76ecc8c970f413ecd3fd0cda3c3accce9ca2f81f183"} err="failed to get container status \"99f1370b5029fee6fc8ee76ecc8c970f413ecd3fd0cda3c3accce9ca2f81f183\": rpc error: code = NotFound desc = could not find container \"99f1370b5029fee6fc8ee76ecc8c970f413ecd3fd0cda3c3accce9ca2f81f183\": container with ID starting with 99f1370b5029fee6fc8ee76ecc8c970f413ecd3fd0cda3c3accce9ca2f81f183 not found: ID does not exist" Nov 02 06:50:37.973004 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:37.972970 5916 scope.go:122] "RemoveContainer" containerID="ef905e16b3a5fd3ea80cf5a33b5275a7425d95738fb1e82d4afbd99a49491dfd" Nov 02 06:50:37.973208 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:50:37.973193 5916 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef905e16b3a5fd3ea80cf5a33b5275a7425d95738fb1e82d4afbd99a49491dfd\": container with ID starting with ef905e16b3a5fd3ea80cf5a33b5275a7425d95738fb1e82d4afbd99a49491dfd not found: ID does not exist" containerID="ef905e16b3a5fd3ea80cf5a33b5275a7425d95738fb1e82d4afbd99a49491dfd" Nov 02 06:50:37.973247 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:37.973211 5916 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef905e16b3a5fd3ea80cf5a33b5275a7425d95738fb1e82d4afbd99a49491dfd"} err="failed to get container status \"ef905e16b3a5fd3ea80cf5a33b5275a7425d95738fb1e82d4afbd99a49491dfd\": rpc error: code = NotFound desc = could not find container \"ef905e16b3a5fd3ea80cf5a33b5275a7425d95738fb1e82d4afbd99a49491dfd\": container with ID starting with ef905e16b3a5fd3ea80cf5a33b5275a7425d95738fb1e82d4afbd99a49491dfd not found: ID does not exist" Nov 02 06:50:38.119546 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:38.119467 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-08bbf9-openshift-default-5479f6b7c-d2hk2"] Nov 02 06:50:38.123109 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:38.123083 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-08bbf9-openshift-default-5479f6b7c-d2hk2"] Nov 02 06:50:39.844415 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:50:39.844377 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b46fdb7-008c-45d7-92cb-8e392c54547f" path="/var/lib/kubelet/pods/1b46fdb7-008c-45d7-92cb-8e392c54547f/volumes" Nov 02 06:50:49.901145 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:50:49.901100 5916 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p98cq" containerName="extractor" Nov 02 06:51:02.299178 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:02.299136 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3bf02ebf-vdjx8"] Nov 02 06:51:02.299722 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:02.299709 5916 cpu_manager.go:414] "containerMap: removing container" podUID="1b46fdb7-008c-45d7-92cb-8e392c54547f" containerName="istio-proxy" Nov 02 06:51:02.299771 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:02.299723 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b46fdb7-008c-45d7-92cb-8e392c54547f" containerName="istio-proxy" Nov 02 06:51:02.299771 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:02.299732 5916 cpu_manager.go:414] "containerMap: removing container" podUID="1b46fdb7-008c-45d7-92cb-8e392c54547f" containerName="istio-proxy" Nov 02 06:51:02.299771 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:02.299739 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b46fdb7-008c-45d7-92cb-8e392c54547f" containerName="istio-proxy" Nov 02 06:51:02.299865 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:02.299786 5916 cpu_manager.go:414] "containerMap: removing container" podUID="f5647086-76cc-485c-bff3-1c520fef97ff" containerName="revoke-keys" Nov 02 06:51:02.299865 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:02.299791 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5647086-76cc-485c-bff3-1c520fef97ff" containerName="revoke-keys" Nov 02 06:51:02.299948 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:02.299876 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5647086-76cc-485c-bff3-1c520fef97ff" containerName="revoke-keys" Nov 02 06:51:02.299948 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:02.299886 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b46fdb7-008c-45d7-92cb-8e392c54547f" containerName="istio-proxy" Nov 02 06:51:02.299948 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:02.299892 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b46fdb7-008c-45d7-92cb-8e392c54547f" containerName="istio-proxy" Nov 02 06:51:02.305309 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:02.305291 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3bf02ebf-vdjx8" Nov 02 06:51:02.361023 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:02.360991 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3bf02ebf-vdjx8"] Nov 02 06:51:02.376944 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:02.372715 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwn26\" (UniqueName: \"kubernetes.io/projected/14f16f4c-56ba-4dbe-bda0-912a4e67dffc-kube-api-access-bwn26\") pod \"maas-api-revoke-keys-e2e-mt-3bf02ebf-vdjx8\" (UID: \"14f16f4c-56ba-4dbe-bda0-912a4e67dffc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3bf02ebf-vdjx8" Nov 02 06:51:02.376944 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:02.372794 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14f16f4c-56ba-4dbe-bda0-912a4e67dffc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-3bf02ebf-vdjx8\" (UID: \"14f16f4c-56ba-4dbe-bda0-912a4e67dffc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3bf02ebf-vdjx8" Nov 02 06:51:02.376944 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:02.373030 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14f16f4c-56ba-4dbe-bda0-912a4e67dffc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-3bf02ebf-vdjx8\" (UID: \"14f16f4c-56ba-4dbe-bda0-912a4e67dffc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3bf02ebf-vdjx8" Nov 02 06:51:02.405981 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:02.405954 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 06:51:02.406149 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:02.406019 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-68l4r\"" Nov 02 06:51:02.474027 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:02.473988 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwn26\" (UniqueName: \"kubernetes.io/projected/14f16f4c-56ba-4dbe-bda0-912a4e67dffc-kube-api-access-bwn26\") pod \"maas-api-revoke-keys-e2e-mt-3bf02ebf-vdjx8\" (UID: \"14f16f4c-56ba-4dbe-bda0-912a4e67dffc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3bf02ebf-vdjx8" Nov 02 06:51:02.474222 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:02.474082 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14f16f4c-56ba-4dbe-bda0-912a4e67dffc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-3bf02ebf-vdjx8\" (UID: \"14f16f4c-56ba-4dbe-bda0-912a4e67dffc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3bf02ebf-vdjx8" Nov 02 06:51:02.474304 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:02.474282 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14f16f4c-56ba-4dbe-bda0-912a4e67dffc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-3bf02ebf-vdjx8\" (UID: \"14f16f4c-56ba-4dbe-bda0-912a4e67dffc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3bf02ebf-vdjx8" Nov 02 06:51:02.475079 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:02.475055 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14f16f4c-56ba-4dbe-bda0-912a4e67dffc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-3bf02ebf-vdjx8\" (UID: \"14f16f4c-56ba-4dbe-bda0-912a4e67dffc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3bf02ebf-vdjx8" Nov 02 06:51:02.476979 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:02.476950 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14f16f4c-56ba-4dbe-bda0-912a4e67dffc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-3bf02ebf-vdjx8\" (UID: \"14f16f4c-56ba-4dbe-bda0-912a4e67dffc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3bf02ebf-vdjx8" Nov 02 06:51:02.539754 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:02.539724 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwn26\" (UniqueName: \"kubernetes.io/projected/14f16f4c-56ba-4dbe-bda0-912a4e67dffc-kube-api-access-bwn26\") pod \"maas-api-revoke-keys-e2e-mt-3bf02ebf-vdjx8\" (UID: \"14f16f4c-56ba-4dbe-bda0-912a4e67dffc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3bf02ebf-vdjx8" Nov 02 06:51:02.619084 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:02.619006 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3bf02ebf-vdjx8" Nov 02 06:51:03.012500 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:03.012474 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3bf02ebf-vdjx8"] Nov 02 06:51:03.013607 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:51:03.013575 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14f16f4c_56ba_4dbe_bda0_912a4e67dffc.slice/crio-aa3ce4f42e8c9d759bbcaa9f4bc76815f507f3d9b860a47db0917093e2c45422 WatchSource:0}: Error finding container aa3ce4f42e8c9d759bbcaa9f4bc76815f507f3d9b860a47db0917093e2c45422: Status 404 returned error can't find the container with id aa3ce4f42e8c9d759bbcaa9f4bc76815f507f3d9b860a47db0917093e2c45422 Nov 02 06:51:03.172060 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:03.172023 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3bf02ebf-vdjx8" event={"ID":"14f16f4c-56ba-4dbe-bda0-912a4e67dffc","Type":"ContainerStarted","Data":"aa3ce4f42e8c9d759bbcaa9f4bc76815f507f3d9b860a47db0917093e2c45422"} Nov 02 06:51:04.182166 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:04.182063 5916 generic.go:356] "Generic (PLEG): container finished" podID="14f16f4c-56ba-4dbe-bda0-912a4e67dffc" containerID="09728591c0eedbce9a14c79e9217b32ab93838d246e60ee190740f2223c5f4de" exitCode=0 Nov 02 06:51:04.182562 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:04.182159 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3bf02ebf-vdjx8" event={"ID":"14f16f4c-56ba-4dbe-bda0-912a4e67dffc","Type":"ContainerDied","Data":"09728591c0eedbce9a14c79e9217b32ab93838d246e60ee190740f2223c5f4de"} Nov 02 06:51:05.365599 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:05.365569 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3bf02ebf-vdjx8" Nov 02 06:51:05.421951 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:05.421890 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14f16f4c-56ba-4dbe-bda0-912a4e67dffc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14f16f4c-56ba-4dbe-bda0-912a4e67dffc-maas-api-openshift-ca\") pod \"14f16f4c-56ba-4dbe-bda0-912a4e67dffc\" (UID: \"14f16f4c-56ba-4dbe-bda0-912a4e67dffc\") " Nov 02 06:51:05.422127 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:05.422114 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/14f16f4c-56ba-4dbe-bda0-912a4e67dffc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14f16f4c-56ba-4dbe-bda0-912a4e67dffc-maas-api-xks-ca\") pod \"14f16f4c-56ba-4dbe-bda0-912a4e67dffc\" (UID: \"14f16f4c-56ba-4dbe-bda0-912a4e67dffc\") " Nov 02 06:51:05.422197 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:05.422185 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14f16f4c-56ba-4dbe-bda0-912a4e67dffc-kube-api-access-bwn26\" (UniqueName: \"kubernetes.io/projected/14f16f4c-56ba-4dbe-bda0-912a4e67dffc-kube-api-access-bwn26\") pod \"14f16f4c-56ba-4dbe-bda0-912a4e67dffc\" (UID: \"14f16f4c-56ba-4dbe-bda0-912a4e67dffc\") " Nov 02 06:51:05.422271 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:05.422245 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14f16f4c-56ba-4dbe-bda0-912a4e67dffc-maas-api-openshift-ca" pod "14f16f4c-56ba-4dbe-bda0-912a4e67dffc" (UID: "14f16f4c-56ba-4dbe-bda0-912a4e67dffc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:51:05.422521 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:05.422505 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14f16f4c-56ba-4dbe-bda0-912a4e67dffc-maas-api-openshift-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:51:05.424127 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:05.424104 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14f16f4c-56ba-4dbe-bda0-912a4e67dffc-maas-api-xks-ca" pod "14f16f4c-56ba-4dbe-bda0-912a4e67dffc" (UID: "14f16f4c-56ba-4dbe-bda0-912a4e67dffc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:51:05.424243 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:05.424227 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14f16f4c-56ba-4dbe-bda0-912a4e67dffc-kube-api-access-bwn26" pod "14f16f4c-56ba-4dbe-bda0-912a4e67dffc" (UID: "14f16f4c-56ba-4dbe-bda0-912a4e67dffc"). InnerVolumeSpecName "kube-api-access-bwn26". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:51:05.524474 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:05.524397 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwn26\" (UniqueName: \"kubernetes.io/projected/14f16f4c-56ba-4dbe-bda0-912a4e67dffc-kube-api-access-bwn26\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:51:05.524474 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:05.524425 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14f16f4c-56ba-4dbe-bda0-912a4e67dffc-maas-api-xks-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:51:06.200091 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:06.200057 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3bf02ebf-vdjx8" event={"ID":"14f16f4c-56ba-4dbe-bda0-912a4e67dffc","Type":"ContainerDied","Data":"aa3ce4f42e8c9d759bbcaa9f4bc76815f507f3d9b860a47db0917093e2c45422"} Nov 02 06:51:06.200280 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:06.200096 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3bf02ebf-vdjx8" Nov 02 06:51:06.200280 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:06.200128 5916 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aa3ce4f42e8c9d759bbcaa9f4bc76815f507f3d9b860a47db0917093e2c45422" Nov 02 06:51:07.557803 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:07.557763 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-p5j6w"] Nov 02 06:51:07.558308 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:07.558018 5916 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-p5j6w" podUID="835047d2-d7ce-4dcf-95a4-449fb987d4a7" containerName="istio-proxy" containerID="cri-o://e695b9e0fa78af668e201eefa11352870bb9c707c017064de549208bdb5e7388" gracePeriod=30 Nov 02 06:51:07.862108 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:07.862020 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-wzvj7"] Nov 02 06:51:07.862329 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:07.862201 5916 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-wzvj7" podUID="beb17737-f6a0-47e1-8524-e56e8d267502" containerName="istio-proxy" containerID="cri-o://91af1497b1fbf0b6bfd3480afccc099dfba64d005ed943094029093396c994fa" gracePeriod=30 Nov 02 06:51:10.384038 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:10.384004 5916 patch_prober.go:29] interesting pod/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-wzvj7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 06:51:10.384422 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:10.384058 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-wzvj7" podUID="beb17737-f6a0-47e1-8524-e56e8d267502" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 06:51:10.517693 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:10.517661 5916 patch_prober.go:29] interesting pod/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-p5j6w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 06:51:10.517877 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:10.517721 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-p5j6w" podUID="835047d2-d7ce-4dcf-95a4-449fb987d4a7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 06:51:12.833029 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:12.832948 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-p5j6w" Nov 02 06:51:12.919125 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:12.919093 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/835047d2-d7ce-4dcf-95a4-449fb987d4a7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/835047d2-d7ce-4dcf-95a4-449fb987d4a7-workload-socket\") pod \"835047d2-d7ce-4dcf-95a4-449fb987d4a7\" (UID: \"835047d2-d7ce-4dcf-95a4-449fb987d4a7\") " Nov 02 06:51:12.919396 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:12.919142 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/835047d2-d7ce-4dcf-95a4-449fb987d4a7-istio-token\" (UniqueName: \"kubernetes.io/projected/835047d2-d7ce-4dcf-95a4-449fb987d4a7-istio-token\") pod \"835047d2-d7ce-4dcf-95a4-449fb987d4a7\" (UID: \"835047d2-d7ce-4dcf-95a4-449fb987d4a7\") " Nov 02 06:51:12.919396 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:12.919208 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/835047d2-d7ce-4dcf-95a4-449fb987d4a7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/835047d2-d7ce-4dcf-95a4-449fb987d4a7-credential-socket\") pod \"835047d2-d7ce-4dcf-95a4-449fb987d4a7\" (UID: \"835047d2-d7ce-4dcf-95a4-449fb987d4a7\") " Nov 02 06:51:12.919396 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:12.919236 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/835047d2-d7ce-4dcf-95a4-449fb987d4a7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/835047d2-d7ce-4dcf-95a4-449fb987d4a7-istio-podinfo\") pod \"835047d2-d7ce-4dcf-95a4-449fb987d4a7\" (UID: \"835047d2-d7ce-4dcf-95a4-449fb987d4a7\") " Nov 02 06:51:12.919396 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:12.919256 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/835047d2-d7ce-4dcf-95a4-449fb987d4a7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/835047d2-d7ce-4dcf-95a4-449fb987d4a7-istio-envoy\") pod \"835047d2-d7ce-4dcf-95a4-449fb987d4a7\" (UID: \"835047d2-d7ce-4dcf-95a4-449fb987d4a7\") " Nov 02 06:51:12.919396 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:12.919284 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/835047d2-d7ce-4dcf-95a4-449fb987d4a7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/835047d2-d7ce-4dcf-95a4-449fb987d4a7-istiod-ca-cert\") pod \"835047d2-d7ce-4dcf-95a4-449fb987d4a7\" (UID: \"835047d2-d7ce-4dcf-95a4-449fb987d4a7\") " Nov 02 06:51:12.919700 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:12.919402 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/835047d2-d7ce-4dcf-95a4-449fb987d4a7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/835047d2-d7ce-4dcf-95a4-449fb987d4a7-workload-certs\") pod \"835047d2-d7ce-4dcf-95a4-449fb987d4a7\" (UID: \"835047d2-d7ce-4dcf-95a4-449fb987d4a7\") " Nov 02 06:51:12.919700 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:12.919442 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/835047d2-d7ce-4dcf-95a4-449fb987d4a7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/835047d2-d7ce-4dcf-95a4-449fb987d4a7-istio-data\") pod \"835047d2-d7ce-4dcf-95a4-449fb987d4a7\" (UID: \"835047d2-d7ce-4dcf-95a4-449fb987d4a7\") " Nov 02 06:51:12.919700 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:12.919477 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/835047d2-d7ce-4dcf-95a4-449fb987d4a7-kube-api-access-czgnt\" (UniqueName: \"kubernetes.io/projected/835047d2-d7ce-4dcf-95a4-449fb987d4a7-kube-api-access-czgnt\") pod \"835047d2-d7ce-4dcf-95a4-449fb987d4a7\" (UID: \"835047d2-d7ce-4dcf-95a4-449fb987d4a7\") " Nov 02 06:51:12.919700 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:12.919485 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/835047d2-d7ce-4dcf-95a4-449fb987d4a7-workload-socket" pod "835047d2-d7ce-4dcf-95a4-449fb987d4a7" (UID: "835047d2-d7ce-4dcf-95a4-449fb987d4a7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:51:12.919700 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:12.919556 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/835047d2-d7ce-4dcf-95a4-449fb987d4a7-credential-socket" pod "835047d2-d7ce-4dcf-95a4-449fb987d4a7" (UID: "835047d2-d7ce-4dcf-95a4-449fb987d4a7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:51:12.919700 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:12.919674 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/835047d2-d7ce-4dcf-95a4-449fb987d4a7-workload-certs" pod "835047d2-d7ce-4dcf-95a4-449fb987d4a7" (UID: "835047d2-d7ce-4dcf-95a4-449fb987d4a7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:51:12.920035 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:12.919704 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/835047d2-d7ce-4dcf-95a4-449fb987d4a7-istio-data" pod "835047d2-d7ce-4dcf-95a4-449fb987d4a7" (UID: "835047d2-d7ce-4dcf-95a4-449fb987d4a7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:51:12.920680 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:12.920420 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/835047d2-d7ce-4dcf-95a4-449fb987d4a7-istiod-ca-cert" pod "835047d2-d7ce-4dcf-95a4-449fb987d4a7" (UID: "835047d2-d7ce-4dcf-95a4-449fb987d4a7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:51:12.920680 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:12.920589 5916 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/835047d2-d7ce-4dcf-95a4-449fb987d4a7-credential-socket\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:51:12.920680 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:12.920610 5916 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/835047d2-d7ce-4dcf-95a4-449fb987d4a7-istiod-ca-cert\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:51:12.920680 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:12.920624 5916 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/835047d2-d7ce-4dcf-95a4-449fb987d4a7-workload-certs\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:51:12.920680 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:12.920640 5916 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/835047d2-d7ce-4dcf-95a4-449fb987d4a7-istio-data\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:51:12.920680 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:12.920652 5916 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/835047d2-d7ce-4dcf-95a4-449fb987d4a7-workload-socket\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:51:12.921858 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:12.921831 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/835047d2-d7ce-4dcf-95a4-449fb987d4a7-kube-api-access-czgnt" pod "835047d2-d7ce-4dcf-95a4-449fb987d4a7" (UID: "835047d2-d7ce-4dcf-95a4-449fb987d4a7"). InnerVolumeSpecName "kube-api-access-czgnt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:51:12.922065 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:12.922044 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/835047d2-d7ce-4dcf-95a4-449fb987d4a7-istio-podinfo" pod "835047d2-d7ce-4dcf-95a4-449fb987d4a7" (UID: "835047d2-d7ce-4dcf-95a4-449fb987d4a7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 06:51:12.922147 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:12.922041 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/835047d2-d7ce-4dcf-95a4-449fb987d4a7-istio-envoy" pod "835047d2-d7ce-4dcf-95a4-449fb987d4a7" (UID: "835047d2-d7ce-4dcf-95a4-449fb987d4a7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:51:12.922593 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:12.922560 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/835047d2-d7ce-4dcf-95a4-449fb987d4a7-istio-token" pod "835047d2-d7ce-4dcf-95a4-449fb987d4a7" (UID: "835047d2-d7ce-4dcf-95a4-449fb987d4a7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:51:13.023755 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:13.023718 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czgnt\" (UniqueName: \"kubernetes.io/projected/835047d2-d7ce-4dcf-95a4-449fb987d4a7-kube-api-access-czgnt\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:51:13.023755 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:13.023754 5916 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/835047d2-d7ce-4dcf-95a4-449fb987d4a7-istio-token\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:51:13.024064 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:13.023767 5916 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/835047d2-d7ce-4dcf-95a4-449fb987d4a7-istio-podinfo\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:51:13.024064 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:13.023781 5916 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/835047d2-d7ce-4dcf-95a4-449fb987d4a7-istio-envoy\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:51:13.154711 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:13.154684 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-wzvj7" Nov 02 06:51:13.225817 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:13.225788 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/beb17737-f6a0-47e1-8524-e56e8d267502-kube-api-access-hf4j6\" (UniqueName: \"kubernetes.io/projected/beb17737-f6a0-47e1-8524-e56e8d267502-kube-api-access-hf4j6\") pod \"beb17737-f6a0-47e1-8524-e56e8d267502\" (UID: \"beb17737-f6a0-47e1-8524-e56e8d267502\") " Nov 02 06:51:13.226023 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:13.225838 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/beb17737-f6a0-47e1-8524-e56e8d267502-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/beb17737-f6a0-47e1-8524-e56e8d267502-workload-certs\") pod \"beb17737-f6a0-47e1-8524-e56e8d267502\" (UID: \"beb17737-f6a0-47e1-8524-e56e8d267502\") " Nov 02 06:51:13.226023 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:13.226007 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/beb17737-f6a0-47e1-8524-e56e8d267502-istio-token\" (UniqueName: \"kubernetes.io/projected/beb17737-f6a0-47e1-8524-e56e8d267502-istio-token\") pod \"beb17737-f6a0-47e1-8524-e56e8d267502\" (UID: \"beb17737-f6a0-47e1-8524-e56e8d267502\") " Nov 02 06:51:13.226136 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:13.226062 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/beb17737-f6a0-47e1-8524-e56e8d267502-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/beb17737-f6a0-47e1-8524-e56e8d267502-istiod-ca-cert\") pod \"beb17737-f6a0-47e1-8524-e56e8d267502\" (UID: \"beb17737-f6a0-47e1-8524-e56e8d267502\") " Nov 02 06:51:13.226196 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:13.226171 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/beb17737-f6a0-47e1-8524-e56e8d267502-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/beb17737-f6a0-47e1-8524-e56e8d267502-workload-socket\") pod \"beb17737-f6a0-47e1-8524-e56e8d267502\" (UID: \"beb17737-f6a0-47e1-8524-e56e8d267502\") " Nov 02 06:51:13.226196 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:13.226180 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/beb17737-f6a0-47e1-8524-e56e8d267502-workload-certs" pod "beb17737-f6a0-47e1-8524-e56e8d267502" (UID: "beb17737-f6a0-47e1-8524-e56e8d267502"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:51:13.226301 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:13.226223 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/beb17737-f6a0-47e1-8524-e56e8d267502-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/beb17737-f6a0-47e1-8524-e56e8d267502-istio-data\") pod \"beb17737-f6a0-47e1-8524-e56e8d267502\" (UID: \"beb17737-f6a0-47e1-8524-e56e8d267502\") " Nov 02 06:51:13.226301 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:13.226262 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/beb17737-f6a0-47e1-8524-e56e8d267502-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/beb17737-f6a0-47e1-8524-e56e8d267502-istio-podinfo\") pod \"beb17737-f6a0-47e1-8524-e56e8d267502\" (UID: \"beb17737-f6a0-47e1-8524-e56e8d267502\") " Nov 02 06:51:13.226410 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:13.226350 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/beb17737-f6a0-47e1-8524-e56e8d267502-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/beb17737-f6a0-47e1-8524-e56e8d267502-credential-socket\") pod \"beb17737-f6a0-47e1-8524-e56e8d267502\" (UID: \"beb17737-f6a0-47e1-8524-e56e8d267502\") " Nov 02 06:51:13.226466 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:13.226386 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/beb17737-f6a0-47e1-8524-e56e8d267502-workload-socket" pod "beb17737-f6a0-47e1-8524-e56e8d267502" (UID: "beb17737-f6a0-47e1-8524-e56e8d267502"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:51:13.226466 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:13.226409 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/beb17737-f6a0-47e1-8524-e56e8d267502-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/beb17737-f6a0-47e1-8524-e56e8d267502-istio-envoy\") pod \"beb17737-f6a0-47e1-8524-e56e8d267502\" (UID: \"beb17737-f6a0-47e1-8524-e56e8d267502\") " Nov 02 06:51:13.226466 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:13.226412 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/beb17737-f6a0-47e1-8524-e56e8d267502-istiod-ca-cert" pod "beb17737-f6a0-47e1-8524-e56e8d267502" (UID: "beb17737-f6a0-47e1-8524-e56e8d267502"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:51:13.226466 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:13.226452 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/beb17737-f6a0-47e1-8524-e56e8d267502-istio-data" pod "beb17737-f6a0-47e1-8524-e56e8d267502" (UID: "beb17737-f6a0-47e1-8524-e56e8d267502"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:51:13.226686 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:13.226626 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/beb17737-f6a0-47e1-8524-e56e8d267502-credential-socket" pod "beb17737-f6a0-47e1-8524-e56e8d267502" (UID: "beb17737-f6a0-47e1-8524-e56e8d267502"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:51:13.227076 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:13.227010 5916 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/beb17737-f6a0-47e1-8524-e56e8d267502-workload-certs\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:51:13.227076 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:13.227038 5916 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/beb17737-f6a0-47e1-8524-e56e8d267502-istiod-ca-cert\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:51:13.227076 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:13.227053 5916 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/beb17737-f6a0-47e1-8524-e56e8d267502-workload-socket\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:51:13.227076 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:13.227067 5916 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/beb17737-f6a0-47e1-8524-e56e8d267502-istio-data\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:51:13.227354 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:13.227081 5916 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/beb17737-f6a0-47e1-8524-e56e8d267502-credential-socket\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:51:13.228300 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:13.228271 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/beb17737-f6a0-47e1-8524-e56e8d267502-kube-api-access-hf4j6" pod "beb17737-f6a0-47e1-8524-e56e8d267502" (UID: "beb17737-f6a0-47e1-8524-e56e8d267502"). InnerVolumeSpecName "kube-api-access-hf4j6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:51:13.228526 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:13.228499 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/beb17737-f6a0-47e1-8524-e56e8d267502-istio-token" pod "beb17737-f6a0-47e1-8524-e56e8d267502" (UID: "beb17737-f6a0-47e1-8524-e56e8d267502"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:51:13.228867 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:13.228844 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/beb17737-f6a0-47e1-8524-e56e8d267502-istio-podinfo" pod "beb17737-f6a0-47e1-8524-e56e8d267502" (UID: "beb17737-f6a0-47e1-8524-e56e8d267502"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 06:51:13.228976 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:13.228849 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/beb17737-f6a0-47e1-8524-e56e8d267502-istio-envoy" pod "beb17737-f6a0-47e1-8524-e56e8d267502" (UID: "beb17737-f6a0-47e1-8524-e56e8d267502"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:51:13.254623 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:13.254598 5916 generic.go:356] "Generic (PLEG): container finished" podID="835047d2-d7ce-4dcf-95a4-449fb987d4a7" containerID="e695b9e0fa78af668e201eefa11352870bb9c707c017064de549208bdb5e7388" exitCode=0 Nov 02 06:51:13.254726 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:13.254641 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-p5j6w" Nov 02 06:51:13.254726 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:13.254688 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-p5j6w" event={"ID":"835047d2-d7ce-4dcf-95a4-449fb987d4a7","Type":"ContainerDied","Data":"e695b9e0fa78af668e201eefa11352870bb9c707c017064de549208bdb5e7388"} Nov 02 06:51:13.255147 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:13.254751 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-p5j6w" event={"ID":"835047d2-d7ce-4dcf-95a4-449fb987d4a7","Type":"ContainerDied","Data":"4b5d6fd272be8c2610d4284a2975d3239ac2b24e3f0e99a8ef96a8fffdcd7905"} Nov 02 06:51:13.255147 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:13.254782 5916 scope.go:122] "RemoveContainer" containerID="e695b9e0fa78af668e201eefa11352870bb9c707c017064de549208bdb5e7388" Nov 02 06:51:13.256377 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:13.256354 5916 generic.go:356] "Generic (PLEG): container finished" podID="beb17737-f6a0-47e1-8524-e56e8d267502" containerID="91af1497b1fbf0b6bfd3480afccc099dfba64d005ed943094029093396c994fa" exitCode=0 Nov 02 06:51:13.256480 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:13.256409 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-wzvj7" Nov 02 06:51:13.256480 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:13.256424 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-wzvj7" event={"ID":"beb17737-f6a0-47e1-8524-e56e8d267502","Type":"ContainerDied","Data":"91af1497b1fbf0b6bfd3480afccc099dfba64d005ed943094029093396c994fa"} Nov 02 06:51:13.256568 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:13.256480 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-wzvj7" event={"ID":"beb17737-f6a0-47e1-8524-e56e8d267502","Type":"ContainerDied","Data":"6bc35fe3f41b0ce11a082cec77ef70832cf96aebf162526c3b092ad49e2805dc"} Nov 02 06:51:13.267701 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:13.267681 5916 scope.go:122] "RemoveContainer" containerID="e695b9e0fa78af668e201eefa11352870bb9c707c017064de549208bdb5e7388" Nov 02 06:51:13.268013 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:51:13.267994 5916 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e695b9e0fa78af668e201eefa11352870bb9c707c017064de549208bdb5e7388\": container with ID starting with e695b9e0fa78af668e201eefa11352870bb9c707c017064de549208bdb5e7388 not found: ID does not exist" containerID="e695b9e0fa78af668e201eefa11352870bb9c707c017064de549208bdb5e7388" Nov 02 06:51:13.268088 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:13.268025 5916 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e695b9e0fa78af668e201eefa11352870bb9c707c017064de549208bdb5e7388"} err="failed to get container status \"e695b9e0fa78af668e201eefa11352870bb9c707c017064de549208bdb5e7388\": rpc error: code = NotFound desc = could not find container \"e695b9e0fa78af668e201eefa11352870bb9c707c017064de549208bdb5e7388\": container with ID starting with e695b9e0fa78af668e201eefa11352870bb9c707c017064de549208bdb5e7388 not found: ID does not exist" Nov 02 06:51:13.268088 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:13.268049 5916 scope.go:122] "RemoveContainer" containerID="91af1497b1fbf0b6bfd3480afccc099dfba64d005ed943094029093396c994fa" Nov 02 06:51:13.281785 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:13.281767 5916 scope.go:122] "RemoveContainer" containerID="91af1497b1fbf0b6bfd3480afccc099dfba64d005ed943094029093396c994fa" Nov 02 06:51:13.282121 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:51:13.282100 5916 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91af1497b1fbf0b6bfd3480afccc099dfba64d005ed943094029093396c994fa\": container with ID starting with 91af1497b1fbf0b6bfd3480afccc099dfba64d005ed943094029093396c994fa not found: ID does not exist" containerID="91af1497b1fbf0b6bfd3480afccc099dfba64d005ed943094029093396c994fa" Nov 02 06:51:13.282201 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:13.282130 5916 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91af1497b1fbf0b6bfd3480afccc099dfba64d005ed943094029093396c994fa"} err="failed to get container status \"91af1497b1fbf0b6bfd3480afccc099dfba64d005ed943094029093396c994fa\": rpc error: code = NotFound desc = could not find container \"91af1497b1fbf0b6bfd3480afccc099dfba64d005ed943094029093396c994fa\": container with ID starting with 91af1497b1fbf0b6bfd3480afccc099dfba64d005ed943094029093396c994fa not found: ID does not exist" Nov 02 06:51:13.328079 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:13.328054 5916 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/beb17737-f6a0-47e1-8524-e56e8d267502-istio-podinfo\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:51:13.328079 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:13.328080 5916 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/beb17737-f6a0-47e1-8524-e56e8d267502-istio-envoy\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:51:13.328210 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:13.328090 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hf4j6\" (UniqueName: \"kubernetes.io/projected/beb17737-f6a0-47e1-8524-e56e8d267502-kube-api-access-hf4j6\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:51:13.328210 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:13.328098 5916 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/beb17737-f6a0-47e1-8524-e56e8d267502-istio-token\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:51:13.388647 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:13.388614 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-p5j6w"] Nov 02 06:51:13.393270 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:13.393244 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-p5j6w"] Nov 02 06:51:13.503566 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:13.503494 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-wzvj7"] Nov 02 06:51:13.509639 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:13.509611 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-wzvj7"] Nov 02 06:51:13.843894 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:13.843815 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="835047d2-d7ce-4dcf-95a4-449fb987d4a7" path="/var/lib/kubelet/pods/835047d2-d7ce-4dcf-95a4-449fb987d4a7/volumes" Nov 02 06:51:13.844326 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:13.844310 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="beb17737-f6a0-47e1-8524-e56e8d267502" path="/var/lib/kubelet/pods/beb17737-f6a0-47e1-8524-e56e8d267502/volumes" Nov 02 06:51:19.895507 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:51:19.895471 5916 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p98cq" containerName="extractor" Nov 02 06:51:21.053438 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:21.053399 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5fe8b848-ght6b"] Nov 02 06:51:21.102727 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:21.102694 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5fe8b848-ght6b"] Nov 02 06:51:21.845885 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:21.845852 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26d48294-0eb7-48c7-868c-73068ae0b883" path="/var/lib/kubelet/pods/26d48294-0eb7-48c7-868c-73068ae0b883/volumes" Nov 02 06:51:29.457044 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:29.457005 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8ecb23-rzfnk"] Nov 02 06:51:29.506021 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:29.505973 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8ecb23-rzfnk"] Nov 02 06:51:29.843811 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:29.843727 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb070200-cf13-49f5-94e4-20d0afdfd322" path="/var/lib/kubelet/pods/eb070200-cf13-49f5-94e4-20d0afdfd322/volumes" Nov 02 06:51:37.522975 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:37.522863 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-37b53e-gvlqd"] Nov 02 06:51:37.523710 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:37.523692 5916 cpu_manager.go:414] "containerMap: removing container" podUID="beb17737-f6a0-47e1-8524-e56e8d267502" containerName="istio-proxy" Nov 02 06:51:37.523815 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:37.523711 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="beb17737-f6a0-47e1-8524-e56e8d267502" containerName="istio-proxy" Nov 02 06:51:37.523815 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:37.523744 5916 cpu_manager.go:414] "containerMap: removing container" podUID="14f16f4c-56ba-4dbe-bda0-912a4e67dffc" containerName="revoke-keys" Nov 02 06:51:37.523815 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:37.523753 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14f16f4c-56ba-4dbe-bda0-912a4e67dffc" containerName="revoke-keys" Nov 02 06:51:37.523815 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:37.523770 5916 cpu_manager.go:414] "containerMap: removing container" podUID="835047d2-d7ce-4dcf-95a4-449fb987d4a7" containerName="istio-proxy" Nov 02 06:51:37.523815 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:37.523778 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="835047d2-d7ce-4dcf-95a4-449fb987d4a7" containerName="istio-proxy" Nov 02 06:51:37.524024 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:37.523958 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="beb17737-f6a0-47e1-8524-e56e8d267502" containerName="istio-proxy" Nov 02 06:51:37.524024 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:37.523972 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="14f16f4c-56ba-4dbe-bda0-912a4e67dffc" containerName="revoke-keys" Nov 02 06:51:37.524024 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:37.523981 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="835047d2-d7ce-4dcf-95a4-449fb987d4a7" containerName="istio-proxy" Nov 02 06:51:37.529776 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:37.529748 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-37b53e-gvlqd" Nov 02 06:51:37.582832 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:37.582801 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-37b53e-gvlqd"] Nov 02 06:51:37.600526 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:37.600488 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qshf\" (UniqueName: \"kubernetes.io/projected/71c05684-cf04-46f4-8321-a211172c839e-kube-api-access-6qshf\") pod \"maas-api-revoke-keys-e2e-api-b-37b53e-gvlqd\" (UID: \"71c05684-cf04-46f4-8321-a211172c839e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-37b53e-gvlqd" Nov 02 06:51:37.600690 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:37.600535 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71c05684-cf04-46f4-8321-a211172c839e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-37b53e-gvlqd\" (UID: \"71c05684-cf04-46f4-8321-a211172c839e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-37b53e-gvlqd" Nov 02 06:51:37.600690 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:37.600651 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71c05684-cf04-46f4-8321-a211172c839e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-37b53e-gvlqd\" (UID: \"71c05684-cf04-46f4-8321-a211172c839e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-37b53e-gvlqd" Nov 02 06:51:37.628065 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:37.628043 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 06:51:37.628243 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:37.628224 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-68l4r\"" Nov 02 06:51:37.702367 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:37.702337 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qshf\" (UniqueName: \"kubernetes.io/projected/71c05684-cf04-46f4-8321-a211172c839e-kube-api-access-6qshf\") pod \"maas-api-revoke-keys-e2e-api-b-37b53e-gvlqd\" (UID: \"71c05684-cf04-46f4-8321-a211172c839e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-37b53e-gvlqd" Nov 02 06:51:37.702555 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:37.702372 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71c05684-cf04-46f4-8321-a211172c839e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-37b53e-gvlqd\" (UID: \"71c05684-cf04-46f4-8321-a211172c839e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-37b53e-gvlqd" Nov 02 06:51:37.702555 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:37.702511 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71c05684-cf04-46f4-8321-a211172c839e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-37b53e-gvlqd\" (UID: \"71c05684-cf04-46f4-8321-a211172c839e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-37b53e-gvlqd" Nov 02 06:51:37.703095 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:37.703078 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71c05684-cf04-46f4-8321-a211172c839e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-37b53e-gvlqd\" (UID: \"71c05684-cf04-46f4-8321-a211172c839e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-37b53e-gvlqd" Nov 02 06:51:37.705263 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:37.705241 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71c05684-cf04-46f4-8321-a211172c839e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-37b53e-gvlqd\" (UID: \"71c05684-cf04-46f4-8321-a211172c839e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-37b53e-gvlqd" Nov 02 06:51:37.766094 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:37.766062 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qshf\" (UniqueName: \"kubernetes.io/projected/71c05684-cf04-46f4-8321-a211172c839e-kube-api-access-6qshf\") pod \"maas-api-revoke-keys-e2e-api-b-37b53e-gvlqd\" (UID: \"71c05684-cf04-46f4-8321-a211172c839e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-37b53e-gvlqd" Nov 02 06:51:37.848186 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:37.848097 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-37b53e-gvlqd" Nov 02 06:51:38.231659 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:38.231632 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-37b53e-gvlqd"] Nov 02 06:51:38.232678 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:51:38.232652 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71c05684_cf04_46f4_8321_a211172c839e.slice/crio-9f7915bf71a0d5c88d95b8188449400875d3465da42a20728be1d6ac217cd513 WatchSource:0}: Error finding container 9f7915bf71a0d5c88d95b8188449400875d3465da42a20728be1d6ac217cd513: Status 404 returned error can't find the container with id 9f7915bf71a0d5c88d95b8188449400875d3465da42a20728be1d6ac217cd513 Nov 02 06:51:38.484806 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:38.484728 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-37b53e-gvlqd" event={"ID":"71c05684-cf04-46f4-8321-a211172c839e","Type":"ContainerStarted","Data":"9f7915bf71a0d5c88d95b8188449400875d3465da42a20728be1d6ac217cd513"} Nov 02 06:51:39.494724 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:39.494689 5916 generic.go:356] "Generic (PLEG): container finished" podID="71c05684-cf04-46f4-8321-a211172c839e" containerID="e8c0446547f624181f5a2d523b52c1a4c4c924dd6eebec7c3d88eb2fdd3033e0" exitCode=0 Nov 02 06:51:39.495169 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:39.494807 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-37b53e-gvlqd" event={"ID":"71c05684-cf04-46f4-8321-a211172c839e","Type":"ContainerDied","Data":"e8c0446547f624181f5a2d523b52c1a4c4c924dd6eebec7c3d88eb2fdd3033e0"} Nov 02 06:51:40.666438 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:40.666410 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-37b53e-gvlqd" Nov 02 06:51:40.744135 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:40.744107 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71c05684-cf04-46f4-8321-a211172c839e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71c05684-cf04-46f4-8321-a211172c839e-maas-api-openshift-ca\") pod \"71c05684-cf04-46f4-8321-a211172c839e\" (UID: \"71c05684-cf04-46f4-8321-a211172c839e\") " Nov 02 06:51:40.744320 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:40.744205 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71c05684-cf04-46f4-8321-a211172c839e-kube-api-access-6qshf\" (UniqueName: \"kubernetes.io/projected/71c05684-cf04-46f4-8321-a211172c839e-kube-api-access-6qshf\") pod \"71c05684-cf04-46f4-8321-a211172c839e\" (UID: \"71c05684-cf04-46f4-8321-a211172c839e\") " Nov 02 06:51:40.744387 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:40.744348 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71c05684-cf04-46f4-8321-a211172c839e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71c05684-cf04-46f4-8321-a211172c839e-maas-api-xks-ca\") pod \"71c05684-cf04-46f4-8321-a211172c839e\" (UID: \"71c05684-cf04-46f4-8321-a211172c839e\") " Nov 02 06:51:40.744468 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:40.744442 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71c05684-cf04-46f4-8321-a211172c839e-maas-api-openshift-ca" pod "71c05684-cf04-46f4-8321-a211172c839e" (UID: "71c05684-cf04-46f4-8321-a211172c839e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:51:40.744764 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:40.744750 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71c05684-cf04-46f4-8321-a211172c839e-maas-api-openshift-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:51:40.746401 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:40.746371 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71c05684-cf04-46f4-8321-a211172c839e-kube-api-access-6qshf" pod "71c05684-cf04-46f4-8321-a211172c839e" (UID: "71c05684-cf04-46f4-8321-a211172c839e"). InnerVolumeSpecName "kube-api-access-6qshf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:51:40.746525 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:40.746413 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71c05684-cf04-46f4-8321-a211172c839e-maas-api-xks-ca" pod "71c05684-cf04-46f4-8321-a211172c839e" (UID: "71c05684-cf04-46f4-8321-a211172c839e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:51:40.845929 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:40.845885 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qshf\" (UniqueName: \"kubernetes.io/projected/71c05684-cf04-46f4-8321-a211172c839e-kube-api-access-6qshf\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:51:40.845929 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:40.845928 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71c05684-cf04-46f4-8321-a211172c839e-maas-api-xks-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:51:41.512536 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:41.512498 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-37b53e-gvlqd" event={"ID":"71c05684-cf04-46f4-8321-a211172c839e","Type":"ContainerDied","Data":"9f7915bf71a0d5c88d95b8188449400875d3465da42a20728be1d6ac217cd513"} Nov 02 06:51:41.512536 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:41.512521 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-37b53e-gvlqd" Nov 02 06:51:41.512751 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:41.512546 5916 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9f7915bf71a0d5c88d95b8188449400875d3465da42a20728be1d6ac217cd513" Nov 02 06:51:49.898271 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:51:49.898231 5916 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p98cq" containerName="extractor" Nov 02 06:51:53.157479 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:53.157451 5916 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-csfhg_17dee72b-2f47-4016-8ffe-3a89934e66ee/console-operator/2.log" Nov 02 06:51:53.173048 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:53.173026 5916 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-csfhg_17dee72b-2f47-4016-8ffe-3a89934e66ee/console-operator/2.log" Nov 02 06:51:54.855727 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:54.855693 5916 scope.go:122] "RemoveContainer" containerID="9b70581fea95847fe100445e724f45952af5ea3437c8d59c15560d04bcd39b63" Nov 02 06:51:54.883917 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:54.883878 5916 scope.go:122] "RemoveContainer" containerID="4a945f5db597914b7a17f6945786b1b49e86b1471c6d08556cc10087eb584e30" Nov 02 06:51:55.779930 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:55.774344 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-29abe7-f28qb"] Nov 02 06:51:55.813519 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:55.813478 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-29abe7-f28qb"] Nov 02 06:51:55.849724 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:55.849665 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f8fbc11-84db-4998-9f70-acc2ea9ccef7" path="/var/lib/kubelet/pods/0f8fbc11-84db-4998-9f70-acc2ea9ccef7/volumes" Nov 02 06:51:57.096681 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:57.096644 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6d13c3-b7rlm"] Nov 02 06:51:57.148285 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:57.148249 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6d13c3-b7rlm"] Nov 02 06:51:57.850188 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:57.850134 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5647086-76cc-485c-bff3-1c520fef97ff" path="/var/lib/kubelet/pods/f5647086-76cc-485c-bff3-1c520fef97ff/volumes" Nov 02 06:51:58.053410 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:58.053377 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ead6d32-5clnf"] Nov 02 06:51:58.112385 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:58.112308 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ead6d32-5clnf"] Nov 02 06:51:59.843723 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:51:59.843691 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44fa3493-ecc1-4bcb-a76a-b7d0c2d53540" path="/var/lib/kubelet/pods/44fa3493-ecc1-4bcb-a76a-b7d0c2d53540/volumes" Nov 02 06:52:02.929376 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:02.929322 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-70898f-hphsf"] Nov 02 06:52:02.977753 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:02.977707 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-70898f-hphsf"] Nov 02 06:52:03.601046 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:03.601016 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-29abe7-openshift-default-555ff8c7fc-6zdzz"] Nov 02 06:52:03.601273 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:03.601228 5916 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-29abe7-openshift-default-555ff8c7fc-6zdzz" podUID="5f971e93-8d12-41cf-945c-c932fee7dc16" containerName="istio-proxy" containerID="cri-o://d7c62ec1ad79cf10ec72e4844e86837466e9d255c1d73289982959ef00790f23" gracePeriod=30 Nov 02 06:52:03.846665 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:03.846623 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b24eac55-a02b-4114-b6b5-0bcfe43c203b" path="/var/lib/kubelet/pods/b24eac55-a02b-4114-b6b5-0bcfe43c203b/volumes" Nov 02 06:52:04.292185 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:04.292141 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3bf02ebf-vdjx8"] Nov 02 06:52:04.295294 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:04.295271 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3bf02ebf-vdjx8"] Nov 02 06:52:04.734640 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:04.734607 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-bdcd80-l5h67"] Nov 02 06:52:04.735318 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:04.735298 5916 cpu_manager.go:414] "containerMap: removing container" podUID="71c05684-cf04-46f4-8321-a211172c839e" containerName="revoke-keys" Nov 02 06:52:04.735441 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:04.735319 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71c05684-cf04-46f4-8321-a211172c839e" containerName="revoke-keys" Nov 02 06:52:04.735499 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:04.735462 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="71c05684-cf04-46f4-8321-a211172c839e" containerName="revoke-keys" Nov 02 06:52:04.740754 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:04.740732 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-bdcd80-l5h67" Nov 02 06:52:04.797175 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:04.797145 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-bdcd80-l5h67"] Nov 02 06:52:04.842341 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:04.842318 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 06:52:04.842614 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:04.842599 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-68l4r\"" Nov 02 06:52:04.908935 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:04.908876 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/caac1c08-d143-4d83-9bda-d804cbb59d44-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-bdcd80-l5h67\" (UID: \"caac1c08-d143-4d83-9bda-d804cbb59d44\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-bdcd80-l5h67" Nov 02 06:52:04.909080 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:04.909048 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rt86q\" (UniqueName: \"kubernetes.io/projected/caac1c08-d143-4d83-9bda-d804cbb59d44-kube-api-access-rt86q\") pod \"maas-api-revoke-keys-e2e-worker-w0-bdcd80-l5h67\" (UID: \"caac1c08-d143-4d83-9bda-d804cbb59d44\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-bdcd80-l5h67" Nov 02 06:52:04.909149 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:04.909138 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/caac1c08-d143-4d83-9bda-d804cbb59d44-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-bdcd80-l5h67\" (UID: \"caac1c08-d143-4d83-9bda-d804cbb59d44\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-bdcd80-l5h67" Nov 02 06:52:05.010423 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:05.010334 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rt86q\" (UniqueName: \"kubernetes.io/projected/caac1c08-d143-4d83-9bda-d804cbb59d44-kube-api-access-rt86q\") pod \"maas-api-revoke-keys-e2e-worker-w0-bdcd80-l5h67\" (UID: \"caac1c08-d143-4d83-9bda-d804cbb59d44\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-bdcd80-l5h67" Nov 02 06:52:05.010632 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:05.010597 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/caac1c08-d143-4d83-9bda-d804cbb59d44-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-bdcd80-l5h67\" (UID: \"caac1c08-d143-4d83-9bda-d804cbb59d44\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-bdcd80-l5h67" Nov 02 06:52:05.010771 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:05.010746 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/caac1c08-d143-4d83-9bda-d804cbb59d44-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-bdcd80-l5h67\" (UID: \"caac1c08-d143-4d83-9bda-d804cbb59d44\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-bdcd80-l5h67" Nov 02 06:52:05.011274 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:05.011252 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/caac1c08-d143-4d83-9bda-d804cbb59d44-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-bdcd80-l5h67\" (UID: \"caac1c08-d143-4d83-9bda-d804cbb59d44\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-bdcd80-l5h67" Nov 02 06:52:05.013097 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:05.013077 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/caac1c08-d143-4d83-9bda-d804cbb59d44-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-bdcd80-l5h67\" (UID: \"caac1c08-d143-4d83-9bda-d804cbb59d44\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-bdcd80-l5h67" Nov 02 06:52:05.072272 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:05.072241 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rt86q\" (UniqueName: \"kubernetes.io/projected/caac1c08-d143-4d83-9bda-d804cbb59d44-kube-api-access-rt86q\") pod \"maas-api-revoke-keys-e2e-worker-w0-bdcd80-l5h67\" (UID: \"caac1c08-d143-4d83-9bda-d804cbb59d44\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-bdcd80-l5h67" Nov 02 06:52:05.359037 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:05.358948 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-bdcd80-l5h67" Nov 02 06:52:05.742986 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:05.742895 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-bdcd80-l5h67"] Nov 02 06:52:05.750866 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:52:05.750832 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcaac1c08_d143_4d83_9bda_d804cbb59d44.slice/crio-285bef64b8faaf22e20657c9961d99d7acc8f488caf75ae9f79a4dce5315d6b9 WatchSource:0}: Error finding container 285bef64b8faaf22e20657c9961d99d7acc8f488caf75ae9f79a4dce5315d6b9: Status 404 returned error can't find the container with id 285bef64b8faaf22e20657c9961d99d7acc8f488caf75ae9f79a4dce5315d6b9 Nov 02 06:52:05.788432 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:05.788398 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-bdcd80-l5h67" event={"ID":"caac1c08-d143-4d83-9bda-d804cbb59d44","Type":"ContainerStarted","Data":"285bef64b8faaf22e20657c9961d99d7acc8f488caf75ae9f79a4dce5315d6b9"} Nov 02 06:52:05.872930 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:05.872363 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14f16f4c-56ba-4dbe-bda0-912a4e67dffc" path="/var/lib/kubelet/pods/14f16f4c-56ba-4dbe-bda0-912a4e67dffc/volumes" Nov 02 06:52:06.137952 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:06.137262 5916 patch_prober.go:29] interesting pod/e2e-shared-b-w0-29abe7-openshift-default-555ff8c7fc-6zdzz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 06:52:06.137952 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:06.137367 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w0-29abe7-openshift-default-555ff8c7fc-6zdzz" podUID="5f971e93-8d12-41cf-945c-c932fee7dc16" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 06:52:06.798951 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:06.798901 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-bdcd80-l5h67" event={"ID":"caac1c08-d143-4d83-9bda-d804cbb59d44","Type":"ContainerStarted","Data":"c654f11aa05caef996e1c78ebd47786b3c89b9e2f7f45b392f59fbfd34a7cc47"} Nov 02 06:52:06.914479 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:06.914430 5916 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-bdcd80-l5h67" podStartSLOduration=2.914415005 podStartE2EDuration="2.914415005s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:52:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:52:06.913014051 +0000 UTC m=+2115.184834464" watchObservedRunningTime="2026-11-02 06:52:06.914415005 +0000 UTC m=+2115.186235449" Nov 02 06:52:07.807442 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:07.807402 5916 generic.go:356] "Generic (PLEG): container finished" podID="caac1c08-d143-4d83-9bda-d804cbb59d44" containerID="c654f11aa05caef996e1c78ebd47786b3c89b9e2f7f45b392f59fbfd34a7cc47" exitCode=0 Nov 02 06:52:07.807928 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:07.807483 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-bdcd80-l5h67" event={"ID":"caac1c08-d143-4d83-9bda-d804cbb59d44","Type":"ContainerDied","Data":"c654f11aa05caef996e1c78ebd47786b3c89b9e2f7f45b392f59fbfd34a7cc47"} Nov 02 06:52:08.821099 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:08.821071 5916 generic.go:356] "Generic (PLEG): container finished" podID="5f971e93-8d12-41cf-945c-c932fee7dc16" containerID="d7c62ec1ad79cf10ec72e4844e86837466e9d255c1d73289982959ef00790f23" exitCode=0 Nov 02 06:52:08.821750 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:08.821324 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-29abe7-openshift-default-555ff8c7fc-6zdzz" event={"ID":"5f971e93-8d12-41cf-945c-c932fee7dc16","Type":"ContainerDied","Data":"d7c62ec1ad79cf10ec72e4844e86837466e9d255c1d73289982959ef00790f23"} Nov 02 06:52:08.821876 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:08.821798 5916 scope.go:122] "RemoveContainer" containerID="35301f49b485c12f73f63e76bade89ff9a832ba49b644c6b94ad4b35f00b0610" Nov 02 06:52:08.906177 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:08.906151 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-29abe7-openshift-default-555ff8c7fc-6zdzz" Nov 02 06:52:08.976094 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:08.976056 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5f971e93-8d12-41cf-945c-c932fee7dc16-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f971e93-8d12-41cf-945c-c932fee7dc16-istio-podinfo\") pod \"5f971e93-8d12-41cf-945c-c932fee7dc16\" (UID: \"5f971e93-8d12-41cf-945c-c932fee7dc16\") " Nov 02 06:52:08.976283 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:08.976137 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f971e93-8d12-41cf-945c-c932fee7dc16-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f971e93-8d12-41cf-945c-c932fee7dc16-istiod-ca-cert\") pod \"5f971e93-8d12-41cf-945c-c932fee7dc16\" (UID: \"5f971e93-8d12-41cf-945c-c932fee7dc16\") " Nov 02 06:52:08.976283 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:08.976201 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f971e93-8d12-41cf-945c-c932fee7dc16-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f971e93-8d12-41cf-945c-c932fee7dc16-istio-envoy\") pod \"5f971e93-8d12-41cf-945c-c932fee7dc16\" (UID: \"5f971e93-8d12-41cf-945c-c932fee7dc16\") " Nov 02 06:52:08.976283 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:08.976243 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f971e93-8d12-41cf-945c-c932fee7dc16-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f971e93-8d12-41cf-945c-c932fee7dc16-istio-data\") pod \"5f971e93-8d12-41cf-945c-c932fee7dc16\" (UID: \"5f971e93-8d12-41cf-945c-c932fee7dc16\") " Nov 02 06:52:08.976461 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:08.976302 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f971e93-8d12-41cf-945c-c932fee7dc16-istio-token\" (UniqueName: \"kubernetes.io/projected/5f971e93-8d12-41cf-945c-c932fee7dc16-istio-token\") pod \"5f971e93-8d12-41cf-945c-c932fee7dc16\" (UID: \"5f971e93-8d12-41cf-945c-c932fee7dc16\") " Nov 02 06:52:08.976461 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:08.976343 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f971e93-8d12-41cf-945c-c932fee7dc16-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f971e93-8d12-41cf-945c-c932fee7dc16-workload-socket\") pod \"5f971e93-8d12-41cf-945c-c932fee7dc16\" (UID: \"5f971e93-8d12-41cf-945c-c932fee7dc16\") " Nov 02 06:52:08.976461 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:08.976402 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f971e93-8d12-41cf-945c-c932fee7dc16-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f971e93-8d12-41cf-945c-c932fee7dc16-workload-certs\") pod \"5f971e93-8d12-41cf-945c-c932fee7dc16\" (UID: \"5f971e93-8d12-41cf-945c-c932fee7dc16\") " Nov 02 06:52:08.976618 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:08.976494 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f971e93-8d12-41cf-945c-c932fee7dc16-kube-api-access-ptcxt\" (UniqueName: \"kubernetes.io/projected/5f971e93-8d12-41cf-945c-c932fee7dc16-kube-api-access-ptcxt\") pod \"5f971e93-8d12-41cf-945c-c932fee7dc16\" (UID: \"5f971e93-8d12-41cf-945c-c932fee7dc16\") " Nov 02 06:52:08.976618 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:08.976529 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f971e93-8d12-41cf-945c-c932fee7dc16-istiod-ca-cert" pod "5f971e93-8d12-41cf-945c-c932fee7dc16" (UID: "5f971e93-8d12-41cf-945c-c932fee7dc16"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:52:08.976618 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:08.976583 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f971e93-8d12-41cf-945c-c932fee7dc16-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f971e93-8d12-41cf-945c-c932fee7dc16-credential-socket\") pod \"5f971e93-8d12-41cf-945c-c932fee7dc16\" (UID: \"5f971e93-8d12-41cf-945c-c932fee7dc16\") " Nov 02 06:52:08.976779 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:08.976681 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f971e93-8d12-41cf-945c-c932fee7dc16-workload-socket" pod "5f971e93-8d12-41cf-945c-c932fee7dc16" (UID: "5f971e93-8d12-41cf-945c-c932fee7dc16"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:52:08.976838 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:08.976815 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f971e93-8d12-41cf-945c-c932fee7dc16-workload-certs" pod "5f971e93-8d12-41cf-945c-c932fee7dc16" (UID: "5f971e93-8d12-41cf-945c-c932fee7dc16"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:52:08.976959 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:08.976939 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f971e93-8d12-41cf-945c-c932fee7dc16-credential-socket" pod "5f971e93-8d12-41cf-945c-c932fee7dc16" (UID: "5f971e93-8d12-41cf-945c-c932fee7dc16"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:52:08.977030 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:08.976956 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f971e93-8d12-41cf-945c-c932fee7dc16-istio-data" pod "5f971e93-8d12-41cf-945c-c932fee7dc16" (UID: "5f971e93-8d12-41cf-945c-c932fee7dc16"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:52:08.977139 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:08.977124 5916 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f971e93-8d12-41cf-945c-c932fee7dc16-istio-data\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:52:08.977201 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:08.977140 5916 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f971e93-8d12-41cf-945c-c932fee7dc16-workload-socket\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:52:08.977201 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:08.977151 5916 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f971e93-8d12-41cf-945c-c932fee7dc16-workload-certs\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:52:08.977201 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:08.977161 5916 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f971e93-8d12-41cf-945c-c932fee7dc16-credential-socket\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:52:08.977201 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:08.977170 5916 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f971e93-8d12-41cf-945c-c932fee7dc16-istiod-ca-cert\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:52:08.979478 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:08.979454 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f971e93-8d12-41cf-945c-c932fee7dc16-istio-token" pod "5f971e93-8d12-41cf-945c-c932fee7dc16" (UID: "5f971e93-8d12-41cf-945c-c932fee7dc16"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:52:08.980210 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:08.979635 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f971e93-8d12-41cf-945c-c932fee7dc16-istio-envoy" pod "5f971e93-8d12-41cf-945c-c932fee7dc16" (UID: "5f971e93-8d12-41cf-945c-c932fee7dc16"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:52:08.980326 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:08.980302 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f971e93-8d12-41cf-945c-c932fee7dc16-kube-api-access-ptcxt" pod "5f971e93-8d12-41cf-945c-c932fee7dc16" (UID: "5f971e93-8d12-41cf-945c-c932fee7dc16"). InnerVolumeSpecName "kube-api-access-ptcxt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:52:08.980510 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:08.980494 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5f971e93-8d12-41cf-945c-c932fee7dc16-istio-podinfo" pod "5f971e93-8d12-41cf-945c-c932fee7dc16" (UID: "5f971e93-8d12-41cf-945c-c932fee7dc16"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 06:52:09.014866 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:09.014837 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-bdcd80/distinct-w0-bdcd80-kserve-5d9df79d48-bn5d4"] Nov 02 06:52:09.018800 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:09.018779 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-bdcd80/simulator-w0-bdcd80-kserve-5749c847cb-9ktcr"] Nov 02 06:52:09.021275 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:09.021256 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-bdcd80-l5h67" Nov 02 06:52:09.078088 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:09.078053 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/caac1c08-d143-4d83-9bda-d804cbb59d44-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/caac1c08-d143-4d83-9bda-d804cbb59d44-maas-api-xks-ca\") pod \"caac1c08-d143-4d83-9bda-d804cbb59d44\" (UID: \"caac1c08-d143-4d83-9bda-d804cbb59d44\") " Nov 02 06:52:09.078269 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:09.078125 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/caac1c08-d143-4d83-9bda-d804cbb59d44-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/caac1c08-d143-4d83-9bda-d804cbb59d44-maas-api-openshift-ca\") pod \"caac1c08-d143-4d83-9bda-d804cbb59d44\" (UID: \"caac1c08-d143-4d83-9bda-d804cbb59d44\") " Nov 02 06:52:09.078337 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:09.078271 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/caac1c08-d143-4d83-9bda-d804cbb59d44-kube-api-access-rt86q\" (UniqueName: \"kubernetes.io/projected/caac1c08-d143-4d83-9bda-d804cbb59d44-kube-api-access-rt86q\") pod \"caac1c08-d143-4d83-9bda-d804cbb59d44\" (UID: \"caac1c08-d143-4d83-9bda-d804cbb59d44\") " Nov 02 06:52:09.078518 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:09.078475 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/caac1c08-d143-4d83-9bda-d804cbb59d44-maas-api-openshift-ca" pod "caac1c08-d143-4d83-9bda-d804cbb59d44" (UID: "caac1c08-d143-4d83-9bda-d804cbb59d44"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:52:09.078888 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:09.078868 5916 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f971e93-8d12-41cf-945c-c932fee7dc16-istio-podinfo\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:52:09.078992 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:09.078893 5916 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f971e93-8d12-41cf-945c-c932fee7dc16-istio-envoy\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:52:09.078992 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:09.078924 5916 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f971e93-8d12-41cf-945c-c932fee7dc16-istio-token\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:52:09.078992 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:09.078939 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptcxt\" (UniqueName: \"kubernetes.io/projected/5f971e93-8d12-41cf-945c-c932fee7dc16-kube-api-access-ptcxt\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:52:09.078992 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:09.078953 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/caac1c08-d143-4d83-9bda-d804cbb59d44-maas-api-openshift-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:52:09.080705 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:09.080677 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/caac1c08-d143-4d83-9bda-d804cbb59d44-maas-api-xks-ca" pod "caac1c08-d143-4d83-9bda-d804cbb59d44" (UID: "caac1c08-d143-4d83-9bda-d804cbb59d44"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:52:09.080794 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:09.080707 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/caac1c08-d143-4d83-9bda-d804cbb59d44-kube-api-access-rt86q" pod "caac1c08-d143-4d83-9bda-d804cbb59d44" (UID: "caac1c08-d143-4d83-9bda-d804cbb59d44"). InnerVolumeSpecName "kube-api-access-rt86q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:52:09.180331 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:09.180298 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/caac1c08-d143-4d83-9bda-d804cbb59d44-maas-api-xks-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:52:09.180331 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:09.180326 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rt86q\" (UniqueName: \"kubernetes.io/projected/caac1c08-d143-4d83-9bda-d804cbb59d44-kube-api-access-rt86q\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:52:09.831127 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:09.831100 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-29abe7-openshift-default-555ff8c7fc-6zdzz" Nov 02 06:52:09.831611 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:09.831139 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-29abe7-openshift-default-555ff8c7fc-6zdzz" event={"ID":"5f971e93-8d12-41cf-945c-c932fee7dc16","Type":"ContainerDied","Data":"47d5ff6aa228002406ac330575aea9e32508f0ad724bedb6101a08773af04ef6"} Nov 02 06:52:09.831611 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:09.831204 5916 scope.go:122] "RemoveContainer" containerID="d7c62ec1ad79cf10ec72e4844e86837466e9d255c1d73289982959ef00790f23" Nov 02 06:52:09.833445 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:09.833416 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-bdcd80-l5h67" event={"ID":"caac1c08-d143-4d83-9bda-d804cbb59d44","Type":"ContainerDied","Data":"285bef64b8faaf22e20657c9961d99d7acc8f488caf75ae9f79a4dce5315d6b9"} Nov 02 06:52:09.833587 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:09.833470 5916 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="285bef64b8faaf22e20657c9961d99d7acc8f488caf75ae9f79a4dce5315d6b9" Nov 02 06:52:09.833587 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:09.833528 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-bdcd80-l5h67" Nov 02 06:52:10.126927 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:10.126838 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-29abe7-openshift-default-555ff8c7fc-6zdzz"] Nov 02 06:52:10.149927 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:10.142559 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-29abe7-openshift-default-555ff8c7fc-6zdzz"] Nov 02 06:52:11.164380 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:11.164338 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e233a7-6l45k"] Nov 02 06:52:11.165308 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:11.165288 5916 cpu_manager.go:414] "containerMap: removing container" podUID="5f971e93-8d12-41cf-945c-c932fee7dc16" containerName="istio-proxy" Nov 02 06:52:11.165366 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:11.165311 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f971e93-8d12-41cf-945c-c932fee7dc16" containerName="istio-proxy" Nov 02 06:52:11.165366 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:11.165332 5916 cpu_manager.go:414] "containerMap: removing container" podUID="5f971e93-8d12-41cf-945c-c932fee7dc16" containerName="istio-proxy" Nov 02 06:52:11.165366 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:11.165339 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f971e93-8d12-41cf-945c-c932fee7dc16" containerName="istio-proxy" Nov 02 06:52:11.165366 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:11.165357 5916 cpu_manager.go:414] "containerMap: removing container" podUID="caac1c08-d143-4d83-9bda-d804cbb59d44" containerName="revoke-keys" Nov 02 06:52:11.165366 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:11.165364 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="caac1c08-d143-4d83-9bda-d804cbb59d44" containerName="revoke-keys" Nov 02 06:52:11.165542 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:11.165531 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f971e93-8d12-41cf-945c-c932fee7dc16" containerName="istio-proxy" Nov 02 06:52:11.165581 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:11.165546 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f971e93-8d12-41cf-945c-c932fee7dc16" containerName="istio-proxy" Nov 02 06:52:11.165581 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:11.165557 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="caac1c08-d143-4d83-9bda-d804cbb59d44" containerName="revoke-keys" Nov 02 06:52:11.171151 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:11.171128 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e233a7-6l45k" Nov 02 06:52:11.224171 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:11.224140 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e233a7-6l45k"] Nov 02 06:52:11.249235 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:11.249196 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gq5xr\" (UniqueName: \"kubernetes.io/projected/d900cf8d-6342-4b18-8d40-f28ab6d347b8-kube-api-access-gq5xr\") pod \"maas-api-revoke-keys-e2e-api-b-e233a7-6l45k\" (UID: \"d900cf8d-6342-4b18-8d40-f28ab6d347b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e233a7-6l45k" Nov 02 06:52:11.249459 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:11.249303 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d900cf8d-6342-4b18-8d40-f28ab6d347b8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-e233a7-6l45k\" (UID: \"d900cf8d-6342-4b18-8d40-f28ab6d347b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e233a7-6l45k" Nov 02 06:52:11.249459 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:11.249412 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d900cf8d-6342-4b18-8d40-f28ab6d347b8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-e233a7-6l45k\" (UID: \"d900cf8d-6342-4b18-8d40-f28ab6d347b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e233a7-6l45k" Nov 02 06:52:11.269606 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:11.269536 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 06:52:11.269787 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:11.269606 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-68l4r\"" Nov 02 06:52:11.350237 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:11.350193 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gq5xr\" (UniqueName: \"kubernetes.io/projected/d900cf8d-6342-4b18-8d40-f28ab6d347b8-kube-api-access-gq5xr\") pod \"maas-api-revoke-keys-e2e-api-b-e233a7-6l45k\" (UID: \"d900cf8d-6342-4b18-8d40-f28ab6d347b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e233a7-6l45k" Nov 02 06:52:11.350460 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:11.350340 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d900cf8d-6342-4b18-8d40-f28ab6d347b8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-e233a7-6l45k\" (UID: \"d900cf8d-6342-4b18-8d40-f28ab6d347b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e233a7-6l45k" Nov 02 06:52:11.350460 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:11.350421 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d900cf8d-6342-4b18-8d40-f28ab6d347b8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-e233a7-6l45k\" (UID: \"d900cf8d-6342-4b18-8d40-f28ab6d347b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e233a7-6l45k" Nov 02 06:52:11.351041 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:11.351022 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d900cf8d-6342-4b18-8d40-f28ab6d347b8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-e233a7-6l45k\" (UID: \"d900cf8d-6342-4b18-8d40-f28ab6d347b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e233a7-6l45k" Nov 02 06:52:11.352986 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:11.352954 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d900cf8d-6342-4b18-8d40-f28ab6d347b8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-e233a7-6l45k\" (UID: \"d900cf8d-6342-4b18-8d40-f28ab6d347b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e233a7-6l45k" Nov 02 06:52:11.411442 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:11.411414 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gq5xr\" (UniqueName: \"kubernetes.io/projected/d900cf8d-6342-4b18-8d40-f28ab6d347b8-kube-api-access-gq5xr\") pod \"maas-api-revoke-keys-e2e-api-b-e233a7-6l45k\" (UID: \"d900cf8d-6342-4b18-8d40-f28ab6d347b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e233a7-6l45k" Nov 02 06:52:11.488394 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:11.488307 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e233a7-6l45k" Nov 02 06:52:11.846961 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:11.846528 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f971e93-8d12-41cf-945c-c932fee7dc16" path="/var/lib/kubelet/pods/5f971e93-8d12-41cf-945c-c932fee7dc16/volumes" Nov 02 06:52:11.877802 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:11.877671 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e233a7-6l45k"] Nov 02 06:52:11.880113 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:52:11.880077 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd900cf8d_6342_4b18_8d40_f28ab6d347b8.slice/crio-b625940f8bdc55a38e6c19f3f0e0a7f473896f61f1b05011e7d43ac327db6b3c WatchSource:0}: Error finding container b625940f8bdc55a38e6c19f3f0e0a7f473896f61f1b05011e7d43ac327db6b3c: Status 404 returned error can't find the container with id b625940f8bdc55a38e6c19f3f0e0a7f473896f61f1b05011e7d43ac327db6b3c Nov 02 06:52:12.865679 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:12.865646 5916 generic.go:356] "Generic (PLEG): container finished" podID="d900cf8d-6342-4b18-8d40-f28ab6d347b8" containerID="dd704b5564325d7bbb957e64b2f7203ecc1e8e1cdd71162f43dee67e47c39570" exitCode=0 Nov 02 06:52:12.866115 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:12.865732 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e233a7-6l45k" event={"ID":"d900cf8d-6342-4b18-8d40-f28ab6d347b8","Type":"ContainerDied","Data":"dd704b5564325d7bbb957e64b2f7203ecc1e8e1cdd71162f43dee67e47c39570"} Nov 02 06:52:12.866115 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:12.865803 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e233a7-6l45k" event={"ID":"d900cf8d-6342-4b18-8d40-f28ab6d347b8","Type":"ContainerStarted","Data":"b625940f8bdc55a38e6c19f3f0e0a7f473896f61f1b05011e7d43ac327db6b3c"} Nov 02 06:52:14.299966 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:14.299940 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e233a7-6l45k" Nov 02 06:52:14.396164 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:14.396134 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d900cf8d-6342-4b18-8d40-f28ab6d347b8-kube-api-access-gq5xr\" (UniqueName: \"kubernetes.io/projected/d900cf8d-6342-4b18-8d40-f28ab6d347b8-kube-api-access-gq5xr\") pod \"d900cf8d-6342-4b18-8d40-f28ab6d347b8\" (UID: \"d900cf8d-6342-4b18-8d40-f28ab6d347b8\") " Nov 02 06:52:14.396346 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:14.396217 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d900cf8d-6342-4b18-8d40-f28ab6d347b8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d900cf8d-6342-4b18-8d40-f28ab6d347b8-maas-api-xks-ca\") pod \"d900cf8d-6342-4b18-8d40-f28ab6d347b8\" (UID: \"d900cf8d-6342-4b18-8d40-f28ab6d347b8\") " Nov 02 06:52:14.396346 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:14.396247 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d900cf8d-6342-4b18-8d40-f28ab6d347b8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d900cf8d-6342-4b18-8d40-f28ab6d347b8-maas-api-openshift-ca\") pod \"d900cf8d-6342-4b18-8d40-f28ab6d347b8\" (UID: \"d900cf8d-6342-4b18-8d40-f28ab6d347b8\") " Nov 02 06:52:14.396661 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:14.396626 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d900cf8d-6342-4b18-8d40-f28ab6d347b8-maas-api-openshift-ca" pod "d900cf8d-6342-4b18-8d40-f28ab6d347b8" (UID: "d900cf8d-6342-4b18-8d40-f28ab6d347b8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:52:14.396771 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:14.396759 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d900cf8d-6342-4b18-8d40-f28ab6d347b8-maas-api-openshift-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:52:14.398295 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:14.398271 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d900cf8d-6342-4b18-8d40-f28ab6d347b8-kube-api-access-gq5xr" pod "d900cf8d-6342-4b18-8d40-f28ab6d347b8" (UID: "d900cf8d-6342-4b18-8d40-f28ab6d347b8"). InnerVolumeSpecName "kube-api-access-gq5xr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:52:14.398389 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:14.398330 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d900cf8d-6342-4b18-8d40-f28ab6d347b8-maas-api-xks-ca" pod "d900cf8d-6342-4b18-8d40-f28ab6d347b8" (UID: "d900cf8d-6342-4b18-8d40-f28ab6d347b8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:52:14.497528 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:14.497451 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gq5xr\" (UniqueName: \"kubernetes.io/projected/d900cf8d-6342-4b18-8d40-f28ab6d347b8-kube-api-access-gq5xr\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:52:14.497528 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:14.497483 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d900cf8d-6342-4b18-8d40-f28ab6d347b8-maas-api-xks-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:52:14.883800 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:14.883767 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e233a7-6l45k" event={"ID":"d900cf8d-6342-4b18-8d40-f28ab6d347b8","Type":"ContainerDied","Data":"b625940f8bdc55a38e6c19f3f0e0a7f473896f61f1b05011e7d43ac327db6b3c"} Nov 02 06:52:14.884080 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:14.883803 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e233a7-6l45k" Nov 02 06:52:14.884080 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:14.883819 5916 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b625940f8bdc55a38e6c19f3f0e0a7f473896f61f1b05011e7d43ac327db6b3c" Nov 02 06:52:19.897048 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:52:19.897010 5916 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p98cq" containerName="extractor" Nov 02 06:52:24.038049 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:24.037990 5916 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-bdcd80/distinct-w0-bdcd80-kserve-5d9df79d48-bn5d4" podUID="1a7c7a03-e0cf-44ca-8eee-6c3e16112b3a" containerName="main" containerID="cri-o://b217ac5694f6fcc460c824d3050dbdc22c73771f5122ce3909539b5d333e98f5" gracePeriod=45 Nov 02 06:52:24.039669 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:24.039517 5916 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-bdcd80/simulator-w0-bdcd80-kserve-5749c847cb-9ktcr" podUID="2043ff0a-8a71-4b71-9750-beb566a12bad" containerName="main" containerID="cri-o://4d71cefed7597633799490ffef0d1abf6c665ef7b6629a4e265df95957a34be7" gracePeriod=45 Nov 02 06:52:24.171452 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:24.171414 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bdcd80/distinct-w0-bdcd80-kserve-5d9df79d48-bn5d4" podUID="1a7c7a03-e0cf-44ca-8eee-6c3e16112b3a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": dial tcp 10.132.0.98:8000: connect: connection refused" Nov 02 06:52:24.416215 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:24.416184 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-bdcd80/distinct-w0-bdcd80-kserve-5d9df79d48-bn5d4" Nov 02 06:52:24.435986 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:24.435873 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a7c7a03-e0cf-44ca-8eee-6c3e16112b3a-tls-certs\" (UniqueName: \"kubernetes.io/secret/1a7c7a03-e0cf-44ca-8eee-6c3e16112b3a-tls-certs\") pod \"1a7c7a03-e0cf-44ca-8eee-6c3e16112b3a\" (UID: \"1a7c7a03-e0cf-44ca-8eee-6c3e16112b3a\") " Nov 02 06:52:24.436120 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:24.436072 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a7c7a03-e0cf-44ca-8eee-6c3e16112b3a-kube-api-access-rc6r5\" (UniqueName: \"kubernetes.io/projected/1a7c7a03-e0cf-44ca-8eee-6c3e16112b3a-kube-api-access-rc6r5\") pod \"1a7c7a03-e0cf-44ca-8eee-6c3e16112b3a\" (UID: \"1a7c7a03-e0cf-44ca-8eee-6c3e16112b3a\") " Nov 02 06:52:24.436167 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:24.436149 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a7c7a03-e0cf-44ca-8eee-6c3e16112b3a-home\" (UniqueName: \"kubernetes.io/empty-dir/1a7c7a03-e0cf-44ca-8eee-6c3e16112b3a-home\") pod \"1a7c7a03-e0cf-44ca-8eee-6c3e16112b3a\" (UID: \"1a7c7a03-e0cf-44ca-8eee-6c3e16112b3a\") " Nov 02 06:52:24.436211 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:24.436191 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a7c7a03-e0cf-44ca-8eee-6c3e16112b3a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a7c7a03-e0cf-44ca-8eee-6c3e16112b3a-dshm\") pod \"1a7c7a03-e0cf-44ca-8eee-6c3e16112b3a\" (UID: \"1a7c7a03-e0cf-44ca-8eee-6c3e16112b3a\") " Nov 02 06:52:24.436252 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:24.436232 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a7c7a03-e0cf-44ca-8eee-6c3e16112b3a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a7c7a03-e0cf-44ca-8eee-6c3e16112b3a-model-cache\") pod \"1a7c7a03-e0cf-44ca-8eee-6c3e16112b3a\" (UID: \"1a7c7a03-e0cf-44ca-8eee-6c3e16112b3a\") " Nov 02 06:52:24.436388 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:24.436364 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a7c7a03-e0cf-44ca-8eee-6c3e16112b3a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a7c7a03-e0cf-44ca-8eee-6c3e16112b3a-tmp-dir\") pod \"1a7c7a03-e0cf-44ca-8eee-6c3e16112b3a\" (UID: \"1a7c7a03-e0cf-44ca-8eee-6c3e16112b3a\") " Nov 02 06:52:24.437098 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:24.437070 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a7c7a03-e0cf-44ca-8eee-6c3e16112b3a-tmp-dir" pod "1a7c7a03-e0cf-44ca-8eee-6c3e16112b3a" (UID: "1a7c7a03-e0cf-44ca-8eee-6c3e16112b3a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:52:24.437648 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:24.437579 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a7c7a03-e0cf-44ca-8eee-6c3e16112b3a-model-cache" pod "1a7c7a03-e0cf-44ca-8eee-6c3e16112b3a" (UID: "1a7c7a03-e0cf-44ca-8eee-6c3e16112b3a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:52:24.437648 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:24.437608 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a7c7a03-e0cf-44ca-8eee-6c3e16112b3a-home" pod "1a7c7a03-e0cf-44ca-8eee-6c3e16112b3a" (UID: "1a7c7a03-e0cf-44ca-8eee-6c3e16112b3a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:52:24.439409 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:24.439385 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a7c7a03-e0cf-44ca-8eee-6c3e16112b3a-dshm" pod "1a7c7a03-e0cf-44ca-8eee-6c3e16112b3a" (UID: "1a7c7a03-e0cf-44ca-8eee-6c3e16112b3a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:52:24.439554 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:24.439536 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a7c7a03-e0cf-44ca-8eee-6c3e16112b3a-tls-certs" pod "1a7c7a03-e0cf-44ca-8eee-6c3e16112b3a" (UID: "1a7c7a03-e0cf-44ca-8eee-6c3e16112b3a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:52:24.439662 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:24.439640 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a7c7a03-e0cf-44ca-8eee-6c3e16112b3a-kube-api-access-rc6r5" pod "1a7c7a03-e0cf-44ca-8eee-6c3e16112b3a" (UID: "1a7c7a03-e0cf-44ca-8eee-6c3e16112b3a"). InnerVolumeSpecName "kube-api-access-rc6r5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:52:24.446781 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:24.446756 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-bdcd80/simulator-w0-bdcd80-kserve-5749c847cb-9ktcr" Nov 02 06:52:24.537252 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:24.537220 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2043ff0a-8a71-4b71-9750-beb566a12bad-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2043ff0a-8a71-4b71-9750-beb566a12bad-tmp-dir\") pod \"2043ff0a-8a71-4b71-9750-beb566a12bad\" (UID: \"2043ff0a-8a71-4b71-9750-beb566a12bad\") " Nov 02 06:52:24.537460 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:24.537269 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2043ff0a-8a71-4b71-9750-beb566a12bad-home\" (UniqueName: \"kubernetes.io/empty-dir/2043ff0a-8a71-4b71-9750-beb566a12bad-home\") pod \"2043ff0a-8a71-4b71-9750-beb566a12bad\" (UID: \"2043ff0a-8a71-4b71-9750-beb566a12bad\") " Nov 02 06:52:24.537460 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:24.537354 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2043ff0a-8a71-4b71-9750-beb566a12bad-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2043ff0a-8a71-4b71-9750-beb566a12bad-model-cache\") pod \"2043ff0a-8a71-4b71-9750-beb566a12bad\" (UID: \"2043ff0a-8a71-4b71-9750-beb566a12bad\") " Nov 02 06:52:24.537460 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:24.537411 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2043ff0a-8a71-4b71-9750-beb566a12bad-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2043ff0a-8a71-4b71-9750-beb566a12bad-dshm\") pod \"2043ff0a-8a71-4b71-9750-beb566a12bad\" (UID: \"2043ff0a-8a71-4b71-9750-beb566a12bad\") " Nov 02 06:52:24.537622 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:24.537482 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2043ff0a-8a71-4b71-9750-beb566a12bad-tmp-dir" pod "2043ff0a-8a71-4b71-9750-beb566a12bad" (UID: "2043ff0a-8a71-4b71-9750-beb566a12bad"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:52:24.537622 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:24.537553 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2043ff0a-8a71-4b71-9750-beb566a12bad-home" pod "2043ff0a-8a71-4b71-9750-beb566a12bad" (UID: "2043ff0a-8a71-4b71-9750-beb566a12bad"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:52:24.537622 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:24.537602 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2043ff0a-8a71-4b71-9750-beb566a12bad-model-cache" pod "2043ff0a-8a71-4b71-9750-beb566a12bad" (UID: "2043ff0a-8a71-4b71-9750-beb566a12bad"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:52:24.537777 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:24.537624 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2043ff0a-8a71-4b71-9750-beb566a12bad-kube-api-access-n6tc7\" (UniqueName: \"kubernetes.io/projected/2043ff0a-8a71-4b71-9750-beb566a12bad-kube-api-access-n6tc7\") pod \"2043ff0a-8a71-4b71-9750-beb566a12bad\" (UID: \"2043ff0a-8a71-4b71-9750-beb566a12bad\") " Nov 02 06:52:24.537777 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:24.537687 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2043ff0a-8a71-4b71-9750-beb566a12bad-tls-certs\" (UniqueName: \"kubernetes.io/secret/2043ff0a-8a71-4b71-9750-beb566a12bad-tls-certs\") pod \"2043ff0a-8a71-4b71-9750-beb566a12bad\" (UID: \"2043ff0a-8a71-4b71-9750-beb566a12bad\") " Nov 02 06:52:24.538140 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:24.538119 5916 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a7c7a03-e0cf-44ca-8eee-6c3e16112b3a-tmp-dir\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:52:24.538248 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:24.538141 5916 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a7c7a03-e0cf-44ca-8eee-6c3e16112b3a-tls-certs\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:52:24.538248 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:24.538156 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rc6r5\" (UniqueName: \"kubernetes.io/projected/1a7c7a03-e0cf-44ca-8eee-6c3e16112b3a-kube-api-access-rc6r5\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:52:24.538248 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:24.538167 5916 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a7c7a03-e0cf-44ca-8eee-6c3e16112b3a-home\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:52:24.538248 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:24.538180 5916 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2043ff0a-8a71-4b71-9750-beb566a12bad-tmp-dir\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:52:24.538248 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:24.538191 5916 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2043ff0a-8a71-4b71-9750-beb566a12bad-home\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:52:24.538248 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:24.538202 5916 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a7c7a03-e0cf-44ca-8eee-6c3e16112b3a-dshm\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:52:24.538248 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:24.538214 5916 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a7c7a03-e0cf-44ca-8eee-6c3e16112b3a-model-cache\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:52:24.538248 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:24.538225 5916 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2043ff0a-8a71-4b71-9750-beb566a12bad-model-cache\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:52:24.540258 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:24.540233 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2043ff0a-8a71-4b71-9750-beb566a12bad-tls-certs" pod "2043ff0a-8a71-4b71-9750-beb566a12bad" (UID: "2043ff0a-8a71-4b71-9750-beb566a12bad"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:52:24.540388 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:24.540350 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2043ff0a-8a71-4b71-9750-beb566a12bad-kube-api-access-n6tc7" pod "2043ff0a-8a71-4b71-9750-beb566a12bad" (UID: "2043ff0a-8a71-4b71-9750-beb566a12bad"). InnerVolumeSpecName "kube-api-access-n6tc7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:52:24.543049 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:24.543019 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2043ff0a-8a71-4b71-9750-beb566a12bad-dshm" pod "2043ff0a-8a71-4b71-9750-beb566a12bad" (UID: "2043ff0a-8a71-4b71-9750-beb566a12bad"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:52:24.639536 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:24.639506 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6tc7\" (UniqueName: \"kubernetes.io/projected/2043ff0a-8a71-4b71-9750-beb566a12bad-kube-api-access-n6tc7\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:52:24.639536 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:24.639536 5916 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2043ff0a-8a71-4b71-9750-beb566a12bad-tls-certs\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:52:24.639730 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:24.639548 5916 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2043ff0a-8a71-4b71-9750-beb566a12bad-dshm\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:52:25.024613 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:25.024527 5916 generic.go:356] "Generic (PLEG): container finished" podID="1a7c7a03-e0cf-44ca-8eee-6c3e16112b3a" containerID="b217ac5694f6fcc460c824d3050dbdc22c73771f5122ce3909539b5d333e98f5" exitCode=0 Nov 02 06:52:25.024613 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:25.024569 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-bdcd80/distinct-w0-bdcd80-kserve-5d9df79d48-bn5d4" Nov 02 06:52:25.024862 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:25.024605 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-bdcd80/distinct-w0-bdcd80-kserve-5d9df79d48-bn5d4" event={"ID":"1a7c7a03-e0cf-44ca-8eee-6c3e16112b3a","Type":"ContainerDied","Data":"b217ac5694f6fcc460c824d3050dbdc22c73771f5122ce3909539b5d333e98f5"} Nov 02 06:52:25.024862 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:25.024646 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-bdcd80/distinct-w0-bdcd80-kserve-5d9df79d48-bn5d4" event={"ID":"1a7c7a03-e0cf-44ca-8eee-6c3e16112b3a","Type":"ContainerDied","Data":"6e34c8f347ae1df192fbcaaea6847044f0477d7133ddae164eff387725face5d"} Nov 02 06:52:25.024862 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:25.024662 5916 scope.go:122] "RemoveContainer" containerID="b217ac5694f6fcc460c824d3050dbdc22c73771f5122ce3909539b5d333e98f5" Nov 02 06:52:25.026265 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:25.026239 5916 generic.go:356] "Generic (PLEG): container finished" podID="2043ff0a-8a71-4b71-9750-beb566a12bad" containerID="4d71cefed7597633799490ffef0d1abf6c665ef7b6629a4e265df95957a34be7" exitCode=0 Nov 02 06:52:25.026404 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:25.026284 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-bdcd80/simulator-w0-bdcd80-kserve-5749c847cb-9ktcr" Nov 02 06:52:25.026404 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:25.026327 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-bdcd80/simulator-w0-bdcd80-kserve-5749c847cb-9ktcr" event={"ID":"2043ff0a-8a71-4b71-9750-beb566a12bad","Type":"ContainerDied","Data":"4d71cefed7597633799490ffef0d1abf6c665ef7b6629a4e265df95957a34be7"} Nov 02 06:52:25.026404 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:25.026355 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-bdcd80/simulator-w0-bdcd80-kserve-5749c847cb-9ktcr" event={"ID":"2043ff0a-8a71-4b71-9750-beb566a12bad","Type":"ContainerDied","Data":"cd78adcea75975059a82e0ce6d43dbc9556dae2594f86432e3fbc7653697c8a8"} Nov 02 06:52:25.037962 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:25.037942 5916 scope.go:122] "RemoveContainer" containerID="b217ac5694f6fcc460c824d3050dbdc22c73771f5122ce3909539b5d333e98f5" Nov 02 06:52:25.038233 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:52:25.038215 5916 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b217ac5694f6fcc460c824d3050dbdc22c73771f5122ce3909539b5d333e98f5\": container with ID starting with b217ac5694f6fcc460c824d3050dbdc22c73771f5122ce3909539b5d333e98f5 not found: ID does not exist" containerID="b217ac5694f6fcc460c824d3050dbdc22c73771f5122ce3909539b5d333e98f5" Nov 02 06:52:25.038574 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:25.038240 5916 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b217ac5694f6fcc460c824d3050dbdc22c73771f5122ce3909539b5d333e98f5"} err="failed to get container status \"b217ac5694f6fcc460c824d3050dbdc22c73771f5122ce3909539b5d333e98f5\": rpc error: code = NotFound desc = could not find container \"b217ac5694f6fcc460c824d3050dbdc22c73771f5122ce3909539b5d333e98f5\": container with ID starting with b217ac5694f6fcc460c824d3050dbdc22c73771f5122ce3909539b5d333e98f5 not found: ID does not exist" Nov 02 06:52:25.038574 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:25.038257 5916 scope.go:122] "RemoveContainer" containerID="4d71cefed7597633799490ffef0d1abf6c665ef7b6629a4e265df95957a34be7" Nov 02 06:52:25.049185 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:25.049005 5916 scope.go:122] "RemoveContainer" containerID="4d71cefed7597633799490ffef0d1abf6c665ef7b6629a4e265df95957a34be7" Nov 02 06:52:25.052383 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:52:25.052356 5916 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d71cefed7597633799490ffef0d1abf6c665ef7b6629a4e265df95957a34be7\": container with ID starting with 4d71cefed7597633799490ffef0d1abf6c665ef7b6629a4e265df95957a34be7 not found: ID does not exist" containerID="4d71cefed7597633799490ffef0d1abf6c665ef7b6629a4e265df95957a34be7" Nov 02 06:52:25.052477 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:25.052391 5916 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d71cefed7597633799490ffef0d1abf6c665ef7b6629a4e265df95957a34be7"} err="failed to get container status \"4d71cefed7597633799490ffef0d1abf6c665ef7b6629a4e265df95957a34be7\": rpc error: code = NotFound desc = could not find container \"4d71cefed7597633799490ffef0d1abf6c665ef7b6629a4e265df95957a34be7\": container with ID starting with 4d71cefed7597633799490ffef0d1abf6c665ef7b6629a4e265df95957a34be7 not found: ID does not exist" Nov 02 06:52:25.193056 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:25.193023 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-bdcd80/distinct-w0-bdcd80-kserve-5d9df79d48-bn5d4"] Nov 02 06:52:25.196283 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:25.196258 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-bdcd80/distinct-w0-bdcd80-kserve-5d9df79d48-bn5d4"] Nov 02 06:52:25.309625 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:25.309552 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-bdcd80/simulator-w0-bdcd80-kserve-5749c847cb-9ktcr"] Nov 02 06:52:25.314861 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:25.314835 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-bdcd80/simulator-w0-bdcd80-kserve-5749c847cb-9ktcr"] Nov 02 06:52:25.844126 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:25.844089 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a7c7a03-e0cf-44ca-8eee-6c3e16112b3a" path="/var/lib/kubelet/pods/1a7c7a03-e0cf-44ca-8eee-6c3e16112b3a/volumes" Nov 02 06:52:25.844699 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:25.844675 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2043ff0a-8a71-4b71-9750-beb566a12bad" path="/var/lib/kubelet/pods/2043ff0a-8a71-4b71-9750-beb566a12bad/volumes" Nov 02 06:52:43.711402 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:43.711368 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1f85e4b-kfhtc"] Nov 02 06:52:43.711994 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:43.711970 5916 cpu_manager.go:414] "containerMap: removing container" podUID="2043ff0a-8a71-4b71-9750-beb566a12bad" containerName="main" Nov 02 06:52:43.711994 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:43.711984 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2043ff0a-8a71-4b71-9750-beb566a12bad" containerName="main" Nov 02 06:52:43.712089 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:43.712030 5916 cpu_manager.go:414] "containerMap: removing container" podUID="1a7c7a03-e0cf-44ca-8eee-6c3e16112b3a" containerName="main" Nov 02 06:52:43.712089 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:43.712036 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a7c7a03-e0cf-44ca-8eee-6c3e16112b3a" containerName="main" Nov 02 06:52:43.712089 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:43.712050 5916 cpu_manager.go:414] "containerMap: removing container" podUID="d900cf8d-6342-4b18-8d40-f28ab6d347b8" containerName="revoke-keys" Nov 02 06:52:43.712089 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:43.712056 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d900cf8d-6342-4b18-8d40-f28ab6d347b8" containerName="revoke-keys" Nov 02 06:52:43.712212 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:43.712139 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="d900cf8d-6342-4b18-8d40-f28ab6d347b8" containerName="revoke-keys" Nov 02 06:52:43.712212 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:43.712150 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="2043ff0a-8a71-4b71-9750-beb566a12bad" containerName="main" Nov 02 06:52:43.712212 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:43.712163 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a7c7a03-e0cf-44ca-8eee-6c3e16112b3a" containerName="main" Nov 02 06:52:43.717062 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:43.717042 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1f85e4b-kfhtc" Nov 02 06:52:43.764282 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:43.764245 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27db47d9-kzr5h"] Nov 02 06:52:43.769184 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:43.769151 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27db47d9-kzr5h" Nov 02 06:52:43.771218 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:43.771192 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1f85e4b-kfhtc"] Nov 02 06:52:43.793814 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:43.793784 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pkc5\" (UniqueName: \"kubernetes.io/projected/bb70237e-4074-4947-a12f-8e45c9e5ecaf-kube-api-access-4pkc5\") pod \"maas-api-revoke-sub-27db47d9-kzr5h\" (UID: \"bb70237e-4074-4947-a12f-8e45c9e5ecaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27db47d9-kzr5h" Nov 02 06:52:43.794023 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:43.793976 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb70237e-4074-4947-a12f-8e45c9e5ecaf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-27db47d9-kzr5h\" (UID: \"bb70237e-4074-4947-a12f-8e45c9e5ecaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27db47d9-kzr5h" Nov 02 06:52:43.794101 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:43.794075 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgfp8\" (UniqueName: \"kubernetes.io/projected/04e1849e-57d9-4230-b35c-62a76cbd90bb-kube-api-access-mgfp8\") pod \"maas-api-revoke-sub-d1f85e4b-kfhtc\" (UID: \"04e1849e-57d9-4230-b35c-62a76cbd90bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1f85e4b-kfhtc" Nov 02 06:52:43.794344 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:43.794219 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04e1849e-57d9-4230-b35c-62a76cbd90bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d1f85e4b-kfhtc\" (UID: \"04e1849e-57d9-4230-b35c-62a76cbd90bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1f85e4b-kfhtc" Nov 02 06:52:43.794344 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:43.794319 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04e1849e-57d9-4230-b35c-62a76cbd90bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d1f85e4b-kfhtc\" (UID: \"04e1849e-57d9-4230-b35c-62a76cbd90bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1f85e4b-kfhtc" Nov 02 06:52:43.794587 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:43.794347 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb70237e-4074-4947-a12f-8e45c9e5ecaf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-27db47d9-kzr5h\" (UID: \"bb70237e-4074-4947-a12f-8e45c9e5ecaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27db47d9-kzr5h" Nov 02 06:52:43.810714 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:43.810602 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 06:52:43.810714 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:43.810645 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-68l4r\"" Nov 02 06:52:43.821640 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:43.821610 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-37b53e-765d6b9f5c-7pb64"] Nov 02 06:52:43.821866 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:43.821825 5916 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-37b53e-765d6b9f5c-7pb64" podUID="1db3e7e8-ea83-42fa-a1d0-88ea901f78aa" containerName="maas-api" containerID="cri-o://dcffa6b2a75bca2f17cfd038bc6f5480c60f4671bd24e955cdf06fe11f0d807e" gracePeriod=30 Nov 02 06:52:43.854025 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:43.851334 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27db47d9-kzr5h"] Nov 02 06:52:43.897047 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:43.897010 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04e1849e-57d9-4230-b35c-62a76cbd90bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d1f85e4b-kfhtc\" (UID: \"04e1849e-57d9-4230-b35c-62a76cbd90bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1f85e4b-kfhtc" Nov 02 06:52:43.897245 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:43.897055 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb70237e-4074-4947-a12f-8e45c9e5ecaf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-27db47d9-kzr5h\" (UID: \"bb70237e-4074-4947-a12f-8e45c9e5ecaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27db47d9-kzr5h" Nov 02 06:52:43.897245 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:43.897139 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pkc5\" (UniqueName: \"kubernetes.io/projected/bb70237e-4074-4947-a12f-8e45c9e5ecaf-kube-api-access-4pkc5\") pod \"maas-api-revoke-sub-27db47d9-kzr5h\" (UID: \"bb70237e-4074-4947-a12f-8e45c9e5ecaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27db47d9-kzr5h" Nov 02 06:52:43.897368 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:43.897275 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb70237e-4074-4947-a12f-8e45c9e5ecaf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-27db47d9-kzr5h\" (UID: \"bb70237e-4074-4947-a12f-8e45c9e5ecaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27db47d9-kzr5h" Nov 02 06:52:43.897368 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:43.897329 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgfp8\" (UniqueName: \"kubernetes.io/projected/04e1849e-57d9-4230-b35c-62a76cbd90bb-kube-api-access-mgfp8\") pod \"maas-api-revoke-sub-d1f85e4b-kfhtc\" (UID: \"04e1849e-57d9-4230-b35c-62a76cbd90bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1f85e4b-kfhtc" Nov 02 06:52:43.897478 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:43.897392 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04e1849e-57d9-4230-b35c-62a76cbd90bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d1f85e4b-kfhtc\" (UID: \"04e1849e-57d9-4230-b35c-62a76cbd90bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1f85e4b-kfhtc" Nov 02 06:52:43.897883 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:43.897845 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04e1849e-57d9-4230-b35c-62a76cbd90bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d1f85e4b-kfhtc\" (UID: \"04e1849e-57d9-4230-b35c-62a76cbd90bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1f85e4b-kfhtc" Nov 02 06:52:43.898156 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:43.898140 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb70237e-4074-4947-a12f-8e45c9e5ecaf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-27db47d9-kzr5h\" (UID: \"bb70237e-4074-4947-a12f-8e45c9e5ecaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27db47d9-kzr5h" Nov 02 06:52:43.900249 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:43.900221 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04e1849e-57d9-4230-b35c-62a76cbd90bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d1f85e4b-kfhtc\" (UID: \"04e1849e-57d9-4230-b35c-62a76cbd90bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1f85e4b-kfhtc" Nov 02 06:52:43.900351 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:43.900291 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb70237e-4074-4947-a12f-8e45c9e5ecaf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-27db47d9-kzr5h\" (UID: \"bb70237e-4074-4947-a12f-8e45c9e5ecaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27db47d9-kzr5h" Nov 02 06:52:43.954723 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:43.954689 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pkc5\" (UniqueName: \"kubernetes.io/projected/bb70237e-4074-4947-a12f-8e45c9e5ecaf-kube-api-access-4pkc5\") pod \"maas-api-revoke-sub-27db47d9-kzr5h\" (UID: \"bb70237e-4074-4947-a12f-8e45c9e5ecaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27db47d9-kzr5h" Nov 02 06:52:43.954849 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:43.954738 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgfp8\" (UniqueName: \"kubernetes.io/projected/04e1849e-57d9-4230-b35c-62a76cbd90bb-kube-api-access-mgfp8\") pod \"maas-api-revoke-sub-d1f85e4b-kfhtc\" (UID: \"04e1849e-57d9-4230-b35c-62a76cbd90bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1f85e4b-kfhtc" Nov 02 06:52:44.044456 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:44.034626 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1f85e4b-kfhtc" Nov 02 06:52:44.101570 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:44.101540 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27db47d9-kzr5h" Nov 02 06:52:44.176380 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:44.176348 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-37b53e-765d6b9f5c-7pb64" Nov 02 06:52:44.188340 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:44.187990 5916 generic.go:356] "Generic (PLEG): container finished" podID="1db3e7e8-ea83-42fa-a1d0-88ea901f78aa" containerID="dcffa6b2a75bca2f17cfd038bc6f5480c60f4671bd24e955cdf06fe11f0d807e" exitCode=0 Nov 02 06:52:44.188340 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:44.188086 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-37b53e-765d6b9f5c-7pb64" Nov 02 06:52:44.188340 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:44.188137 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-37b53e-765d6b9f5c-7pb64" event={"ID":"1db3e7e8-ea83-42fa-a1d0-88ea901f78aa","Type":"ContainerDied","Data":"dcffa6b2a75bca2f17cfd038bc6f5480c60f4671bd24e955cdf06fe11f0d807e"} Nov 02 06:52:44.188340 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:44.188183 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-37b53e-765d6b9f5c-7pb64" event={"ID":"1db3e7e8-ea83-42fa-a1d0-88ea901f78aa","Type":"ContainerDied","Data":"7c872318fcebdd9ce33a5064d00ccca4ca35c093104d18a6524379159f98d68a"} Nov 02 06:52:44.188340 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:44.188210 5916 scope.go:122] "RemoveContainer" containerID="dcffa6b2a75bca2f17cfd038bc6f5480c60f4671bd24e955cdf06fe11f0d807e" Nov 02 06:52:44.273413 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:44.272451 5916 scope.go:122] "RemoveContainer" containerID="dcffa6b2a75bca2f17cfd038bc6f5480c60f4671bd24e955cdf06fe11f0d807e" Nov 02 06:52:44.279260 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:52:44.279140 5916 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dcffa6b2a75bca2f17cfd038bc6f5480c60f4671bd24e955cdf06fe11f0d807e\": container with ID starting with dcffa6b2a75bca2f17cfd038bc6f5480c60f4671bd24e955cdf06fe11f0d807e not found: ID does not exist" containerID="dcffa6b2a75bca2f17cfd038bc6f5480c60f4671bd24e955cdf06fe11f0d807e" Nov 02 06:52:44.279260 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:44.279182 5916 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dcffa6b2a75bca2f17cfd038bc6f5480c60f4671bd24e955cdf06fe11f0d807e"} err="failed to get container status \"dcffa6b2a75bca2f17cfd038bc6f5480c60f4671bd24e955cdf06fe11f0d807e\": rpc error: code = NotFound desc = could not find container \"dcffa6b2a75bca2f17cfd038bc6f5480c60f4671bd24e955cdf06fe11f0d807e\": container with ID starting with dcffa6b2a75bca2f17cfd038bc6f5480c60f4671bd24e955cdf06fe11f0d807e not found: ID does not exist" Nov 02 06:52:44.332987 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:44.332952 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1db3e7e8-ea83-42fa-a1d0-88ea901f78aa-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1db3e7e8-ea83-42fa-a1d0-88ea901f78aa-maas-api-tls\") pod \"1db3e7e8-ea83-42fa-a1d0-88ea901f78aa\" (UID: \"1db3e7e8-ea83-42fa-a1d0-88ea901f78aa\") " Nov 02 06:52:44.333221 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:44.333016 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1db3e7e8-ea83-42fa-a1d0-88ea901f78aa-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1db3e7e8-ea83-42fa-a1d0-88ea901f78aa-metrics-certs\") pod \"1db3e7e8-ea83-42fa-a1d0-88ea901f78aa\" (UID: \"1db3e7e8-ea83-42fa-a1d0-88ea901f78aa\") " Nov 02 06:52:44.333306 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:44.333232 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1db3e7e8-ea83-42fa-a1d0-88ea901f78aa-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1db3e7e8-ea83-42fa-a1d0-88ea901f78aa-openshift-service-ca\") pod \"1db3e7e8-ea83-42fa-a1d0-88ea901f78aa\" (UID: \"1db3e7e8-ea83-42fa-a1d0-88ea901f78aa\") " Nov 02 06:52:44.333370 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:44.333308 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1db3e7e8-ea83-42fa-a1d0-88ea901f78aa-kube-api-access-kf5px\" (UniqueName: \"kubernetes.io/projected/1db3e7e8-ea83-42fa-a1d0-88ea901f78aa-kube-api-access-kf5px\") pod \"1db3e7e8-ea83-42fa-a1d0-88ea901f78aa\" (UID: \"1db3e7e8-ea83-42fa-a1d0-88ea901f78aa\") " Nov 02 06:52:44.334148 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:44.334117 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1db3e7e8-ea83-42fa-a1d0-88ea901f78aa-openshift-service-ca" pod "1db3e7e8-ea83-42fa-a1d0-88ea901f78aa" (UID: "1db3e7e8-ea83-42fa-a1d0-88ea901f78aa"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:52:44.339937 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:44.339860 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1db3e7e8-ea83-42fa-a1d0-88ea901f78aa-maas-api-tls" pod "1db3e7e8-ea83-42fa-a1d0-88ea901f78aa" (UID: "1db3e7e8-ea83-42fa-a1d0-88ea901f78aa"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:52:44.343117 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:44.343078 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1db3e7e8-ea83-42fa-a1d0-88ea901f78aa-metrics-certs" pod "1db3e7e8-ea83-42fa-a1d0-88ea901f78aa" (UID: "1db3e7e8-ea83-42fa-a1d0-88ea901f78aa"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:52:44.349700 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:44.349636 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1db3e7e8-ea83-42fa-a1d0-88ea901f78aa-kube-api-access-kf5px" pod "1db3e7e8-ea83-42fa-a1d0-88ea901f78aa" (UID: "1db3e7e8-ea83-42fa-a1d0-88ea901f78aa"). InnerVolumeSpecName "kube-api-access-kf5px". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:52:44.435104 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:44.435006 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1db3e7e8-ea83-42fa-a1d0-88ea901f78aa-maas-api-tls\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:52:44.435104 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:44.435045 5916 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1db3e7e8-ea83-42fa-a1d0-88ea901f78aa-metrics-certs\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:52:44.435104 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:44.435060 5916 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1db3e7e8-ea83-42fa-a1d0-88ea901f78aa-openshift-service-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:52:44.435104 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:44.435076 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kf5px\" (UniqueName: \"kubernetes.io/projected/1db3e7e8-ea83-42fa-a1d0-88ea901f78aa-kube-api-access-kf5px\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:52:44.593562 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:44.593533 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1f85e4b-kfhtc"] Nov 02 06:52:44.594961 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:52:44.594935 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04e1849e_57d9_4230_b35c_62a76cbd90bb.slice/crio-cd93eaa2552d2c16fdd593335b9f6d0c626e4cec0baf382407b787e55a3a2da1 WatchSource:0}: Error finding container cd93eaa2552d2c16fdd593335b9f6d0c626e4cec0baf382407b787e55a3a2da1: Status 404 returned error can't find the container with id cd93eaa2552d2c16fdd593335b9f6d0c626e4cec0baf382407b787e55a3a2da1 Nov 02 06:52:44.610159 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:44.610136 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-37b53e-765d6b9f5c-7pb64"] Nov 02 06:52:44.614270 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:44.614248 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-37b53e-765d6b9f5c-7pb64"] Nov 02 06:52:44.643865 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:44.643805 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27db47d9-kzr5h"] Nov 02 06:52:44.646603 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:52:44.646561 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb70237e_4074_4947_a12f_8e45c9e5ecaf.slice/crio-2c0dc7c458716c331e7bae7d4b4a612e6fc6d807c2587603a2468580394ee3b5 WatchSource:0}: Error finding container 2c0dc7c458716c331e7bae7d4b4a612e6fc6d807c2587603a2468580394ee3b5: Status 404 returned error can't find the container with id 2c0dc7c458716c331e7bae7d4b4a612e6fc6d807c2587603a2468580394ee3b5 Nov 02 06:52:45.197689 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:45.197642 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27db47d9-kzr5h" event={"ID":"bb70237e-4074-4947-a12f-8e45c9e5ecaf","Type":"ContainerStarted","Data":"281e3aba2f06a1ee9a069e25cfd4da779287572122a8dc092de91392b0cb0d97"} Nov 02 06:52:45.198193 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:45.197746 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27db47d9-kzr5h" event={"ID":"bb70237e-4074-4947-a12f-8e45c9e5ecaf","Type":"ContainerStarted","Data":"2c0dc7c458716c331e7bae7d4b4a612e6fc6d807c2587603a2468580394ee3b5"} Nov 02 06:52:45.199363 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:45.199337 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1f85e4b-kfhtc" event={"ID":"04e1849e-57d9-4230-b35c-62a76cbd90bb","Type":"ContainerStarted","Data":"be7ee466641f270568ad09ba9a0eb619474b27969d1443a16e6aa6111322a0bb"} Nov 02 06:52:45.199476 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:45.199377 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1f85e4b-kfhtc" event={"ID":"04e1849e-57d9-4230-b35c-62a76cbd90bb","Type":"ContainerStarted","Data":"cd93eaa2552d2c16fdd593335b9f6d0c626e4cec0baf382407b787e55a3a2da1"} Nov 02 06:52:45.316009 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:45.315962 5916 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27db47d9-kzr5h" podStartSLOduration=2.315948843 podStartE2EDuration="2.315948843s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:52:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:52:45.314719005 +0000 UTC m=+2153.586539400" watchObservedRunningTime="2026-11-02 06:52:45.315948843 +0000 UTC m=+2153.587769236" Nov 02 06:52:45.436872 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:45.436820 5916 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1f85e4b-kfhtc" podStartSLOduration=2.436805875 podStartE2EDuration="2.436805875s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:52:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:52:45.433165913 +0000 UTC m=+2153.704986307" watchObservedRunningTime="2026-11-02 06:52:45.436805875 +0000 UTC m=+2153.708626307" Nov 02 06:52:45.845339 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:45.845306 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1db3e7e8-ea83-42fa-a1d0-88ea901f78aa" path="/var/lib/kubelet/pods/1db3e7e8-ea83-42fa-a1d0-88ea901f78aa/volumes" Nov 02 06:52:45.943173 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:45.942984 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-37b53e-gvlqd"] Nov 02 06:52:45.955095 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:45.955041 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-37b53e-gvlqd"] Nov 02 06:52:46.212883 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:46.212788 5916 generic.go:356] "Generic (PLEG): container finished" podID="bb70237e-4074-4947-a12f-8e45c9e5ecaf" containerID="281e3aba2f06a1ee9a069e25cfd4da779287572122a8dc092de91392b0cb0d97" exitCode=0 Nov 02 06:52:46.213569 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:46.212876 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27db47d9-kzr5h" event={"ID":"bb70237e-4074-4947-a12f-8e45c9e5ecaf","Type":"ContainerDied","Data":"281e3aba2f06a1ee9a069e25cfd4da779287572122a8dc092de91392b0cb0d97"} Nov 02 06:52:46.214811 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:46.214785 5916 generic.go:356] "Generic (PLEG): container finished" podID="04e1849e-57d9-4230-b35c-62a76cbd90bb" containerID="be7ee466641f270568ad09ba9a0eb619474b27969d1443a16e6aa6111322a0bb" exitCode=0 Nov 02 06:52:46.214951 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:46.214863 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1f85e4b-kfhtc" event={"ID":"04e1849e-57d9-4230-b35c-62a76cbd90bb","Type":"ContainerDied","Data":"be7ee466641f270568ad09ba9a0eb619474b27969d1443a16e6aa6111322a0bb"} Nov 02 06:52:47.445191 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:47.445162 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27db47d9-kzr5h" Nov 02 06:52:47.448370 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:47.448346 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1f85e4b-kfhtc" Nov 02 06:52:47.480489 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:47.480424 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bb70237e-4074-4947-a12f-8e45c9e5ecaf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb70237e-4074-4947-a12f-8e45c9e5ecaf-maas-api-xks-ca\") pod \"bb70237e-4074-4947-a12f-8e45c9e5ecaf\" (UID: \"bb70237e-4074-4947-a12f-8e45c9e5ecaf\") " Nov 02 06:52:47.480639 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:47.480524 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04e1849e-57d9-4230-b35c-62a76cbd90bb-kube-api-access-mgfp8\" (UniqueName: \"kubernetes.io/projected/04e1849e-57d9-4230-b35c-62a76cbd90bb-kube-api-access-mgfp8\") pod \"04e1849e-57d9-4230-b35c-62a76cbd90bb\" (UID: \"04e1849e-57d9-4230-b35c-62a76cbd90bb\") " Nov 02 06:52:47.480639 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:47.480611 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb70237e-4074-4947-a12f-8e45c9e5ecaf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb70237e-4074-4947-a12f-8e45c9e5ecaf-maas-api-openshift-ca\") pod \"bb70237e-4074-4947-a12f-8e45c9e5ecaf\" (UID: \"bb70237e-4074-4947-a12f-8e45c9e5ecaf\") " Nov 02 06:52:47.480759 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:47.480657 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb70237e-4074-4947-a12f-8e45c9e5ecaf-kube-api-access-4pkc5\" (UniqueName: \"kubernetes.io/projected/bb70237e-4074-4947-a12f-8e45c9e5ecaf-kube-api-access-4pkc5\") pod \"bb70237e-4074-4947-a12f-8e45c9e5ecaf\" (UID: \"bb70237e-4074-4947-a12f-8e45c9e5ecaf\") " Nov 02 06:52:47.480759 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:47.480712 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/04e1849e-57d9-4230-b35c-62a76cbd90bb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04e1849e-57d9-4230-b35c-62a76cbd90bb-maas-api-xks-ca\") pod \"04e1849e-57d9-4230-b35c-62a76cbd90bb\" (UID: \"04e1849e-57d9-4230-b35c-62a76cbd90bb\") " Nov 02 06:52:47.480836 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:47.480768 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04e1849e-57d9-4230-b35c-62a76cbd90bb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04e1849e-57d9-4230-b35c-62a76cbd90bb-maas-api-openshift-ca\") pod \"04e1849e-57d9-4230-b35c-62a76cbd90bb\" (UID: \"04e1849e-57d9-4230-b35c-62a76cbd90bb\") " Nov 02 06:52:47.481189 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:47.480978 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb70237e-4074-4947-a12f-8e45c9e5ecaf-maas-api-openshift-ca" pod "bb70237e-4074-4947-a12f-8e45c9e5ecaf" (UID: "bb70237e-4074-4947-a12f-8e45c9e5ecaf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:52:47.481306 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:47.481276 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04e1849e-57d9-4230-b35c-62a76cbd90bb-maas-api-openshift-ca" pod "04e1849e-57d9-4230-b35c-62a76cbd90bb" (UID: "04e1849e-57d9-4230-b35c-62a76cbd90bb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:52:47.481675 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:47.481462 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb70237e-4074-4947-a12f-8e45c9e5ecaf-maas-api-openshift-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:52:47.481675 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:47.481482 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04e1849e-57d9-4230-b35c-62a76cbd90bb-maas-api-openshift-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:52:47.482866 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:47.482830 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb70237e-4074-4947-a12f-8e45c9e5ecaf-maas-api-xks-ca" pod "bb70237e-4074-4947-a12f-8e45c9e5ecaf" (UID: "bb70237e-4074-4947-a12f-8e45c9e5ecaf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:52:47.483302 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:47.483278 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04e1849e-57d9-4230-b35c-62a76cbd90bb-maas-api-xks-ca" pod "04e1849e-57d9-4230-b35c-62a76cbd90bb" (UID: "04e1849e-57d9-4230-b35c-62a76cbd90bb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:52:47.483570 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:47.483440 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04e1849e-57d9-4230-b35c-62a76cbd90bb-kube-api-access-mgfp8" pod "04e1849e-57d9-4230-b35c-62a76cbd90bb" (UID: "04e1849e-57d9-4230-b35c-62a76cbd90bb"). InnerVolumeSpecName "kube-api-access-mgfp8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:52:47.483570 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:47.483534 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb70237e-4074-4947-a12f-8e45c9e5ecaf-kube-api-access-4pkc5" pod "bb70237e-4074-4947-a12f-8e45c9e5ecaf" (UID: "bb70237e-4074-4947-a12f-8e45c9e5ecaf"). InnerVolumeSpecName "kube-api-access-4pkc5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:52:47.582833 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:47.582797 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04e1849e-57d9-4230-b35c-62a76cbd90bb-maas-api-xks-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:52:47.582833 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:47.582826 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb70237e-4074-4947-a12f-8e45c9e5ecaf-maas-api-xks-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:52:47.582833 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:47.582836 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mgfp8\" (UniqueName: \"kubernetes.io/projected/04e1849e-57d9-4230-b35c-62a76cbd90bb-kube-api-access-mgfp8\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:52:47.583185 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:47.582846 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4pkc5\" (UniqueName: \"kubernetes.io/projected/bb70237e-4074-4947-a12f-8e45c9e5ecaf-kube-api-access-4pkc5\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:52:47.844382 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:47.844288 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71c05684-cf04-46f4-8321-a211172c839e" path="/var/lib/kubelet/pods/71c05684-cf04-46f4-8321-a211172c839e/volumes" Nov 02 06:52:48.232853 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:48.232822 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1f85e4b-kfhtc" Nov 02 06:52:48.233061 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:48.232857 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1f85e4b-kfhtc" event={"ID":"04e1849e-57d9-4230-b35c-62a76cbd90bb","Type":"ContainerDied","Data":"cd93eaa2552d2c16fdd593335b9f6d0c626e4cec0baf382407b787e55a3a2da1"} Nov 02 06:52:48.233061 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:48.232934 5916 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cd93eaa2552d2c16fdd593335b9f6d0c626e4cec0baf382407b787e55a3a2da1" Nov 02 06:52:48.234777 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:48.234749 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27db47d9-kzr5h" event={"ID":"bb70237e-4074-4947-a12f-8e45c9e5ecaf","Type":"ContainerDied","Data":"2c0dc7c458716c331e7bae7d4b4a612e6fc6d807c2587603a2468580394ee3b5"} Nov 02 06:52:48.234777 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:48.234791 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27db47d9-kzr5h" Nov 02 06:52:48.234777 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:48.234798 5916 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2c0dc7c458716c331e7bae7d4b4a612e6fc6d807c2587603a2468580394ee3b5" Nov 02 06:52:49.894843 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:52:49.894799 5916 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p98cq" containerName="extractor" Nov 02 06:52:53.870321 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:53.870286 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-dcf9c3-xt2lx"] Nov 02 06:52:53.871207 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:53.871188 5916 cpu_manager.go:414] "containerMap: removing container" podUID="1db3e7e8-ea83-42fa-a1d0-88ea901f78aa" containerName="maas-api" Nov 02 06:52:53.871269 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:53.871210 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1db3e7e8-ea83-42fa-a1d0-88ea901f78aa" containerName="maas-api" Nov 02 06:52:53.871269 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:53.871227 5916 cpu_manager.go:414] "containerMap: removing container" podUID="bb70237e-4074-4947-a12f-8e45c9e5ecaf" containerName="revoke-keys" Nov 02 06:52:53.871269 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:53.871235 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb70237e-4074-4947-a12f-8e45c9e5ecaf" containerName="revoke-keys" Nov 02 06:52:53.871269 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:53.871259 5916 cpu_manager.go:414] "containerMap: removing container" podUID="04e1849e-57d9-4230-b35c-62a76cbd90bb" containerName="revoke-keys" Nov 02 06:52:53.871269 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:53.871268 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04e1849e-57d9-4230-b35c-62a76cbd90bb" containerName="revoke-keys" Nov 02 06:52:53.871413 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:53.871390 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="1db3e7e8-ea83-42fa-a1d0-88ea901f78aa" containerName="maas-api" Nov 02 06:52:53.871413 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:53.871400 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb70237e-4074-4947-a12f-8e45c9e5ecaf" containerName="revoke-keys" Nov 02 06:52:53.871471 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:53.871412 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="04e1849e-57d9-4230-b35c-62a76cbd90bb" containerName="revoke-keys" Nov 02 06:52:53.876731 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:53.876710 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-dcf9c3-xt2lx" Nov 02 06:52:53.930754 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:53.930717 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-dcf9c3-xt2lx"] Nov 02 06:52:53.975624 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:53.975590 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 06:52:53.976017 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:53.975995 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-68l4r\"" Nov 02 06:52:53.985449 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:53.985423 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnmfj\" (UniqueName: \"kubernetes.io/projected/18305d70-7fdf-4ffa-8d86-8c97ce8dea4d-kube-api-access-hnmfj\") pod \"maas-api-revoke-keys-e2e-worker-w5-dcf9c3-xt2lx\" (UID: \"18305d70-7fdf-4ffa-8d86-8c97ce8dea4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-dcf9c3-xt2lx" Nov 02 06:52:53.985546 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:53.985464 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18305d70-7fdf-4ffa-8d86-8c97ce8dea4d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-dcf9c3-xt2lx\" (UID: \"18305d70-7fdf-4ffa-8d86-8c97ce8dea4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-dcf9c3-xt2lx" Nov 02 06:52:53.985777 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:53.985747 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18305d70-7fdf-4ffa-8d86-8c97ce8dea4d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-dcf9c3-xt2lx\" (UID: \"18305d70-7fdf-4ffa-8d86-8c97ce8dea4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-dcf9c3-xt2lx" Nov 02 06:52:54.087017 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:54.086973 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18305d70-7fdf-4ffa-8d86-8c97ce8dea4d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-dcf9c3-xt2lx\" (UID: \"18305d70-7fdf-4ffa-8d86-8c97ce8dea4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-dcf9c3-xt2lx" Nov 02 06:52:54.087220 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:54.087119 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnmfj\" (UniqueName: \"kubernetes.io/projected/18305d70-7fdf-4ffa-8d86-8c97ce8dea4d-kube-api-access-hnmfj\") pod \"maas-api-revoke-keys-e2e-worker-w5-dcf9c3-xt2lx\" (UID: \"18305d70-7fdf-4ffa-8d86-8c97ce8dea4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-dcf9c3-xt2lx" Nov 02 06:52:54.087220 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:54.087162 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18305d70-7fdf-4ffa-8d86-8c97ce8dea4d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-dcf9c3-xt2lx\" (UID: \"18305d70-7fdf-4ffa-8d86-8c97ce8dea4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-dcf9c3-xt2lx" Nov 02 06:52:54.087872 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:54.087843 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18305d70-7fdf-4ffa-8d86-8c97ce8dea4d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-dcf9c3-xt2lx\" (UID: \"18305d70-7fdf-4ffa-8d86-8c97ce8dea4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-dcf9c3-xt2lx" Nov 02 06:52:54.089732 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:54.089699 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18305d70-7fdf-4ffa-8d86-8c97ce8dea4d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-dcf9c3-xt2lx\" (UID: \"18305d70-7fdf-4ffa-8d86-8c97ce8dea4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-dcf9c3-xt2lx" Nov 02 06:52:54.149466 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:54.149424 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnmfj\" (UniqueName: \"kubernetes.io/projected/18305d70-7fdf-4ffa-8d86-8c97ce8dea4d-kube-api-access-hnmfj\") pod \"maas-api-revoke-keys-e2e-worker-w5-dcf9c3-xt2lx\" (UID: \"18305d70-7fdf-4ffa-8d86-8c97ce8dea4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-dcf9c3-xt2lx" Nov 02 06:52:54.191369 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:54.191334 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-dcf9c3-xt2lx" Nov 02 06:52:54.575315 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:52:54.575279 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18305d70_7fdf_4ffa_8d86_8c97ce8dea4d.slice/crio-f3ff78dc3f96d90b6c6e84ac064517e7176759fff20ef9652e750ab32dd05ee4 WatchSource:0}: Error finding container f3ff78dc3f96d90b6c6e84ac064517e7176759fff20ef9652e750ab32dd05ee4: Status 404 returned error can't find the container with id f3ff78dc3f96d90b6c6e84ac064517e7176759fff20ef9652e750ab32dd05ee4 Nov 02 06:52:54.575607 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:54.575583 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-dcf9c3-xt2lx"] Nov 02 06:52:54.952608 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:54.952575 5916 scope.go:122] "RemoveContainer" containerID="5fdc52ba9477367a2a7cb6fe24e449629b9df2540c07030e4fc4997c648b534e" Nov 02 06:52:54.965405 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:54.965381 5916 scope.go:122] "RemoveContainer" containerID="e665c4a934ceb1ef67cd50b2b11ce22fe989ee8bbf86d6fb82c5e059b539f54d" Nov 02 06:52:55.293100 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:55.293056 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-dcf9c3-xt2lx" event={"ID":"18305d70-7fdf-4ffa-8d86-8c97ce8dea4d","Type":"ContainerStarted","Data":"92ef91855217a5c0da50a5c0e25a551020a3d90dc6f4c3450366294c1c4ad792"} Nov 02 06:52:55.293307 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:55.293116 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-dcf9c3-xt2lx" event={"ID":"18305d70-7fdf-4ffa-8d86-8c97ce8dea4d","Type":"ContainerStarted","Data":"f3ff78dc3f96d90b6c6e84ac064517e7176759fff20ef9652e750ab32dd05ee4"} Nov 02 06:52:55.412855 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:55.412799 5916 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-dcf9c3-xt2lx" podStartSLOduration=2.412785875 podStartE2EDuration="2.412785875s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:52:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:52:55.409952989 +0000 UTC m=+2163.681773388" watchObservedRunningTime="2026-11-02 06:52:55.412785875 +0000 UTC m=+2163.684606268" Nov 02 06:52:56.053090 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:56.053055 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-tzzcm"] Nov 02 06:52:56.065892 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:56.065857 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-5c92q"] Nov 02 06:52:56.066141 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:56.066113 5916 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-5c92q" podUID="15eca023-182b-433b-a2ba-38c0b4e8a1ec" containerName="istio-proxy" containerID="cri-o://2e631eb9f7e93acbef72ef7d30ef7417c936f9f3f96386eabbab20d7deb3f2b3" gracePeriod=30 Nov 02 06:52:56.302400 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:56.302366 5916 generic.go:356] "Generic (PLEG): container finished" podID="18305d70-7fdf-4ffa-8d86-8c97ce8dea4d" containerID="92ef91855217a5c0da50a5c0e25a551020a3d90dc6f4c3450366294c1c4ad792" exitCode=0 Nov 02 06:52:56.302546 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:56.302450 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-dcf9c3-xt2lx" event={"ID":"18305d70-7fdf-4ffa-8d86-8c97ce8dea4d","Type":"ContainerDied","Data":"92ef91855217a5c0da50a5c0e25a551020a3d90dc6f4c3450366294c1c4ad792"} Nov 02 06:52:56.302701 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:56.302680 5916 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-tzzcm" podUID="b64d8a67-82cc-4714-87b8-2181f9388f07" containerName="istio-proxy" containerID="cri-o://d27cea362f78aa9bdaeee118c6f93dce1368069453c17c2f286e136c4231f442" gracePeriod=30 Nov 02 06:52:57.457341 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:57.457312 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-dcf9c3-xt2lx" Nov 02 06:52:57.535610 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:57.535580 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18305d70-7fdf-4ffa-8d86-8c97ce8dea4d-kube-api-access-hnmfj\" (UniqueName: \"kubernetes.io/projected/18305d70-7fdf-4ffa-8d86-8c97ce8dea4d-kube-api-access-hnmfj\") pod \"18305d70-7fdf-4ffa-8d86-8c97ce8dea4d\" (UID: \"18305d70-7fdf-4ffa-8d86-8c97ce8dea4d\") " Nov 02 06:52:57.535789 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:57.535616 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18305d70-7fdf-4ffa-8d86-8c97ce8dea4d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18305d70-7fdf-4ffa-8d86-8c97ce8dea4d-maas-api-openshift-ca\") pod \"18305d70-7fdf-4ffa-8d86-8c97ce8dea4d\" (UID: \"18305d70-7fdf-4ffa-8d86-8c97ce8dea4d\") " Nov 02 06:52:57.535789 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:57.535646 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18305d70-7fdf-4ffa-8d86-8c97ce8dea4d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18305d70-7fdf-4ffa-8d86-8c97ce8dea4d-maas-api-xks-ca\") pod \"18305d70-7fdf-4ffa-8d86-8c97ce8dea4d\" (UID: \"18305d70-7fdf-4ffa-8d86-8c97ce8dea4d\") " Nov 02 06:52:57.536038 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:57.536014 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18305d70-7fdf-4ffa-8d86-8c97ce8dea4d-maas-api-openshift-ca" pod "18305d70-7fdf-4ffa-8d86-8c97ce8dea4d" (UID: "18305d70-7fdf-4ffa-8d86-8c97ce8dea4d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:52:57.536168 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:57.536149 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18305d70-7fdf-4ffa-8d86-8c97ce8dea4d-maas-api-openshift-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:52:57.538273 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:57.538251 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18305d70-7fdf-4ffa-8d86-8c97ce8dea4d-maas-api-xks-ca" pod "18305d70-7fdf-4ffa-8d86-8c97ce8dea4d" (UID: "18305d70-7fdf-4ffa-8d86-8c97ce8dea4d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:52:57.538355 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:57.538317 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18305d70-7fdf-4ffa-8d86-8c97ce8dea4d-kube-api-access-hnmfj" pod "18305d70-7fdf-4ffa-8d86-8c97ce8dea4d" (UID: "18305d70-7fdf-4ffa-8d86-8c97ce8dea4d"). InnerVolumeSpecName "kube-api-access-hnmfj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:52:57.637525 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:57.637495 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18305d70-7fdf-4ffa-8d86-8c97ce8dea4d-maas-api-xks-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:52:57.637525 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:57.637526 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hnmfj\" (UniqueName: \"kubernetes.io/projected/18305d70-7fdf-4ffa-8d86-8c97ce8dea4d-kube-api-access-hnmfj\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:52:58.310072 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:58.310032 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-dcf9c3/distinct-2-w5-dcf9c3-kserve-64747dbd8f-wlwng"] Nov 02 06:52:58.318577 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:58.318547 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-dcf9c3-xt2lx" event={"ID":"18305d70-7fdf-4ffa-8d86-8c97ce8dea4d","Type":"ContainerDied","Data":"f3ff78dc3f96d90b6c6e84ac064517e7176759fff20ef9652e750ab32dd05ee4"} Nov 02 06:52:58.318738 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:58.318602 5916 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f3ff78dc3f96d90b6c6e84ac064517e7176759fff20ef9652e750ab32dd05ee4" Nov 02 06:52:58.318738 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:52:58.318554 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-dcf9c3-xt2lx" Nov 02 06:53:01.363243 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:01.362957 5916 generic.go:356] "Generic (PLEG): container finished" podID="15eca023-182b-433b-a2ba-38c0b4e8a1ec" containerID="2e631eb9f7e93acbef72ef7d30ef7417c936f9f3f96386eabbab20d7deb3f2b3" exitCode=0 Nov 02 06:53:01.364640 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:01.363478 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-5c92q" event={"ID":"15eca023-182b-433b-a2ba-38c0b4e8a1ec","Type":"ContainerDied","Data":"2e631eb9f7e93acbef72ef7d30ef7417c936f9f3f96386eabbab20d7deb3f2b3"} Nov 02 06:53:01.603206 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:01.603132 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-5c92q" Nov 02 06:53:01.724012 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:01.723985 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-tzzcm" Nov 02 06:53:01.745289 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:01.745255 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/15eca023-182b-433b-a2ba-38c0b4e8a1ec-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15eca023-182b-433b-a2ba-38c0b4e8a1ec-istio-podinfo\") pod \"15eca023-182b-433b-a2ba-38c0b4e8a1ec\" (UID: \"15eca023-182b-433b-a2ba-38c0b4e8a1ec\") " Nov 02 06:53:01.745527 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:01.745510 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15eca023-182b-433b-a2ba-38c0b4e8a1ec-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15eca023-182b-433b-a2ba-38c0b4e8a1ec-credential-socket\") pod \"15eca023-182b-433b-a2ba-38c0b4e8a1ec\" (UID: \"15eca023-182b-433b-a2ba-38c0b4e8a1ec\") " Nov 02 06:53:01.745677 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:01.745663 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15eca023-182b-433b-a2ba-38c0b4e8a1ec-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15eca023-182b-433b-a2ba-38c0b4e8a1ec-workload-certs\") pod \"15eca023-182b-433b-a2ba-38c0b4e8a1ec\" (UID: \"15eca023-182b-433b-a2ba-38c0b4e8a1ec\") " Nov 02 06:53:01.745814 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:01.745801 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15eca023-182b-433b-a2ba-38c0b4e8a1ec-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15eca023-182b-433b-a2ba-38c0b4e8a1ec-workload-socket\") pod \"15eca023-182b-433b-a2ba-38c0b4e8a1ec\" (UID: \"15eca023-182b-433b-a2ba-38c0b4e8a1ec\") " Nov 02 06:53:01.746185 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:01.746165 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15eca023-182b-433b-a2ba-38c0b4e8a1ec-kube-api-access-dnks7\" (UniqueName: \"kubernetes.io/projected/15eca023-182b-433b-a2ba-38c0b4e8a1ec-kube-api-access-dnks7\") pod \"15eca023-182b-433b-a2ba-38c0b4e8a1ec\" (UID: \"15eca023-182b-433b-a2ba-38c0b4e8a1ec\") " Nov 02 06:53:01.746364 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:01.746347 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15eca023-182b-433b-a2ba-38c0b4e8a1ec-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15eca023-182b-433b-a2ba-38c0b4e8a1ec-istio-envoy\") pod \"15eca023-182b-433b-a2ba-38c0b4e8a1ec\" (UID: \"15eca023-182b-433b-a2ba-38c0b4e8a1ec\") " Nov 02 06:53:01.746483 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:01.746470 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15eca023-182b-433b-a2ba-38c0b4e8a1ec-istio-token\" (UniqueName: \"kubernetes.io/projected/15eca023-182b-433b-a2ba-38c0b4e8a1ec-istio-token\") pod \"15eca023-182b-433b-a2ba-38c0b4e8a1ec\" (UID: \"15eca023-182b-433b-a2ba-38c0b4e8a1ec\") " Nov 02 06:53:01.746586 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:01.746572 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15eca023-182b-433b-a2ba-38c0b4e8a1ec-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15eca023-182b-433b-a2ba-38c0b4e8a1ec-istiod-ca-cert\") pod \"15eca023-182b-433b-a2ba-38c0b4e8a1ec\" (UID: \"15eca023-182b-433b-a2ba-38c0b4e8a1ec\") " Nov 02 06:53:01.746819 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:01.746250 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15eca023-182b-433b-a2ba-38c0b4e8a1ec-workload-certs" pod "15eca023-182b-433b-a2ba-38c0b4e8a1ec" (UID: "15eca023-182b-433b-a2ba-38c0b4e8a1ec"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:53:01.746819 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:01.746698 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15eca023-182b-433b-a2ba-38c0b4e8a1ec-credential-socket" pod "15eca023-182b-433b-a2ba-38c0b4e8a1ec" (UID: "15eca023-182b-433b-a2ba-38c0b4e8a1ec"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:53:01.747014 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:01.746967 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15eca023-182b-433b-a2ba-38c0b4e8a1ec-workload-socket" pod "15eca023-182b-433b-a2ba-38c0b4e8a1ec" (UID: "15eca023-182b-433b-a2ba-38c0b4e8a1ec"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:53:01.747071 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:01.747052 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15eca023-182b-433b-a2ba-38c0b4e8a1ec-istiod-ca-cert" pod "15eca023-182b-433b-a2ba-38c0b4e8a1ec" (UID: "15eca023-182b-433b-a2ba-38c0b4e8a1ec"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:53:01.747365 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:01.747347 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15eca023-182b-433b-a2ba-38c0b4e8a1ec-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15eca023-182b-433b-a2ba-38c0b4e8a1ec-istio-data\") pod \"15eca023-182b-433b-a2ba-38c0b4e8a1ec\" (UID: \"15eca023-182b-433b-a2ba-38c0b4e8a1ec\") " Nov 02 06:53:01.748291 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:01.748269 5916 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15eca023-182b-433b-a2ba-38c0b4e8a1ec-istiod-ca-cert\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:53:01.748434 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:01.748420 5916 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15eca023-182b-433b-a2ba-38c0b4e8a1ec-credential-socket\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:53:01.748604 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:01.748592 5916 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15eca023-182b-433b-a2ba-38c0b4e8a1ec-workload-certs\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:53:01.748716 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:01.748705 5916 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15eca023-182b-433b-a2ba-38c0b4e8a1ec-workload-socket\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:53:01.748806 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:01.747670 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15eca023-182b-433b-a2ba-38c0b4e8a1ec-istio-data" pod "15eca023-182b-433b-a2ba-38c0b4e8a1ec" (UID: "15eca023-182b-433b-a2ba-38c0b4e8a1ec"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:53:01.749312 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:01.749276 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15eca023-182b-433b-a2ba-38c0b4e8a1ec-kube-api-access-dnks7" pod "15eca023-182b-433b-a2ba-38c0b4e8a1ec" (UID: "15eca023-182b-433b-a2ba-38c0b4e8a1ec"). InnerVolumeSpecName "kube-api-access-dnks7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:53:01.749423 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:01.749316 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15eca023-182b-433b-a2ba-38c0b4e8a1ec-istio-envoy" pod "15eca023-182b-433b-a2ba-38c0b4e8a1ec" (UID: "15eca023-182b-433b-a2ba-38c0b4e8a1ec"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:53:01.749486 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:01.749466 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15eca023-182b-433b-a2ba-38c0b4e8a1ec-istio-token" pod "15eca023-182b-433b-a2ba-38c0b4e8a1ec" (UID: "15eca023-182b-433b-a2ba-38c0b4e8a1ec"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:53:01.750876 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:01.750852 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/15eca023-182b-433b-a2ba-38c0b4e8a1ec-istio-podinfo" pod "15eca023-182b-433b-a2ba-38c0b4e8a1ec" (UID: "15eca023-182b-433b-a2ba-38c0b4e8a1ec"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 06:53:01.849948 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:01.849900 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b64d8a67-82cc-4714-87b8-2181f9388f07-istio-token\" (UniqueName: \"kubernetes.io/projected/b64d8a67-82cc-4714-87b8-2181f9388f07-istio-token\") pod \"b64d8a67-82cc-4714-87b8-2181f9388f07\" (UID: \"b64d8a67-82cc-4714-87b8-2181f9388f07\") " Nov 02 06:53:01.850303 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:01.850272 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b64d8a67-82cc-4714-87b8-2181f9388f07-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b64d8a67-82cc-4714-87b8-2181f9388f07-workload-socket\") pod \"b64d8a67-82cc-4714-87b8-2181f9388f07\" (UID: \"b64d8a67-82cc-4714-87b8-2181f9388f07\") " Nov 02 06:53:01.850430 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:01.850373 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b64d8a67-82cc-4714-87b8-2181f9388f07-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b64d8a67-82cc-4714-87b8-2181f9388f07-workload-certs\") pod \"b64d8a67-82cc-4714-87b8-2181f9388f07\" (UID: \"b64d8a67-82cc-4714-87b8-2181f9388f07\") " Nov 02 06:53:01.850491 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:01.850430 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b64d8a67-82cc-4714-87b8-2181f9388f07-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b64d8a67-82cc-4714-87b8-2181f9388f07-istio-data\") pod \"b64d8a67-82cc-4714-87b8-2181f9388f07\" (UID: \"b64d8a67-82cc-4714-87b8-2181f9388f07\") " Nov 02 06:53:01.850491 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:01.850459 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b64d8a67-82cc-4714-87b8-2181f9388f07-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b64d8a67-82cc-4714-87b8-2181f9388f07-istio-podinfo\") pod \"b64d8a67-82cc-4714-87b8-2181f9388f07\" (UID: \"b64d8a67-82cc-4714-87b8-2181f9388f07\") " Nov 02 06:53:01.850611 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:01.850488 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b64d8a67-82cc-4714-87b8-2181f9388f07-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b64d8a67-82cc-4714-87b8-2181f9388f07-istiod-ca-cert\") pod \"b64d8a67-82cc-4714-87b8-2181f9388f07\" (UID: \"b64d8a67-82cc-4714-87b8-2181f9388f07\") " Nov 02 06:53:01.850611 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:01.850575 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b64d8a67-82cc-4714-87b8-2181f9388f07-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b64d8a67-82cc-4714-87b8-2181f9388f07-credential-socket\") pod \"b64d8a67-82cc-4714-87b8-2181f9388f07\" (UID: \"b64d8a67-82cc-4714-87b8-2181f9388f07\") " Nov 02 06:53:01.850716 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:01.850613 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b64d8a67-82cc-4714-87b8-2181f9388f07-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b64d8a67-82cc-4714-87b8-2181f9388f07-istio-envoy\") pod \"b64d8a67-82cc-4714-87b8-2181f9388f07\" (UID: \"b64d8a67-82cc-4714-87b8-2181f9388f07\") " Nov 02 06:53:01.850716 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:01.850702 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b64d8a67-82cc-4714-87b8-2181f9388f07-kube-api-access-h5srz\" (UniqueName: \"kubernetes.io/projected/b64d8a67-82cc-4714-87b8-2181f9388f07-kube-api-access-h5srz\") pod \"b64d8a67-82cc-4714-87b8-2181f9388f07\" (UID: \"b64d8a67-82cc-4714-87b8-2181f9388f07\") " Nov 02 06:53:01.851177 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:01.851156 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnks7\" (UniqueName: \"kubernetes.io/projected/15eca023-182b-433b-a2ba-38c0b4e8a1ec-kube-api-access-dnks7\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:53:01.851248 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:01.851183 5916 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15eca023-182b-433b-a2ba-38c0b4e8a1ec-istio-envoy\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:53:01.851248 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:01.851198 5916 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15eca023-182b-433b-a2ba-38c0b4e8a1ec-istio-token\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:53:01.851248 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:01.851211 5916 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15eca023-182b-433b-a2ba-38c0b4e8a1ec-istio-data\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:53:01.851248 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:01.851225 5916 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15eca023-182b-433b-a2ba-38c0b4e8a1ec-istio-podinfo\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:53:01.852199 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:01.852171 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b64d8a67-82cc-4714-87b8-2181f9388f07-workload-socket" pod "b64d8a67-82cc-4714-87b8-2181f9388f07" (UID: "b64d8a67-82cc-4714-87b8-2181f9388f07"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:53:01.853140 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:01.852616 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b64d8a67-82cc-4714-87b8-2181f9388f07-credential-socket" pod "b64d8a67-82cc-4714-87b8-2181f9388f07" (UID: "b64d8a67-82cc-4714-87b8-2181f9388f07"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:53:01.853140 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:01.852700 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b64d8a67-82cc-4714-87b8-2181f9388f07-istiod-ca-cert" pod "b64d8a67-82cc-4714-87b8-2181f9388f07" (UID: "b64d8a67-82cc-4714-87b8-2181f9388f07"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:53:01.855835 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:01.855753 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b64d8a67-82cc-4714-87b8-2181f9388f07-istio-data" pod "b64d8a67-82cc-4714-87b8-2181f9388f07" (UID: "b64d8a67-82cc-4714-87b8-2181f9388f07"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:53:01.857022 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:01.856996 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b64d8a67-82cc-4714-87b8-2181f9388f07-workload-certs" pod "b64d8a67-82cc-4714-87b8-2181f9388f07" (UID: "b64d8a67-82cc-4714-87b8-2181f9388f07"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:53:01.860872 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:01.860549 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b64d8a67-82cc-4714-87b8-2181f9388f07-istio-podinfo" pod "b64d8a67-82cc-4714-87b8-2181f9388f07" (UID: "b64d8a67-82cc-4714-87b8-2181f9388f07"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 06:53:01.860872 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:01.860710 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b64d8a67-82cc-4714-87b8-2181f9388f07-istio-token" pod "b64d8a67-82cc-4714-87b8-2181f9388f07" (UID: "b64d8a67-82cc-4714-87b8-2181f9388f07"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:53:01.861271 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:01.861249 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b64d8a67-82cc-4714-87b8-2181f9388f07-kube-api-access-h5srz" pod "b64d8a67-82cc-4714-87b8-2181f9388f07" (UID: "b64d8a67-82cc-4714-87b8-2181f9388f07"). InnerVolumeSpecName "kube-api-access-h5srz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:53:01.861572 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:01.861554 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b64d8a67-82cc-4714-87b8-2181f9388f07-istio-envoy" pod "b64d8a67-82cc-4714-87b8-2181f9388f07" (UID: "b64d8a67-82cc-4714-87b8-2181f9388f07"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:53:01.952573 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:01.952529 5916 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b64d8a67-82cc-4714-87b8-2181f9388f07-credential-socket\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:53:01.952573 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:01.952566 5916 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b64d8a67-82cc-4714-87b8-2181f9388f07-istio-envoy\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:53:01.952573 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:01.952581 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5srz\" (UniqueName: \"kubernetes.io/projected/b64d8a67-82cc-4714-87b8-2181f9388f07-kube-api-access-h5srz\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:53:01.952887 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:01.952597 5916 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b64d8a67-82cc-4714-87b8-2181f9388f07-istio-token\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:53:01.952887 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:01.952615 5916 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b64d8a67-82cc-4714-87b8-2181f9388f07-workload-socket\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:53:01.952887 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:01.952629 5916 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b64d8a67-82cc-4714-87b8-2181f9388f07-workload-certs\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:53:01.952887 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:01.952645 5916 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b64d8a67-82cc-4714-87b8-2181f9388f07-istio-data\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:53:01.952887 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:01.952661 5916 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b64d8a67-82cc-4714-87b8-2181f9388f07-istio-podinfo\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:53:01.952887 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:01.952675 5916 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b64d8a67-82cc-4714-87b8-2181f9388f07-istiod-ca-cert\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:53:02.374698 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:02.374354 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-5c92q" Nov 02 06:53:02.374698 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:02.374387 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-5c92q" event={"ID":"15eca023-182b-433b-a2ba-38c0b4e8a1ec","Type":"ContainerDied","Data":"8aa6d8dc8af8b036297686a421554f116118df1cde24d673c34fa74c41e12015"} Nov 02 06:53:02.374698 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:02.374461 5916 scope.go:122] "RemoveContainer" containerID="2e631eb9f7e93acbef72ef7d30ef7417c936f9f3f96386eabbab20d7deb3f2b3" Nov 02 06:53:02.376418 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:02.376391 5916 generic.go:356] "Generic (PLEG): container finished" podID="b64d8a67-82cc-4714-87b8-2181f9388f07" containerID="d27cea362f78aa9bdaeee118c6f93dce1368069453c17c2f286e136c4231f442" exitCode=0 Nov 02 06:53:02.376537 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:02.376449 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-tzzcm" event={"ID":"b64d8a67-82cc-4714-87b8-2181f9388f07","Type":"ContainerDied","Data":"d27cea362f78aa9bdaeee118c6f93dce1368069453c17c2f286e136c4231f442"} Nov 02 06:53:02.376537 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:02.376457 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-tzzcm" Nov 02 06:53:02.376537 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:02.376490 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-tzzcm" event={"ID":"b64d8a67-82cc-4714-87b8-2181f9388f07","Type":"ContainerDied","Data":"d53139037d7e83c2390d25bfbc4565c1784eac48832d5cab06a438ba53d0b761"} Nov 02 06:53:02.387809 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:02.387783 5916 scope.go:122] "RemoveContainer" containerID="d27cea362f78aa9bdaeee118c6f93dce1368069453c17c2f286e136c4231f442" Nov 02 06:53:02.397973 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:02.397954 5916 scope.go:122] "RemoveContainer" containerID="d27cea362f78aa9bdaeee118c6f93dce1368069453c17c2f286e136c4231f442" Nov 02 06:53:02.398246 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:53:02.398224 5916 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d27cea362f78aa9bdaeee118c6f93dce1368069453c17c2f286e136c4231f442\": container with ID starting with d27cea362f78aa9bdaeee118c6f93dce1368069453c17c2f286e136c4231f442 not found: ID does not exist" containerID="d27cea362f78aa9bdaeee118c6f93dce1368069453c17c2f286e136c4231f442" Nov 02 06:53:02.398311 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:02.398256 5916 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d27cea362f78aa9bdaeee118c6f93dce1368069453c17c2f286e136c4231f442"} err="failed to get container status \"d27cea362f78aa9bdaeee118c6f93dce1368069453c17c2f286e136c4231f442\": rpc error: code = NotFound desc = could not find container \"d27cea362f78aa9bdaeee118c6f93dce1368069453c17c2f286e136c4231f442\": container with ID starting with d27cea362f78aa9bdaeee118c6f93dce1368069453c17c2f286e136c4231f442 not found: ID does not exist" Nov 02 06:53:02.549263 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:02.549231 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-5c92q"] Nov 02 06:53:02.599217 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:02.599180 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-5c92q"] Nov 02 06:53:02.709174 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:02.709142 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-tzzcm"] Nov 02 06:53:02.714168 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:02.714130 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-tzzcm"] Nov 02 06:53:03.846477 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:03.846444 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15eca023-182b-433b-a2ba-38c0b4e8a1ec" path="/var/lib/kubelet/pods/15eca023-182b-433b-a2ba-38c0b4e8a1ec/volumes" Nov 02 06:53:03.846980 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:03.846965 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b64d8a67-82cc-4714-87b8-2181f9388f07" path="/var/lib/kubelet/pods/b64d8a67-82cc-4714-87b8-2181f9388f07/volumes" Nov 02 06:53:05.305418 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:05.305387 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-dcf9c3-66797968df-l4z8j"] Nov 02 06:53:05.413509 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:05.413467 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-dcf9c3-694d5779d7-cwfqq"] Nov 02 06:53:10.306034 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:10.305979 5916 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w5-dcf9c3-66797968df-l4z8j" podUID="9b7d318a-026f-4857-904d-20713b54266c" containerName="payload-processing" containerID="cri-o://7355631c7872a7c15867a0737444c6536a06decaeec309a9b83b4967a3d81ce9" gracePeriod=25 Nov 02 06:53:10.414584 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:10.414531 5916 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-dcf9c3-694d5779d7-cwfqq" podUID="8205f559-100b-4fff-8e3c-4dea453db8e6" containerName="payload-pre-processing" containerID="cri-o://fc7f5391da15d863a20fe22c17206e8ac5c704b3011130ef09451f6dc0d915ea" gracePeriod=25 Nov 02 06:53:13.331995 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:13.331958 5916 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-dcf9c3/distinct-2-w5-dcf9c3-kserve-64747dbd8f-wlwng" podUID="e37d42ab-7b62-4f07-8a29-cdedeb0fe89a" containerName="main" containerID="cri-o://5099831e5bb3125d1e33570a3097a3425678fb23b95c4a134e7e94c45db9ed52" gracePeriod=45 Nov 02 06:53:13.474757 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:13.474712 5916 generic.go:356] "Generic (PLEG): container finished" podID="e37d42ab-7b62-4f07-8a29-cdedeb0fe89a" containerID="5099831e5bb3125d1e33570a3097a3425678fb23b95c4a134e7e94c45db9ed52" exitCode=0 Nov 02 06:53:13.475012 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:13.474813 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-dcf9c3/distinct-2-w5-dcf9c3-kserve-64747dbd8f-wlwng" event={"ID":"e37d42ab-7b62-4f07-8a29-cdedeb0fe89a","Type":"ContainerDied","Data":"5099831e5bb3125d1e33570a3097a3425678fb23b95c4a134e7e94c45db9ed52"} Nov 02 06:53:13.686672 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:13.686640 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-dcf9c3/distinct-2-w5-dcf9c3-kserve-64747dbd8f-wlwng" Nov 02 06:53:13.806457 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:13.806416 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-dcf9c3-66797968df-l4z8j" podUID="9b7d318a-026f-4857-904d-20713b54266c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 06:53:13.806692 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:13.806420 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-dcf9c3-694d5779d7-cwfqq" podUID="8205f559-100b-4fff-8e3c-4dea453db8e6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 06:53:13.875550 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:13.875514 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e37d42ab-7b62-4f07-8a29-cdedeb0fe89a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e37d42ab-7b62-4f07-8a29-cdedeb0fe89a-model-cache\") pod \"e37d42ab-7b62-4f07-8a29-cdedeb0fe89a\" (UID: \"e37d42ab-7b62-4f07-8a29-cdedeb0fe89a\") " Nov 02 06:53:13.875755 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:13.875563 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e37d42ab-7b62-4f07-8a29-cdedeb0fe89a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e37d42ab-7b62-4f07-8a29-cdedeb0fe89a-dshm\") pod \"e37d42ab-7b62-4f07-8a29-cdedeb0fe89a\" (UID: \"e37d42ab-7b62-4f07-8a29-cdedeb0fe89a\") " Nov 02 06:53:13.875755 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:13.875648 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e37d42ab-7b62-4f07-8a29-cdedeb0fe89a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e37d42ab-7b62-4f07-8a29-cdedeb0fe89a-tmp-dir\") pod \"e37d42ab-7b62-4f07-8a29-cdedeb0fe89a\" (UID: \"e37d42ab-7b62-4f07-8a29-cdedeb0fe89a\") " Nov 02 06:53:13.875755 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:13.875746 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e37d42ab-7b62-4f07-8a29-cdedeb0fe89a-kube-api-access-g5xvl\" (UniqueName: \"kubernetes.io/projected/e37d42ab-7b62-4f07-8a29-cdedeb0fe89a-kube-api-access-g5xvl\") pod \"e37d42ab-7b62-4f07-8a29-cdedeb0fe89a\" (UID: \"e37d42ab-7b62-4f07-8a29-cdedeb0fe89a\") " Nov 02 06:53:13.875971 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:13.875802 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e37d42ab-7b62-4f07-8a29-cdedeb0fe89a-home\" (UniqueName: \"kubernetes.io/empty-dir/e37d42ab-7b62-4f07-8a29-cdedeb0fe89a-home\") pod \"e37d42ab-7b62-4f07-8a29-cdedeb0fe89a\" (UID: \"e37d42ab-7b62-4f07-8a29-cdedeb0fe89a\") " Nov 02 06:53:13.875971 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:13.875837 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e37d42ab-7b62-4f07-8a29-cdedeb0fe89a-model-cache" pod "e37d42ab-7b62-4f07-8a29-cdedeb0fe89a" (UID: "e37d42ab-7b62-4f07-8a29-cdedeb0fe89a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:53:13.875971 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:13.875872 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e37d42ab-7b62-4f07-8a29-cdedeb0fe89a-tls-certs\" (UniqueName: \"kubernetes.io/secret/e37d42ab-7b62-4f07-8a29-cdedeb0fe89a-tls-certs\") pod \"e37d42ab-7b62-4f07-8a29-cdedeb0fe89a\" (UID: \"e37d42ab-7b62-4f07-8a29-cdedeb0fe89a\") " Nov 02 06:53:13.876470 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:13.876448 5916 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e37d42ab-7b62-4f07-8a29-cdedeb0fe89a-model-cache\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:53:13.876846 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:13.876812 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e37d42ab-7b62-4f07-8a29-cdedeb0fe89a-tmp-dir" pod "e37d42ab-7b62-4f07-8a29-cdedeb0fe89a" (UID: "e37d42ab-7b62-4f07-8a29-cdedeb0fe89a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:53:13.877045 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:13.877024 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e37d42ab-7b62-4f07-8a29-cdedeb0fe89a-home" pod "e37d42ab-7b62-4f07-8a29-cdedeb0fe89a" (UID: "e37d42ab-7b62-4f07-8a29-cdedeb0fe89a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:53:13.878948 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:13.878874 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e37d42ab-7b62-4f07-8a29-cdedeb0fe89a-tls-certs" pod "e37d42ab-7b62-4f07-8a29-cdedeb0fe89a" (UID: "e37d42ab-7b62-4f07-8a29-cdedeb0fe89a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:53:13.879998 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:13.879971 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e37d42ab-7b62-4f07-8a29-cdedeb0fe89a-dshm" pod "e37d42ab-7b62-4f07-8a29-cdedeb0fe89a" (UID: "e37d42ab-7b62-4f07-8a29-cdedeb0fe89a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:53:13.881098 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:13.881038 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e37d42ab-7b62-4f07-8a29-cdedeb0fe89a-kube-api-access-g5xvl" pod "e37d42ab-7b62-4f07-8a29-cdedeb0fe89a" (UID: "e37d42ab-7b62-4f07-8a29-cdedeb0fe89a"). InnerVolumeSpecName "kube-api-access-g5xvl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:53:13.978189 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:13.978158 5916 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e37d42ab-7b62-4f07-8a29-cdedeb0fe89a-tls-certs\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:53:13.978189 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:13.978186 5916 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e37d42ab-7b62-4f07-8a29-cdedeb0fe89a-dshm\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:53:13.978189 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:13.978195 5916 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e37d42ab-7b62-4f07-8a29-cdedeb0fe89a-tmp-dir\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:53:13.978423 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:13.978204 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5xvl\" (UniqueName: \"kubernetes.io/projected/e37d42ab-7b62-4f07-8a29-cdedeb0fe89a-kube-api-access-g5xvl\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:53:13.978423 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:13.978214 5916 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e37d42ab-7b62-4f07-8a29-cdedeb0fe89a-home\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:53:14.484059 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:14.484026 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-dcf9c3/distinct-2-w5-dcf9c3-kserve-64747dbd8f-wlwng" Nov 02 06:53:14.484555 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:14.484066 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-dcf9c3/distinct-2-w5-dcf9c3-kserve-64747dbd8f-wlwng" event={"ID":"e37d42ab-7b62-4f07-8a29-cdedeb0fe89a","Type":"ContainerDied","Data":"cb536d243ae5318c718f7a4792a345d35a101dec52019ff7febed153e464a89c"} Nov 02 06:53:14.484555 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:14.484126 5916 scope.go:122] "RemoveContainer" containerID="5099831e5bb3125d1e33570a3097a3425678fb23b95c4a134e7e94c45db9ed52" Nov 02 06:53:14.654476 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:14.654439 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-dcf9c3/distinct-2-w5-dcf9c3-kserve-64747dbd8f-wlwng"] Nov 02 06:53:14.704525 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:14.704490 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-dcf9c3/distinct-2-w5-dcf9c3-kserve-64747dbd8f-wlwng"] Nov 02 06:53:15.843645 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:15.843569 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e37d42ab-7b62-4f07-8a29-cdedeb0fe89a" path="/var/lib/kubelet/pods/e37d42ab-7b62-4f07-8a29-cdedeb0fe89a/volumes" Nov 02 06:53:19.894167 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:53:19.894130 5916 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p98cq" containerName="extractor" Nov 02 06:53:23.298296 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:23.298261 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e233a7-6l45k"] Nov 02 06:53:23.348257 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:23.348227 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e233a7-6l45k"] Nov 02 06:53:23.806385 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:23.806342 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-dcf9c3-66797968df-l4z8j" podUID="9b7d318a-026f-4857-904d-20713b54266c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 06:53:23.806562 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:23.806407 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-dcf9c3-694d5779d7-cwfqq" podUID="8205f559-100b-4fff-8e3c-4dea453db8e6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 06:53:23.843446 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:23.843409 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d900cf8d-6342-4b18-8d40-f28ab6d347b8" path="/var/lib/kubelet/pods/d900cf8d-6342-4b18-8d40-f28ab6d347b8/volumes" Nov 02 06:53:30.621318 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:30.621232 5916 generic.go:356] "Generic (PLEG): container finished" podID="8205f559-100b-4fff-8e3c-4dea453db8e6" containerID="fc7f5391da15d863a20fe22c17206e8ac5c704b3011130ef09451f6dc0d915ea" exitCode=0 Nov 02 06:53:30.622144 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:30.622113 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-dcf9c3-694d5779d7-cwfqq" event={"ID":"8205f559-100b-4fff-8e3c-4dea453db8e6","Type":"ContainerDied","Data":"fc7f5391da15d863a20fe22c17206e8ac5c704b3011130ef09451f6dc0d915ea"} Nov 02 06:53:30.625052 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:30.625005 5916 generic.go:356] "Generic (PLEG): container finished" podID="9b7d318a-026f-4857-904d-20713b54266c" containerID="7355631c7872a7c15867a0737444c6536a06decaeec309a9b83b4967a3d81ce9" exitCode=0 Nov 02 06:53:30.625251 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:30.625230 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-dcf9c3-66797968df-l4z8j" event={"ID":"9b7d318a-026f-4857-904d-20713b54266c","Type":"ContainerDied","Data":"7355631c7872a7c15867a0737444c6536a06decaeec309a9b83b4967a3d81ce9"} Nov 02 06:53:30.691801 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:30.691774 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-dcf9c3-66797968df-l4z8j" Nov 02 06:53:30.750376 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:30.750353 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-dcf9c3-694d5779d7-cwfqq" Nov 02 06:53:30.797164 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:30.797137 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b7d318a-026f-4857-904d-20713b54266c-kube-api-access-4h96p\" (UniqueName: \"kubernetes.io/projected/9b7d318a-026f-4857-904d-20713b54266c-kube-api-access-4h96p\") pod \"9b7d318a-026f-4857-904d-20713b54266c\" (UID: \"9b7d318a-026f-4857-904d-20713b54266c\") " Nov 02 06:53:30.797291 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:30.797192 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b7d318a-026f-4857-904d-20713b54266c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b7d318a-026f-4857-904d-20713b54266c-plugins-config-volume\") pod \"9b7d318a-026f-4857-904d-20713b54266c\" (UID: \"9b7d318a-026f-4857-904d-20713b54266c\") " Nov 02 06:53:30.797614 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:30.797588 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b7d318a-026f-4857-904d-20713b54266c-plugins-config-volume" pod "9b7d318a-026f-4857-904d-20713b54266c" (UID: "9b7d318a-026f-4857-904d-20713b54266c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:53:30.797773 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:30.797758 5916 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b7d318a-026f-4857-904d-20713b54266c-plugins-config-volume\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:53:30.799260 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:30.799237 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b7d318a-026f-4857-904d-20713b54266c-kube-api-access-4h96p" pod "9b7d318a-026f-4857-904d-20713b54266c" (UID: "9b7d318a-026f-4857-904d-20713b54266c"). InnerVolumeSpecName "kube-api-access-4h96p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:53:30.898751 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:30.898709 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8205f559-100b-4fff-8e3c-4dea453db8e6-kube-api-access-b4w7l\" (UniqueName: \"kubernetes.io/projected/8205f559-100b-4fff-8e3c-4dea453db8e6-kube-api-access-b4w7l\") pod \"8205f559-100b-4fff-8e3c-4dea453db8e6\" (UID: \"8205f559-100b-4fff-8e3c-4dea453db8e6\") " Nov 02 06:53:30.898956 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:30.898761 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8205f559-100b-4fff-8e3c-4dea453db8e6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8205f559-100b-4fff-8e3c-4dea453db8e6-plugins-config-volume\") pod \"8205f559-100b-4fff-8e3c-4dea453db8e6\" (UID: \"8205f559-100b-4fff-8e3c-4dea453db8e6\") " Nov 02 06:53:30.899152 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:30.899128 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8205f559-100b-4fff-8e3c-4dea453db8e6-plugins-config-volume" pod "8205f559-100b-4fff-8e3c-4dea453db8e6" (UID: "8205f559-100b-4fff-8e3c-4dea453db8e6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:53:30.899225 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:30.899200 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4h96p\" (UniqueName: \"kubernetes.io/projected/9b7d318a-026f-4857-904d-20713b54266c-kube-api-access-4h96p\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:53:30.900809 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:30.900787 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8205f559-100b-4fff-8e3c-4dea453db8e6-kube-api-access-b4w7l" pod "8205f559-100b-4fff-8e3c-4dea453db8e6" (UID: "8205f559-100b-4fff-8e3c-4dea453db8e6"). InnerVolumeSpecName "kube-api-access-b4w7l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:53:31.001146 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:31.000975 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4w7l\" (UniqueName: \"kubernetes.io/projected/8205f559-100b-4fff-8e3c-4dea453db8e6-kube-api-access-b4w7l\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:53:31.001146 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:31.001008 5916 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8205f559-100b-4fff-8e3c-4dea453db8e6-plugins-config-volume\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:53:31.386044 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:31.386010 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-fb1398-gpj8g"] Nov 02 06:53:31.386557 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:31.386544 5916 cpu_manager.go:414] "containerMap: removing container" podUID="8205f559-100b-4fff-8e3c-4dea453db8e6" containerName="payload-pre-processing" Nov 02 06:53:31.386611 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:31.386558 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8205f559-100b-4fff-8e3c-4dea453db8e6" containerName="payload-pre-processing" Nov 02 06:53:31.386611 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:31.386584 5916 cpu_manager.go:414] "containerMap: removing container" podUID="e37d42ab-7b62-4f07-8a29-cdedeb0fe89a" containerName="main" Nov 02 06:53:31.386611 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:31.386589 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e37d42ab-7b62-4f07-8a29-cdedeb0fe89a" containerName="main" Nov 02 06:53:31.386611 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:31.386609 5916 cpu_manager.go:414] "containerMap: removing container" podUID="b64d8a67-82cc-4714-87b8-2181f9388f07" containerName="istio-proxy" Nov 02 06:53:31.386737 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:31.386615 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b64d8a67-82cc-4714-87b8-2181f9388f07" containerName="istio-proxy" Nov 02 06:53:31.386737 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:31.386633 5916 cpu_manager.go:414] "containerMap: removing container" podUID="9b7d318a-026f-4857-904d-20713b54266c" containerName="payload-processing" Nov 02 06:53:31.386737 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:31.386638 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b7d318a-026f-4857-904d-20713b54266c" containerName="payload-processing" Nov 02 06:53:31.386737 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:31.386646 5916 cpu_manager.go:414] "containerMap: removing container" podUID="18305d70-7fdf-4ffa-8d86-8c97ce8dea4d" containerName="revoke-keys" Nov 02 06:53:31.386737 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:31.386650 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18305d70-7fdf-4ffa-8d86-8c97ce8dea4d" containerName="revoke-keys" Nov 02 06:53:31.386737 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:31.386660 5916 cpu_manager.go:414] "containerMap: removing container" podUID="15eca023-182b-433b-a2ba-38c0b4e8a1ec" containerName="istio-proxy" Nov 02 06:53:31.386737 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:31.386665 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15eca023-182b-433b-a2ba-38c0b4e8a1ec" containerName="istio-proxy" Nov 02 06:53:31.386975 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:31.386747 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b7d318a-026f-4857-904d-20713b54266c" containerName="payload-processing" Nov 02 06:53:31.386975 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:31.386757 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="18305d70-7fdf-4ffa-8d86-8c97ce8dea4d" containerName="revoke-keys" Nov 02 06:53:31.386975 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:31.386763 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="15eca023-182b-433b-a2ba-38c0b4e8a1ec" containerName="istio-proxy" Nov 02 06:53:31.386975 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:31.386771 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="8205f559-100b-4fff-8e3c-4dea453db8e6" containerName="payload-pre-processing" Nov 02 06:53:31.386975 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:31.386779 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="e37d42ab-7b62-4f07-8a29-cdedeb0fe89a" containerName="main" Nov 02 06:53:31.386975 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:31.386785 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="b64d8a67-82cc-4714-87b8-2181f9388f07" containerName="istio-proxy" Nov 02 06:53:31.390792 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:31.390769 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-fb1398-gpj8g" Nov 02 06:53:31.447775 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:31.447740 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-fb1398-gpj8g"] Nov 02 06:53:31.492027 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:31.491993 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 06:53:31.492027 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:31.492027 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-68l4r\"" Nov 02 06:53:31.506537 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:31.506502 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c7b54e7-87ea-48d5-b868-7a81da8f2099-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-fb1398-gpj8g\" (UID: \"6c7b54e7-87ea-48d5-b868-7a81da8f2099\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-fb1398-gpj8g" Nov 02 06:53:31.506693 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:31.506653 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c7b54e7-87ea-48d5-b868-7a81da8f2099-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-fb1398-gpj8g\" (UID: \"6c7b54e7-87ea-48d5-b868-7a81da8f2099\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-fb1398-gpj8g" Nov 02 06:53:31.506790 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:31.506776 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpt9k\" (UniqueName: \"kubernetes.io/projected/6c7b54e7-87ea-48d5-b868-7a81da8f2099-kube-api-access-jpt9k\") pod \"maas-api-revoke-keys-e2e-worker-w3-fb1398-gpj8g\" (UID: \"6c7b54e7-87ea-48d5-b868-7a81da8f2099\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-fb1398-gpj8g" Nov 02 06:53:31.607820 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:31.607740 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c7b54e7-87ea-48d5-b868-7a81da8f2099-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-fb1398-gpj8g\" (UID: \"6c7b54e7-87ea-48d5-b868-7a81da8f2099\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-fb1398-gpj8g" Nov 02 06:53:31.607820 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:31.607814 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c7b54e7-87ea-48d5-b868-7a81da8f2099-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-fb1398-gpj8g\" (UID: \"6c7b54e7-87ea-48d5-b868-7a81da8f2099\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-fb1398-gpj8g" Nov 02 06:53:31.608078 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:31.607889 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpt9k\" (UniqueName: \"kubernetes.io/projected/6c7b54e7-87ea-48d5-b868-7a81da8f2099-kube-api-access-jpt9k\") pod \"maas-api-revoke-keys-e2e-worker-w3-fb1398-gpj8g\" (UID: \"6c7b54e7-87ea-48d5-b868-7a81da8f2099\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-fb1398-gpj8g" Nov 02 06:53:31.608485 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:31.608463 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c7b54e7-87ea-48d5-b868-7a81da8f2099-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-fb1398-gpj8g\" (UID: \"6c7b54e7-87ea-48d5-b868-7a81da8f2099\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-fb1398-gpj8g" Nov 02 06:53:31.610108 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:31.610080 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c7b54e7-87ea-48d5-b868-7a81da8f2099-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-fb1398-gpj8g\" (UID: \"6c7b54e7-87ea-48d5-b868-7a81da8f2099\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-fb1398-gpj8g" Nov 02 06:53:31.636262 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:31.636234 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-dcf9c3-66797968df-l4z8j" Nov 02 06:53:31.636262 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:31.636260 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-dcf9c3-66797968df-l4z8j" event={"ID":"9b7d318a-026f-4857-904d-20713b54266c","Type":"ContainerDied","Data":"59553fcaf56dcad33d026924ae1790eb183853ae099d7d13470ed683c1783ec1"} Nov 02 06:53:31.636262 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:31.636325 5916 scope.go:122] "RemoveContainer" containerID="7355631c7872a7c15867a0737444c6536a06decaeec309a9b83b4967a3d81ce9" Nov 02 06:53:31.638321 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:31.638144 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-dcf9c3-694d5779d7-cwfqq" event={"ID":"8205f559-100b-4fff-8e3c-4dea453db8e6","Type":"ContainerDied","Data":"f19ffdcc787bef41e9e5539e31a4f96bc3df921ec505a19204f8392d62cc48e5"} Nov 02 06:53:31.638321 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:31.638164 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-dcf9c3-694d5779d7-cwfqq" Nov 02 06:53:31.649985 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:31.649962 5916 scope.go:122] "RemoveContainer" containerID="fc7f5391da15d863a20fe22c17206e8ac5c704b3011130ef09451f6dc0d915ea" Nov 02 06:53:31.665338 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:31.665316 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpt9k\" (UniqueName: \"kubernetes.io/projected/6c7b54e7-87ea-48d5-b868-7a81da8f2099-kube-api-access-jpt9k\") pod \"maas-api-revoke-keys-e2e-worker-w3-fb1398-gpj8g\" (UID: \"6c7b54e7-87ea-48d5-b868-7a81da8f2099\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-fb1398-gpj8g" Nov 02 06:53:31.705638 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:31.705604 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-fb1398-gpj8g" Nov 02 06:53:31.809066 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:31.808613 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-dcf9c3-694d5779d7-cwfqq"] Nov 02 06:53:31.815934 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:31.814628 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-dcf9c3-694d5779d7-cwfqq"] Nov 02 06:53:31.857974 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:31.857233 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8205f559-100b-4fff-8e3c-4dea453db8e6" path="/var/lib/kubelet/pods/8205f559-100b-4fff-8e3c-4dea453db8e6/volumes" Nov 02 06:53:31.921044 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:31.921001 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-dcf9c3-66797968df-l4z8j"] Nov 02 06:53:31.927100 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:31.927064 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-dcf9c3-66797968df-l4z8j"] Nov 02 06:53:32.100809 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:32.100783 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-fb1398-gpj8g"] Nov 02 06:53:32.101260 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:53:32.101238 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c7b54e7_87ea_48d5_b868_7a81da8f2099.slice/crio-337c561ae5a6d4fd0c18cd3c20ab77662da6ff03804ddffd6a71042851c0226e WatchSource:0}: Error finding container 337c561ae5a6d4fd0c18cd3c20ab77662da6ff03804ddffd6a71042851c0226e: Status 404 returned error can't find the container with id 337c561ae5a6d4fd0c18cd3c20ab77662da6ff03804ddffd6a71042851c0226e Nov 02 06:53:32.651183 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:32.651142 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-fb1398-gpj8g" event={"ID":"6c7b54e7-87ea-48d5-b868-7a81da8f2099","Type":"ContainerStarted","Data":"0ee1e996bb8bdc5cd5e9c5d8802822da295e1b63d86654aa38a87b9aa46016db"} Nov 02 06:53:32.651702 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:32.651201 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-fb1398-gpj8g" event={"ID":"6c7b54e7-87ea-48d5-b868-7a81da8f2099","Type":"ContainerStarted","Data":"337c561ae5a6d4fd0c18cd3c20ab77662da6ff03804ddffd6a71042851c0226e"} Nov 02 06:53:32.767569 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:32.767443 5916 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-fb1398-gpj8g" podStartSLOduration=1.767428698 podStartE2EDuration="1.767428698s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:53:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:53:32.765064839 +0000 UTC m=+2201.036885235" watchObservedRunningTime="2026-11-02 06:53:32.767428698 +0000 UTC m=+2201.039249091" Nov 02 06:53:33.661035 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:33.660999 5916 generic.go:356] "Generic (PLEG): container finished" podID="6c7b54e7-87ea-48d5-b868-7a81da8f2099" containerID="0ee1e996bb8bdc5cd5e9c5d8802822da295e1b63d86654aa38a87b9aa46016db" exitCode=0 Nov 02 06:53:33.661529 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:33.661075 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-fb1398-gpj8g" event={"ID":"6c7b54e7-87ea-48d5-b868-7a81da8f2099","Type":"ContainerDied","Data":"0ee1e996bb8bdc5cd5e9c5d8802822da295e1b63d86654aa38a87b9aa46016db"} Nov 02 06:53:33.845117 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:33.845086 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b7d318a-026f-4857-904d-20713b54266c" path="/var/lib/kubelet/pods/9b7d318a-026f-4857-904d-20713b54266c/volumes" Nov 02 06:53:34.859833 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:34.859557 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-fb1398-gpj8g" Nov 02 06:53:34.959596 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:34.959498 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c7b54e7-87ea-48d5-b868-7a81da8f2099-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c7b54e7-87ea-48d5-b868-7a81da8f2099-maas-api-openshift-ca\") pod \"6c7b54e7-87ea-48d5-b868-7a81da8f2099\" (UID: \"6c7b54e7-87ea-48d5-b868-7a81da8f2099\") " Nov 02 06:53:34.959596 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:34.959550 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c7b54e7-87ea-48d5-b868-7a81da8f2099-kube-api-access-jpt9k\" (UniqueName: \"kubernetes.io/projected/6c7b54e7-87ea-48d5-b868-7a81da8f2099-kube-api-access-jpt9k\") pod \"6c7b54e7-87ea-48d5-b868-7a81da8f2099\" (UID: \"6c7b54e7-87ea-48d5-b868-7a81da8f2099\") " Nov 02 06:53:34.959596 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:34.959590 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6c7b54e7-87ea-48d5-b868-7a81da8f2099-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c7b54e7-87ea-48d5-b868-7a81da8f2099-maas-api-xks-ca\") pod \"6c7b54e7-87ea-48d5-b868-7a81da8f2099\" (UID: \"6c7b54e7-87ea-48d5-b868-7a81da8f2099\") " Nov 02 06:53:34.959986 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:34.959898 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c7b54e7-87ea-48d5-b868-7a81da8f2099-maas-api-openshift-ca" pod "6c7b54e7-87ea-48d5-b868-7a81da8f2099" (UID: "6c7b54e7-87ea-48d5-b868-7a81da8f2099"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:53:34.960274 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:34.960250 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c7b54e7-87ea-48d5-b868-7a81da8f2099-maas-api-openshift-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:53:34.962144 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:34.962113 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c7b54e7-87ea-48d5-b868-7a81da8f2099-maas-api-xks-ca" pod "6c7b54e7-87ea-48d5-b868-7a81da8f2099" (UID: "6c7b54e7-87ea-48d5-b868-7a81da8f2099"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:53:34.962663 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:34.962642 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c7b54e7-87ea-48d5-b868-7a81da8f2099-kube-api-access-jpt9k" pod "6c7b54e7-87ea-48d5-b868-7a81da8f2099" (UID: "6c7b54e7-87ea-48d5-b868-7a81da8f2099"). InnerVolumeSpecName "kube-api-access-jpt9k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:53:35.061847 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:35.061807 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jpt9k\" (UniqueName: \"kubernetes.io/projected/6c7b54e7-87ea-48d5-b868-7a81da8f2099-kube-api-access-jpt9k\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:53:35.061847 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:35.061850 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c7b54e7-87ea-48d5-b868-7a81da8f2099-maas-api-xks-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:53:35.684333 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:35.684300 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-fb1398-gpj8g" Nov 02 06:53:35.684548 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:35.684290 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-fb1398-gpj8g" event={"ID":"6c7b54e7-87ea-48d5-b868-7a81da8f2099","Type":"ContainerDied","Data":"337c561ae5a6d4fd0c18cd3c20ab77662da6ff03804ddffd6a71042851c0226e"} Nov 02 06:53:35.684548 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:35.684435 5916 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="337c561ae5a6d4fd0c18cd3c20ab77662da6ff03804ddffd6a71042851c0226e" Nov 02 06:53:35.832684 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:35.832523 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-fb1398/distinct-2-w3-fb1398-kserve-597b7dc99b-dp4jf"] Nov 02 06:53:39.052792 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:39.052747 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f83fbf8-fhcqz"] Nov 02 06:53:39.107258 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:39.107206 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f83fbf8-fhcqz"] Nov 02 06:53:39.921933 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:39.918958 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5aa6bbf7-bc9e-4c67-bbfb-bc24f450b185" path="/var/lib/kubelet/pods/5aa6bbf7-bc9e-4c67-bbfb-bc24f450b185/volumes" Nov 02 06:53:40.532936 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:40.527998 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6ff85e/simulator-w6-6ff85e-kserve-9b9fd978f-q76nl" podUID="30114285-d730-4abb-a618-6833d5746feb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Nov 02 06:53:40.532936 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:40.528442 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-581608/distinct-2-w2-581608-kserve-57dd49c4b-qnt9k" podUID="837780ad-8ea3-4ed2-922f-917cffc7bb93" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": context deadline exceeded" Nov 02 06:53:40.713930 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:40.713086 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-680b7b/simulator-w4-680b7b-kserve-6c4f574c56-p5xtc" podUID="be8a1039-5934-4885-93fe-070575f936ec" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:53:42.159960 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:42.159922 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13c46a92-dfjmg"] Nov 02 06:53:42.160477 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:42.160464 5916 cpu_manager.go:414] "containerMap: removing container" podUID="6c7b54e7-87ea-48d5-b868-7a81da8f2099" containerName="revoke-keys" Nov 02 06:53:42.160524 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:42.160478 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c7b54e7-87ea-48d5-b868-7a81da8f2099" containerName="revoke-keys" Nov 02 06:53:42.160587 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:42.160578 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c7b54e7-87ea-48d5-b868-7a81da8f2099" containerName="revoke-keys" Nov 02 06:53:42.164389 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:42.164358 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13c46a92-dfjmg" Nov 02 06:53:42.192357 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:42.192323 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f0419bf-81cc-4bda-a4c4-6da385635e12-maas-api-xks-ca\") pod \"maas-api-revoke-sub-13c46a92-dfjmg\" (UID: \"5f0419bf-81cc-4bda-a4c4-6da385635e12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13c46a92-dfjmg" Nov 02 06:53:42.192513 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:42.192390 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghmd9\" (UniqueName: \"kubernetes.io/projected/5f0419bf-81cc-4bda-a4c4-6da385635e12-kube-api-access-ghmd9\") pod \"maas-api-revoke-sub-13c46a92-dfjmg\" (UID: \"5f0419bf-81cc-4bda-a4c4-6da385635e12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13c46a92-dfjmg" Nov 02 06:53:42.192587 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:42.192565 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f0419bf-81cc-4bda-a4c4-6da385635e12-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-13c46a92-dfjmg\" (UID: \"5f0419bf-81cc-4bda-a4c4-6da385635e12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13c46a92-dfjmg" Nov 02 06:53:42.216967 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:42.216927 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1eab520f-9qkbc"] Nov 02 06:53:42.221931 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:42.221872 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13c46a92-dfjmg"] Nov 02 06:53:42.222046 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:42.221997 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1eab520f-9qkbc" Nov 02 06:53:42.261408 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:42.261367 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 06:53:42.261658 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:42.261635 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-68l4r\"" Nov 02 06:53:42.275849 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:42.275821 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1eab520f-9qkbc"] Nov 02 06:53:42.293435 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:42.293406 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64gz9\" (UniqueName: \"kubernetes.io/projected/f33e7b06-ffea-4c8b-8905-bd9a469213c4-kube-api-access-64gz9\") pod \"maas-api-revoke-sub-1eab520f-9qkbc\" (UID: \"f33e7b06-ffea-4c8b-8905-bd9a469213c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1eab520f-9qkbc" Nov 02 06:53:42.293599 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:42.293445 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f33e7b06-ffea-4c8b-8905-bd9a469213c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1eab520f-9qkbc\" (UID: \"f33e7b06-ffea-4c8b-8905-bd9a469213c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1eab520f-9qkbc" Nov 02 06:53:42.293599 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:42.293551 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f0419bf-81cc-4bda-a4c4-6da385635e12-maas-api-xks-ca\") pod \"maas-api-revoke-sub-13c46a92-dfjmg\" (UID: \"5f0419bf-81cc-4bda-a4c4-6da385635e12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13c46a92-dfjmg" Nov 02 06:53:42.293689 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:42.293654 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghmd9\" (UniqueName: \"kubernetes.io/projected/5f0419bf-81cc-4bda-a4c4-6da385635e12-kube-api-access-ghmd9\") pod \"maas-api-revoke-sub-13c46a92-dfjmg\" (UID: \"5f0419bf-81cc-4bda-a4c4-6da385635e12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13c46a92-dfjmg" Nov 02 06:53:42.293739 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:42.293697 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f33e7b06-ffea-4c8b-8905-bd9a469213c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1eab520f-9qkbc\" (UID: \"f33e7b06-ffea-4c8b-8905-bd9a469213c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1eab520f-9qkbc" Nov 02 06:53:42.293928 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:42.293890 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f0419bf-81cc-4bda-a4c4-6da385635e12-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-13c46a92-dfjmg\" (UID: \"5f0419bf-81cc-4bda-a4c4-6da385635e12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13c46a92-dfjmg" Nov 02 06:53:42.294753 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:42.294730 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f0419bf-81cc-4bda-a4c4-6da385635e12-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-13c46a92-dfjmg\" (UID: \"5f0419bf-81cc-4bda-a4c4-6da385635e12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13c46a92-dfjmg" Nov 02 06:53:42.296033 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:42.296011 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f0419bf-81cc-4bda-a4c4-6da385635e12-maas-api-xks-ca\") pod \"maas-api-revoke-sub-13c46a92-dfjmg\" (UID: \"5f0419bf-81cc-4bda-a4c4-6da385635e12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13c46a92-dfjmg" Nov 02 06:53:42.350977 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:42.350936 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghmd9\" (UniqueName: \"kubernetes.io/projected/5f0419bf-81cc-4bda-a4c4-6da385635e12-kube-api-access-ghmd9\") pod \"maas-api-revoke-sub-13c46a92-dfjmg\" (UID: \"5f0419bf-81cc-4bda-a4c4-6da385635e12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13c46a92-dfjmg" Nov 02 06:53:42.395102 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:42.395062 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64gz9\" (UniqueName: \"kubernetes.io/projected/f33e7b06-ffea-4c8b-8905-bd9a469213c4-kube-api-access-64gz9\") pod \"maas-api-revoke-sub-1eab520f-9qkbc\" (UID: \"f33e7b06-ffea-4c8b-8905-bd9a469213c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1eab520f-9qkbc" Nov 02 06:53:42.395287 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:42.395110 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f33e7b06-ffea-4c8b-8905-bd9a469213c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1eab520f-9qkbc\" (UID: \"f33e7b06-ffea-4c8b-8905-bd9a469213c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1eab520f-9qkbc" Nov 02 06:53:42.395287 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:42.395216 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f33e7b06-ffea-4c8b-8905-bd9a469213c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1eab520f-9qkbc\" (UID: \"f33e7b06-ffea-4c8b-8905-bd9a469213c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1eab520f-9qkbc" Nov 02 06:53:42.395925 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:42.395893 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f33e7b06-ffea-4c8b-8905-bd9a469213c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1eab520f-9qkbc\" (UID: \"f33e7b06-ffea-4c8b-8905-bd9a469213c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1eab520f-9qkbc" Nov 02 06:53:42.397580 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:42.397563 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f33e7b06-ffea-4c8b-8905-bd9a469213c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1eab520f-9qkbc\" (UID: \"f33e7b06-ffea-4c8b-8905-bd9a469213c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1eab520f-9qkbc" Nov 02 06:53:42.453222 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:42.453144 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64gz9\" (UniqueName: \"kubernetes.io/projected/f33e7b06-ffea-4c8b-8905-bd9a469213c4-kube-api-access-64gz9\") pod \"maas-api-revoke-sub-1eab520f-9qkbc\" (UID: \"f33e7b06-ffea-4c8b-8905-bd9a469213c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1eab520f-9qkbc" Nov 02 06:53:42.478417 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:42.478386 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13c46a92-dfjmg" Nov 02 06:53:42.540663 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:42.540628 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1eab520f-9qkbc" Nov 02 06:53:42.874228 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:42.874199 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13c46a92-dfjmg"] Nov 02 06:53:42.875518 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:53:42.875493 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f0419bf_81cc_4bda_a4c4_6da385635e12.slice/crio-7a0109e6c0b2db1c90da1beffcefcb1ef8712a648b6a0b59c2bee627f231e836 WatchSource:0}: Error finding container 7a0109e6c0b2db1c90da1beffcefcb1ef8712a648b6a0b59c2bee627f231e836: Status 404 returned error can't find the container with id 7a0109e6c0b2db1c90da1beffcefcb1ef8712a648b6a0b59c2bee627f231e836 Nov 02 06:53:42.972183 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:42.972131 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1eab520f-9qkbc"] Nov 02 06:53:42.973016 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:53:42.972988 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf33e7b06_ffea_4c8b_8905_bd9a469213c4.slice/crio-e62b3cbe84b137872750d8a825bac6ecfb5a2bcb437c0e411927868c5a441314 WatchSource:0}: Error finding container e62b3cbe84b137872750d8a825bac6ecfb5a2bcb437c0e411927868c5a441314: Status 404 returned error can't find the container with id e62b3cbe84b137872750d8a825bac6ecfb5a2bcb437c0e411927868c5a441314 Nov 02 06:53:43.847618 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:43.847572 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13c46a92-dfjmg" event={"ID":"5f0419bf-81cc-4bda-a4c4-6da385635e12","Type":"ContainerStarted","Data":"c80128b3c89d29226f7b7aa8bbf582be9fb3a9ae71def60ea64a0178c4e9580a"} Nov 02 06:53:43.848278 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:43.847626 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13c46a92-dfjmg" event={"ID":"5f0419bf-81cc-4bda-a4c4-6da385635e12","Type":"ContainerStarted","Data":"7a0109e6c0b2db1c90da1beffcefcb1ef8712a648b6a0b59c2bee627f231e836"} Nov 02 06:53:43.848572 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:43.848549 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1eab520f-9qkbc" event={"ID":"f33e7b06-ffea-4c8b-8905-bd9a469213c4","Type":"ContainerStarted","Data":"6531cf3632be20069dc9a5e418d81f175ba6fd354a196ea10e3802fd6d2fe2c1"} Nov 02 06:53:43.848670 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:43.848593 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1eab520f-9qkbc" event={"ID":"f33e7b06-ffea-4c8b-8905-bd9a469213c4","Type":"ContainerStarted","Data":"e62b3cbe84b137872750d8a825bac6ecfb5a2bcb437c0e411927868c5a441314"} Nov 02 06:53:44.094356 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:44.094237 5916 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1eab520f-9qkbc" podStartSLOduration=2.09422116 podStartE2EDuration="2.09422116s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:53:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:53:44.09286304 +0000 UTC m=+2212.364683435" watchObservedRunningTime="2026-11-02 06:53:44.09422116 +0000 UTC m=+2212.366041556" Nov 02 06:53:44.859887 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:44.859852 5916 generic.go:356] "Generic (PLEG): container finished" podID="5f0419bf-81cc-4bda-a4c4-6da385635e12" containerID="c80128b3c89d29226f7b7aa8bbf582be9fb3a9ae71def60ea64a0178c4e9580a" exitCode=0 Nov 02 06:53:44.860359 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:44.859938 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13c46a92-dfjmg" event={"ID":"5f0419bf-81cc-4bda-a4c4-6da385635e12","Type":"ContainerDied","Data":"c80128b3c89d29226f7b7aa8bbf582be9fb3a9ae71def60ea64a0178c4e9580a"} Nov 02 06:53:44.861859 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:44.861832 5916 generic.go:356] "Generic (PLEG): container finished" podID="f33e7b06-ffea-4c8b-8905-bd9a469213c4" containerID="6531cf3632be20069dc9a5e418d81f175ba6fd354a196ea10e3802fd6d2fe2c1" exitCode=0 Nov 02 06:53:44.861986 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:44.861895 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1eab520f-9qkbc" event={"ID":"f33e7b06-ffea-4c8b-8905-bd9a469213c4","Type":"ContainerDied","Data":"6531cf3632be20069dc9a5e418d81f175ba6fd354a196ea10e3802fd6d2fe2c1"} Nov 02 06:53:45.092640 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:45.092614 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13c46a92-dfjmg" Nov 02 06:53:45.148942 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:45.148896 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f0419bf-81cc-4bda-a4c4-6da385635e12-kube-api-access-ghmd9\" (UniqueName: \"kubernetes.io/projected/5f0419bf-81cc-4bda-a4c4-6da385635e12-kube-api-access-ghmd9\") pod \"5f0419bf-81cc-4bda-a4c4-6da385635e12\" (UID: \"5f0419bf-81cc-4bda-a4c4-6da385635e12\") " Nov 02 06:53:45.149128 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:45.148952 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f0419bf-81cc-4bda-a4c4-6da385635e12-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f0419bf-81cc-4bda-a4c4-6da385635e12-maas-api-openshift-ca\") pod \"5f0419bf-81cc-4bda-a4c4-6da385635e12\" (UID: \"5f0419bf-81cc-4bda-a4c4-6da385635e12\") " Nov 02 06:53:45.149128 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:45.148984 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f0419bf-81cc-4bda-a4c4-6da385635e12-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f0419bf-81cc-4bda-a4c4-6da385635e12-maas-api-xks-ca\") pod \"5f0419bf-81cc-4bda-a4c4-6da385635e12\" (UID: \"5f0419bf-81cc-4bda-a4c4-6da385635e12\") " Nov 02 06:53:45.149354 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:45.149326 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f0419bf-81cc-4bda-a4c4-6da385635e12-maas-api-openshift-ca" pod "5f0419bf-81cc-4bda-a4c4-6da385635e12" (UID: "5f0419bf-81cc-4bda-a4c4-6da385635e12"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:53:45.149473 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:45.149445 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f0419bf-81cc-4bda-a4c4-6da385635e12-maas-api-openshift-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:53:45.151215 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:45.151186 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f0419bf-81cc-4bda-a4c4-6da385635e12-kube-api-access-ghmd9" pod "5f0419bf-81cc-4bda-a4c4-6da385635e12" (UID: "5f0419bf-81cc-4bda-a4c4-6da385635e12"). InnerVolumeSpecName "kube-api-access-ghmd9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:53:45.151215 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:45.151211 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f0419bf-81cc-4bda-a4c4-6da385635e12-maas-api-xks-ca" pod "5f0419bf-81cc-4bda-a4c4-6da385635e12" (UID: "5f0419bf-81cc-4bda-a4c4-6da385635e12"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:53:45.271198 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:45.252076 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghmd9\" (UniqueName: \"kubernetes.io/projected/5f0419bf-81cc-4bda-a4c4-6da385635e12-kube-api-access-ghmd9\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:53:45.271198 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:45.252119 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f0419bf-81cc-4bda-a4c4-6da385635e12-maas-api-xks-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:53:45.873216 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:45.873181 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13c46a92-dfjmg" Nov 02 06:53:45.873216 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:45.873201 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13c46a92-dfjmg" event={"ID":"5f0419bf-81cc-4bda-a4c4-6da385635e12","Type":"ContainerDied","Data":"7a0109e6c0b2db1c90da1beffcefcb1ef8712a648b6a0b59c2bee627f231e836"} Nov 02 06:53:45.873805 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:45.873247 5916 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7a0109e6c0b2db1c90da1beffcefcb1ef8712a648b6a0b59c2bee627f231e836" Nov 02 06:53:46.035585 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:46.035558 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1eab520f-9qkbc" Nov 02 06:53:46.163730 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:46.163696 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f33e7b06-ffea-4c8b-8905-bd9a469213c4-kube-api-access-64gz9\" (UniqueName: \"kubernetes.io/projected/f33e7b06-ffea-4c8b-8905-bd9a469213c4-kube-api-access-64gz9\") pod \"f33e7b06-ffea-4c8b-8905-bd9a469213c4\" (UID: \"f33e7b06-ffea-4c8b-8905-bd9a469213c4\") " Nov 02 06:53:46.164456 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:46.163955 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f33e7b06-ffea-4c8b-8905-bd9a469213c4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f33e7b06-ffea-4c8b-8905-bd9a469213c4-maas-api-xks-ca\") pod \"f33e7b06-ffea-4c8b-8905-bd9a469213c4\" (UID: \"f33e7b06-ffea-4c8b-8905-bd9a469213c4\") " Nov 02 06:53:46.164456 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:46.164324 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f33e7b06-ffea-4c8b-8905-bd9a469213c4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f33e7b06-ffea-4c8b-8905-bd9a469213c4-maas-api-openshift-ca\") pod \"f33e7b06-ffea-4c8b-8905-bd9a469213c4\" (UID: \"f33e7b06-ffea-4c8b-8905-bd9a469213c4\") " Nov 02 06:53:46.164601 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:46.164571 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f33e7b06-ffea-4c8b-8905-bd9a469213c4-maas-api-openshift-ca" pod "f33e7b06-ffea-4c8b-8905-bd9a469213c4" (UID: "f33e7b06-ffea-4c8b-8905-bd9a469213c4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:53:46.165426 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:46.165400 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f33e7b06-ffea-4c8b-8905-bd9a469213c4-maas-api-openshift-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:53:46.166877 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:46.166843 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f33e7b06-ffea-4c8b-8905-bd9a469213c4-kube-api-access-64gz9" pod "f33e7b06-ffea-4c8b-8905-bd9a469213c4" (UID: "f33e7b06-ffea-4c8b-8905-bd9a469213c4"). InnerVolumeSpecName "kube-api-access-64gz9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:53:46.167400 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:46.167364 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f33e7b06-ffea-4c8b-8905-bd9a469213c4-maas-api-xks-ca" pod "f33e7b06-ffea-4c8b-8905-bd9a469213c4" (UID: "f33e7b06-ffea-4c8b-8905-bd9a469213c4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:53:46.266500 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:46.266463 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64gz9\" (UniqueName: \"kubernetes.io/projected/f33e7b06-ffea-4c8b-8905-bd9a469213c4-kube-api-access-64gz9\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:53:46.266500 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:46.266495 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f33e7b06-ffea-4c8b-8905-bd9a469213c4-maas-api-xks-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:53:46.882989 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:46.882954 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1eab520f-9qkbc" event={"ID":"f33e7b06-ffea-4c8b-8905-bd9a469213c4","Type":"ContainerDied","Data":"e62b3cbe84b137872750d8a825bac6ecfb5a2bcb437c0e411927868c5a441314"} Nov 02 06:53:46.883394 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:46.883007 5916 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e62b3cbe84b137872750d8a825bac6ecfb5a2bcb437c0e411927868c5a441314" Nov 02 06:53:46.883394 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:46.882964 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1eab520f-9qkbc" Nov 02 06:53:49.281846 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:49.281805 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-bdcd80-l5h67"] Nov 02 06:53:49.331629 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:49.331592 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-bdcd80-l5h67"] Nov 02 06:53:49.845751 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:49.845718 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="caac1c08-d143-4d83-9bda-d804cbb59d44" path="/var/lib/kubelet/pods/caac1c08-d143-4d83-9bda-d804cbb59d44/volumes" Nov 02 06:53:49.894488 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:53:49.894454 5916 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p98cq" containerName="extractor" Nov 02 06:53:50.881075 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:50.879487 5916 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-fb1398/distinct-2-w3-fb1398-kserve-597b7dc99b-dp4jf" podUID="523a31c4-4aea-4e9f-9ec4-7c28faefbf7f" containerName="main" containerID="cri-o://60af55daf28a4fcff0cc0f411c7c1c80e6a9a5429faa7b85469067dfdea55d0e" gracePeriod=45 Nov 02 06:53:51.005515 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:51.005479 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fb1398/distinct-2-w3-fb1398-kserve-597b7dc99b-dp4jf" podUID="523a31c4-4aea-4e9f-9ec4-7c28faefbf7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": dial tcp 10.132.0.102:8000: connect: connection refused" Nov 02 06:53:51.157566 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:51.157540 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-fb1398/distinct-2-w3-fb1398-kserve-597b7dc99b-dp4jf" Nov 02 06:53:51.227166 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:51.227136 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/523a31c4-4aea-4e9f-9ec4-7c28faefbf7f-home\" (UniqueName: \"kubernetes.io/empty-dir/523a31c4-4aea-4e9f-9ec4-7c28faefbf7f-home\") pod \"523a31c4-4aea-4e9f-9ec4-7c28faefbf7f\" (UID: \"523a31c4-4aea-4e9f-9ec4-7c28faefbf7f\") " Nov 02 06:53:51.227349 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:51.227275 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/523a31c4-4aea-4e9f-9ec4-7c28faefbf7f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/523a31c4-4aea-4e9f-9ec4-7c28faefbf7f-dshm\") pod \"523a31c4-4aea-4e9f-9ec4-7c28faefbf7f\" (UID: \"523a31c4-4aea-4e9f-9ec4-7c28faefbf7f\") " Nov 02 06:53:51.227349 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:51.227328 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/523a31c4-4aea-4e9f-9ec4-7c28faefbf7f-kube-api-access-ppnvp\" (UniqueName: \"kubernetes.io/projected/523a31c4-4aea-4e9f-9ec4-7c28faefbf7f-kube-api-access-ppnvp\") pod \"523a31c4-4aea-4e9f-9ec4-7c28faefbf7f\" (UID: \"523a31c4-4aea-4e9f-9ec4-7c28faefbf7f\") " Nov 02 06:53:51.227476 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:51.227394 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/523a31c4-4aea-4e9f-9ec4-7c28faefbf7f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/523a31c4-4aea-4e9f-9ec4-7c28faefbf7f-tmp-dir\") pod \"523a31c4-4aea-4e9f-9ec4-7c28faefbf7f\" (UID: \"523a31c4-4aea-4e9f-9ec4-7c28faefbf7f\") " Nov 02 06:53:51.227476 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:51.227461 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/523a31c4-4aea-4e9f-9ec4-7c28faefbf7f-home" pod "523a31c4-4aea-4e9f-9ec4-7c28faefbf7f" (UID: "523a31c4-4aea-4e9f-9ec4-7c28faefbf7f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:53:51.227605 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:51.227498 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/523a31c4-4aea-4e9f-9ec4-7c28faefbf7f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/523a31c4-4aea-4e9f-9ec4-7c28faefbf7f-model-cache\") pod \"523a31c4-4aea-4e9f-9ec4-7c28faefbf7f\" (UID: \"523a31c4-4aea-4e9f-9ec4-7c28faefbf7f\") " Nov 02 06:53:51.227605 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:51.227591 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/523a31c4-4aea-4e9f-9ec4-7c28faefbf7f-tls-certs\" (UniqueName: \"kubernetes.io/secret/523a31c4-4aea-4e9f-9ec4-7c28faefbf7f-tls-certs\") pod \"523a31c4-4aea-4e9f-9ec4-7c28faefbf7f\" (UID: \"523a31c4-4aea-4e9f-9ec4-7c28faefbf7f\") " Nov 02 06:53:51.227714 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:51.227674 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/523a31c4-4aea-4e9f-9ec4-7c28faefbf7f-tmp-dir" pod "523a31c4-4aea-4e9f-9ec4-7c28faefbf7f" (UID: "523a31c4-4aea-4e9f-9ec4-7c28faefbf7f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:53:51.227788 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:51.227766 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/523a31c4-4aea-4e9f-9ec4-7c28faefbf7f-model-cache" pod "523a31c4-4aea-4e9f-9ec4-7c28faefbf7f" (UID: "523a31c4-4aea-4e9f-9ec4-7c28faefbf7f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:53:51.228132 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:51.228109 5916 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/523a31c4-4aea-4e9f-9ec4-7c28faefbf7f-tmp-dir\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:53:51.228246 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:51.228134 5916 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/523a31c4-4aea-4e9f-9ec4-7c28faefbf7f-model-cache\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:53:51.228246 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:51.228148 5916 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/523a31c4-4aea-4e9f-9ec4-7c28faefbf7f-home\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:53:51.229878 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:51.229847 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/523a31c4-4aea-4e9f-9ec4-7c28faefbf7f-dshm" pod "523a31c4-4aea-4e9f-9ec4-7c28faefbf7f" (UID: "523a31c4-4aea-4e9f-9ec4-7c28faefbf7f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:53:51.230053 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:51.230034 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/523a31c4-4aea-4e9f-9ec4-7c28faefbf7f-tls-certs" pod "523a31c4-4aea-4e9f-9ec4-7c28faefbf7f" (UID: "523a31c4-4aea-4e9f-9ec4-7c28faefbf7f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:53:51.230053 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:51.230044 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/523a31c4-4aea-4e9f-9ec4-7c28faefbf7f-kube-api-access-ppnvp" pod "523a31c4-4aea-4e9f-9ec4-7c28faefbf7f" (UID: "523a31c4-4aea-4e9f-9ec4-7c28faefbf7f"). InnerVolumeSpecName "kube-api-access-ppnvp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:53:51.329639 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:51.329601 5916 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/523a31c4-4aea-4e9f-9ec4-7c28faefbf7f-dshm\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:53:51.329639 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:51.329637 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ppnvp\" (UniqueName: \"kubernetes.io/projected/523a31c4-4aea-4e9f-9ec4-7c28faefbf7f-kube-api-access-ppnvp\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:53:51.329891 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:51.329654 5916 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/523a31c4-4aea-4e9f-9ec4-7c28faefbf7f-tls-certs\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:53:51.913108 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:53:51.913079 5916 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod523a31c4_4aea_4e9f_9ec4_7c28faefbf7f.slice\": RecentStats: unable to find data in memory cache]" Nov 02 06:53:51.923699 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:51.923673 5916 generic.go:356] "Generic (PLEG): container finished" podID="523a31c4-4aea-4e9f-9ec4-7c28faefbf7f" containerID="60af55daf28a4fcff0cc0f411c7c1c80e6a9a5429faa7b85469067dfdea55d0e" exitCode=0 Nov 02 06:53:51.923847 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:51.923718 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-fb1398/distinct-2-w3-fb1398-kserve-597b7dc99b-dp4jf" Nov 02 06:53:51.923847 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:51.923747 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-fb1398/distinct-2-w3-fb1398-kserve-597b7dc99b-dp4jf" event={"ID":"523a31c4-4aea-4e9f-9ec4-7c28faefbf7f","Type":"ContainerDied","Data":"60af55daf28a4fcff0cc0f411c7c1c80e6a9a5429faa7b85469067dfdea55d0e"} Nov 02 06:53:51.923847 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:51.923785 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-fb1398/distinct-2-w3-fb1398-kserve-597b7dc99b-dp4jf" event={"ID":"523a31c4-4aea-4e9f-9ec4-7c28faefbf7f","Type":"ContainerDied","Data":"138c7b1ee60e1b768650e701a52e3f2e54990dc8cb9ff0b6799d283e121791df"} Nov 02 06:53:51.923847 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:51.923802 5916 scope.go:122] "RemoveContainer" containerID="60af55daf28a4fcff0cc0f411c7c1c80e6a9a5429faa7b85469067dfdea55d0e" Nov 02 06:53:51.936353 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:51.936334 5916 scope.go:122] "RemoveContainer" containerID="60af55daf28a4fcff0cc0f411c7c1c80e6a9a5429faa7b85469067dfdea55d0e" Nov 02 06:53:51.936624 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:53:51.936602 5916 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60af55daf28a4fcff0cc0f411c7c1c80e6a9a5429faa7b85469067dfdea55d0e\": container with ID starting with 60af55daf28a4fcff0cc0f411c7c1c80e6a9a5429faa7b85469067dfdea55d0e not found: ID does not exist" containerID="60af55daf28a4fcff0cc0f411c7c1c80e6a9a5429faa7b85469067dfdea55d0e" Nov 02 06:53:51.936727 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:51.936632 5916 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60af55daf28a4fcff0cc0f411c7c1c80e6a9a5429faa7b85469067dfdea55d0e"} err="failed to get container status \"60af55daf28a4fcff0cc0f411c7c1c80e6a9a5429faa7b85469067dfdea55d0e\": rpc error: code = NotFound desc = could not find container \"60af55daf28a4fcff0cc0f411c7c1c80e6a9a5429faa7b85469067dfdea55d0e\": container with ID starting with 60af55daf28a4fcff0cc0f411c7c1c80e6a9a5429faa7b85469067dfdea55d0e not found: ID does not exist" Nov 02 06:53:52.089257 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:52.089222 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-fb1398/distinct-2-w3-fb1398-kserve-597b7dc99b-dp4jf"] Nov 02 06:53:52.141323 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:52.141287 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-fb1398/distinct-2-w3-fb1398-kserve-597b7dc99b-dp4jf"] Nov 02 06:53:53.845649 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:53.845610 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="523a31c4-4aea-4e9f-9ec4-7c28faefbf7f" path="/var/lib/kubelet/pods/523a31c4-4aea-4e9f-9ec4-7c28faefbf7f/volumes" Nov 02 06:53:58.039381 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:58.039343 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89d4bdbf-pw8vc"] Nov 02 06:53:58.039897 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:58.039881 5916 cpu_manager.go:414] "containerMap: removing container" podUID="f33e7b06-ffea-4c8b-8905-bd9a469213c4" containerName="revoke-keys" Nov 02 06:53:58.039983 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:58.039897 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f33e7b06-ffea-4c8b-8905-bd9a469213c4" containerName="revoke-keys" Nov 02 06:53:58.039983 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:58.039931 5916 cpu_manager.go:414] "containerMap: removing container" podUID="5f0419bf-81cc-4bda-a4c4-6da385635e12" containerName="revoke-keys" Nov 02 06:53:58.039983 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:58.039936 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f0419bf-81cc-4bda-a4c4-6da385635e12" containerName="revoke-keys" Nov 02 06:53:58.039983 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:58.039960 5916 cpu_manager.go:414] "containerMap: removing container" podUID="523a31c4-4aea-4e9f-9ec4-7c28faefbf7f" containerName="main" Nov 02 06:53:58.039983 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:58.039966 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="523a31c4-4aea-4e9f-9ec4-7c28faefbf7f" containerName="main" Nov 02 06:53:58.040178 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:58.040048 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="523a31c4-4aea-4e9f-9ec4-7c28faefbf7f" containerName="main" Nov 02 06:53:58.040178 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:58.040061 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="f33e7b06-ffea-4c8b-8905-bd9a469213c4" containerName="revoke-keys" Nov 02 06:53:58.040178 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:58.040069 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f0419bf-81cc-4bda-a4c4-6da385635e12" containerName="revoke-keys" Nov 02 06:53:58.044966 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:58.044948 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89d4bdbf-pw8vc" Nov 02 06:53:58.097395 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:58.097357 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-adcb334f-cn4xj"] Nov 02 06:53:58.101603 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:58.101577 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-adcb334f-cn4xj" Nov 02 06:53:58.106464 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:58.106436 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89d4bdbf-pw8vc"] Nov 02 06:53:58.143545 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:58.143516 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 06:53:58.152031 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:58.151997 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-68l4r\"" Nov 02 06:53:58.160714 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:58.160685 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-adcb334f-cn4xj"] Nov 02 06:53:58.225055 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:58.225016 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cab8160e-1169-4676-b764-92a3435dcc99-maas-api-xks-ca\") pod \"maas-api-revoke-sub-adcb334f-cn4xj\" (UID: \"cab8160e-1169-4676-b764-92a3435dcc99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-adcb334f-cn4xj" Nov 02 06:53:58.225226 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:58.225095 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xqk8\" (UniqueName: \"kubernetes.io/projected/cab8160e-1169-4676-b764-92a3435dcc99-kube-api-access-9xqk8\") pod \"maas-api-revoke-sub-adcb334f-cn4xj\" (UID: \"cab8160e-1169-4676-b764-92a3435dcc99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-adcb334f-cn4xj" Nov 02 06:53:58.225226 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:58.225125 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48757757-4e3d-404d-8af4-93ed81f60565-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-89d4bdbf-pw8vc\" (UID: \"48757757-4e3d-404d-8af4-93ed81f60565\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89d4bdbf-pw8vc" Nov 02 06:53:58.225226 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:58.225161 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cab8160e-1169-4676-b764-92a3435dcc99-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-adcb334f-cn4xj\" (UID: \"cab8160e-1169-4676-b764-92a3435dcc99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-adcb334f-cn4xj" Nov 02 06:53:58.225381 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:58.225262 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7pnt\" (UniqueName: \"kubernetes.io/projected/48757757-4e3d-404d-8af4-93ed81f60565-kube-api-access-b7pnt\") pod \"maas-api-revoke-sub-89d4bdbf-pw8vc\" (UID: \"48757757-4e3d-404d-8af4-93ed81f60565\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89d4bdbf-pw8vc" Nov 02 06:53:58.225415 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:58.225405 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48757757-4e3d-404d-8af4-93ed81f60565-maas-api-xks-ca\") pod \"maas-api-revoke-sub-89d4bdbf-pw8vc\" (UID: \"48757757-4e3d-404d-8af4-93ed81f60565\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89d4bdbf-pw8vc" Nov 02 06:53:58.326497 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:58.326410 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48757757-4e3d-404d-8af4-93ed81f60565-maas-api-xks-ca\") pod \"maas-api-revoke-sub-89d4bdbf-pw8vc\" (UID: \"48757757-4e3d-404d-8af4-93ed81f60565\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89d4bdbf-pw8vc" Nov 02 06:53:58.326670 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:58.326507 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cab8160e-1169-4676-b764-92a3435dcc99-maas-api-xks-ca\") pod \"maas-api-revoke-sub-adcb334f-cn4xj\" (UID: \"cab8160e-1169-4676-b764-92a3435dcc99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-adcb334f-cn4xj" Nov 02 06:53:58.326670 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:58.326568 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xqk8\" (UniqueName: \"kubernetes.io/projected/cab8160e-1169-4676-b764-92a3435dcc99-kube-api-access-9xqk8\") pod \"maas-api-revoke-sub-adcb334f-cn4xj\" (UID: \"cab8160e-1169-4676-b764-92a3435dcc99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-adcb334f-cn4xj" Nov 02 06:53:58.326670 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:58.326597 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48757757-4e3d-404d-8af4-93ed81f60565-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-89d4bdbf-pw8vc\" (UID: \"48757757-4e3d-404d-8af4-93ed81f60565\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89d4bdbf-pw8vc" Nov 02 06:53:58.326670 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:58.326643 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cab8160e-1169-4676-b764-92a3435dcc99-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-adcb334f-cn4xj\" (UID: \"cab8160e-1169-4676-b764-92a3435dcc99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-adcb334f-cn4xj" Nov 02 06:53:58.326951 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:58.326673 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7pnt\" (UniqueName: \"kubernetes.io/projected/48757757-4e3d-404d-8af4-93ed81f60565-kube-api-access-b7pnt\") pod \"maas-api-revoke-sub-89d4bdbf-pw8vc\" (UID: \"48757757-4e3d-404d-8af4-93ed81f60565\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89d4bdbf-pw8vc" Nov 02 06:53:58.327388 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:58.327360 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48757757-4e3d-404d-8af4-93ed81f60565-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-89d4bdbf-pw8vc\" (UID: \"48757757-4e3d-404d-8af4-93ed81f60565\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89d4bdbf-pw8vc" Nov 02 06:53:58.327522 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:58.327432 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cab8160e-1169-4676-b764-92a3435dcc99-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-adcb334f-cn4xj\" (UID: \"cab8160e-1169-4676-b764-92a3435dcc99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-adcb334f-cn4xj" Nov 02 06:53:58.329078 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:58.329056 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48757757-4e3d-404d-8af4-93ed81f60565-maas-api-xks-ca\") pod \"maas-api-revoke-sub-89d4bdbf-pw8vc\" (UID: \"48757757-4e3d-404d-8af4-93ed81f60565\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89d4bdbf-pw8vc" Nov 02 06:53:58.329179 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:58.329090 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cab8160e-1169-4676-b764-92a3435dcc99-maas-api-xks-ca\") pod \"maas-api-revoke-sub-adcb334f-cn4xj\" (UID: \"cab8160e-1169-4676-b764-92a3435dcc99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-adcb334f-cn4xj" Nov 02 06:53:58.384212 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:58.384184 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7pnt\" (UniqueName: \"kubernetes.io/projected/48757757-4e3d-404d-8af4-93ed81f60565-kube-api-access-b7pnt\") pod \"maas-api-revoke-sub-89d4bdbf-pw8vc\" (UID: \"48757757-4e3d-404d-8af4-93ed81f60565\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89d4bdbf-pw8vc" Nov 02 06:53:58.385147 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:58.385120 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xqk8\" (UniqueName: \"kubernetes.io/projected/cab8160e-1169-4676-b764-92a3435dcc99-kube-api-access-9xqk8\") pod \"maas-api-revoke-sub-adcb334f-cn4xj\" (UID: \"cab8160e-1169-4676-b764-92a3435dcc99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-adcb334f-cn4xj" Nov 02 06:53:58.417685 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:58.417661 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-adcb334f-cn4xj" Nov 02 06:53:58.658341 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:58.658312 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89d4bdbf-pw8vc" Nov 02 06:53:58.796849 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:58.796818 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-adcb334f-cn4xj"] Nov 02 06:53:58.798256 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:53:58.798232 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcab8160e_1169_4676_b764_92a3435dcc99.slice/crio-682826fe589cbba2b4c408baf9abfaafe72adda33b396d839069f9163200e884 WatchSource:0}: Error finding container 682826fe589cbba2b4c408baf9abfaafe72adda33b396d839069f9163200e884: Status 404 returned error can't find the container with id 682826fe589cbba2b4c408baf9abfaafe72adda33b396d839069f9163200e884 Nov 02 06:53:58.998654 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:58.998544 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-adcb334f-cn4xj" event={"ID":"cab8160e-1169-4676-b764-92a3435dcc99","Type":"ContainerStarted","Data":"682826fe589cbba2b4c408baf9abfaafe72adda33b396d839069f9163200e884"} Nov 02 06:53:59.037587 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:53:59.037565 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89d4bdbf-pw8vc"] Nov 02 06:53:59.045034 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:53:59.045007 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48757757_4e3d_404d_8af4_93ed81f60565.slice/crio-5c1d9dfd5848cf5dcc8ff6cc10efb0b63ac41f70f329fa08966fb0a00d6610d2 WatchSource:0}: Error finding container 5c1d9dfd5848cf5dcc8ff6cc10efb0b63ac41f70f329fa08966fb0a00d6610d2: Status 404 returned error can't find the container with id 5c1d9dfd5848cf5dcc8ff6cc10efb0b63ac41f70f329fa08966fb0a00d6610d2 Nov 02 06:54:00.007337 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:54:00.007301 5916 generic.go:356] "Generic (PLEG): container finished" podID="cab8160e-1169-4676-b764-92a3435dcc99" containerID="b66ea00bf9c33cf0075b1d2ce34c30a67aff3aa516da411c1ed4ceaecc2f5b1e" exitCode=0 Nov 02 06:54:00.007501 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:54:00.007375 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-adcb334f-cn4xj" event={"ID":"cab8160e-1169-4676-b764-92a3435dcc99","Type":"ContainerDied","Data":"b66ea00bf9c33cf0075b1d2ce34c30a67aff3aa516da411c1ed4ceaecc2f5b1e"} Nov 02 06:54:00.009044 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:54:00.009021 5916 generic.go:356] "Generic (PLEG): container finished" podID="48757757-4e3d-404d-8af4-93ed81f60565" containerID="430f3dd7caf08ceac50097a8c8899b1ded2b83ec11bea0a5d3f427d6f4f8b1ad" exitCode=0 Nov 02 06:54:00.009157 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:54:00.009049 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89d4bdbf-pw8vc" event={"ID":"48757757-4e3d-404d-8af4-93ed81f60565","Type":"ContainerDied","Data":"430f3dd7caf08ceac50097a8c8899b1ded2b83ec11bea0a5d3f427d6f4f8b1ad"} Nov 02 06:54:00.009157 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:54:00.009075 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89d4bdbf-pw8vc" event={"ID":"48757757-4e3d-404d-8af4-93ed81f60565","Type":"ContainerStarted","Data":"5c1d9dfd5848cf5dcc8ff6cc10efb0b63ac41f70f329fa08966fb0a00d6610d2"} Nov 02 06:54:01.184443 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:54:01.184414 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89d4bdbf-pw8vc" Nov 02 06:54:01.217663 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:54:01.217642 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-adcb334f-cn4xj" Nov 02 06:54:01.263171 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:54:01.263144 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cab8160e-1169-4676-b764-92a3435dcc99-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cab8160e-1169-4676-b764-92a3435dcc99-maas-api-xks-ca\") pod \"cab8160e-1169-4676-b764-92a3435dcc99\" (UID: \"cab8160e-1169-4676-b764-92a3435dcc99\") " Nov 02 06:54:01.263309 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:54:01.263205 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cab8160e-1169-4676-b764-92a3435dcc99-kube-api-access-9xqk8\" (UniqueName: \"kubernetes.io/projected/cab8160e-1169-4676-b764-92a3435dcc99-kube-api-access-9xqk8\") pod \"cab8160e-1169-4676-b764-92a3435dcc99\" (UID: \"cab8160e-1169-4676-b764-92a3435dcc99\") " Nov 02 06:54:01.263309 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:54:01.263231 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/48757757-4e3d-404d-8af4-93ed81f60565-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48757757-4e3d-404d-8af4-93ed81f60565-maas-api-xks-ca\") pod \"48757757-4e3d-404d-8af4-93ed81f60565\" (UID: \"48757757-4e3d-404d-8af4-93ed81f60565\") " Nov 02 06:54:01.263309 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:54:01.263254 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cab8160e-1169-4676-b764-92a3435dcc99-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cab8160e-1169-4676-b764-92a3435dcc99-maas-api-openshift-ca\") pod \"cab8160e-1169-4676-b764-92a3435dcc99\" (UID: \"cab8160e-1169-4676-b764-92a3435dcc99\") " Nov 02 06:54:01.263309 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:54:01.263286 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48757757-4e3d-404d-8af4-93ed81f60565-kube-api-access-b7pnt\" (UniqueName: \"kubernetes.io/projected/48757757-4e3d-404d-8af4-93ed81f60565-kube-api-access-b7pnt\") pod \"48757757-4e3d-404d-8af4-93ed81f60565\" (UID: \"48757757-4e3d-404d-8af4-93ed81f60565\") " Nov 02 06:54:01.263528 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:54:01.263313 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48757757-4e3d-404d-8af4-93ed81f60565-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48757757-4e3d-404d-8af4-93ed81f60565-maas-api-openshift-ca\") pod \"48757757-4e3d-404d-8af4-93ed81f60565\" (UID: \"48757757-4e3d-404d-8af4-93ed81f60565\") " Nov 02 06:54:01.263685 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:54:01.263654 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cab8160e-1169-4676-b764-92a3435dcc99-maas-api-openshift-ca" pod "cab8160e-1169-4676-b764-92a3435dcc99" (UID: "cab8160e-1169-4676-b764-92a3435dcc99"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:54:01.263685 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:54:01.263678 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48757757-4e3d-404d-8af4-93ed81f60565-maas-api-openshift-ca" pod "48757757-4e3d-404d-8af4-93ed81f60565" (UID: "48757757-4e3d-404d-8af4-93ed81f60565"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:54:01.263827 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:54:01.263818 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cab8160e-1169-4676-b764-92a3435dcc99-maas-api-openshift-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:54:01.263888 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:54:01.263832 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48757757-4e3d-404d-8af4-93ed81f60565-maas-api-openshift-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:54:01.265517 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:54:01.265488 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48757757-4e3d-404d-8af4-93ed81f60565-maas-api-xks-ca" pod "48757757-4e3d-404d-8af4-93ed81f60565" (UID: "48757757-4e3d-404d-8af4-93ed81f60565"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:54:01.265623 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:54:01.265518 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48757757-4e3d-404d-8af4-93ed81f60565-kube-api-access-b7pnt" pod "48757757-4e3d-404d-8af4-93ed81f60565" (UID: "48757757-4e3d-404d-8af4-93ed81f60565"). InnerVolumeSpecName "kube-api-access-b7pnt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:54:01.265773 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:54:01.265754 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cab8160e-1169-4676-b764-92a3435dcc99-maas-api-xks-ca" pod "cab8160e-1169-4676-b764-92a3435dcc99" (UID: "cab8160e-1169-4676-b764-92a3435dcc99"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:54:01.265837 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:54:01.265768 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cab8160e-1169-4676-b764-92a3435dcc99-kube-api-access-9xqk8" pod "cab8160e-1169-4676-b764-92a3435dcc99" (UID: "cab8160e-1169-4676-b764-92a3435dcc99"). InnerVolumeSpecName "kube-api-access-9xqk8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:54:01.364829 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:54:01.364774 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cab8160e-1169-4676-b764-92a3435dcc99-maas-api-xks-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:54:01.364829 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:54:01.364806 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xqk8\" (UniqueName: \"kubernetes.io/projected/cab8160e-1169-4676-b764-92a3435dcc99-kube-api-access-9xqk8\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:54:01.364829 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:54:01.364816 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48757757-4e3d-404d-8af4-93ed81f60565-maas-api-xks-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:54:01.364829 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:54:01.364829 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7pnt\" (UniqueName: \"kubernetes.io/projected/48757757-4e3d-404d-8af4-93ed81f60565-kube-api-access-b7pnt\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:54:02.023932 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:54:02.023881 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89d4bdbf-pw8vc" Nov 02 06:54:02.023932 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:54:02.023919 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89d4bdbf-pw8vc" event={"ID":"48757757-4e3d-404d-8af4-93ed81f60565","Type":"ContainerDied","Data":"5c1d9dfd5848cf5dcc8ff6cc10efb0b63ac41f70f329fa08966fb0a00d6610d2"} Nov 02 06:54:02.024191 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:54:02.023965 5916 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5c1d9dfd5848cf5dcc8ff6cc10efb0b63ac41f70f329fa08966fb0a00d6610d2" Nov 02 06:54:02.025468 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:54:02.025451 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-adcb334f-cn4xj" Nov 02 06:54:02.025596 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:54:02.025488 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-adcb334f-cn4xj" event={"ID":"cab8160e-1169-4676-b764-92a3435dcc99","Type":"ContainerDied","Data":"682826fe589cbba2b4c408baf9abfaafe72adda33b396d839069f9163200e884"} Nov 02 06:54:02.025596 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:54:02.025530 5916 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="682826fe589cbba2b4c408baf9abfaafe72adda33b396d839069f9163200e884" Nov 02 06:54:19.896789 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:54:19.896747 5916 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p98cq" containerName="extractor" Nov 02 06:54:35.588649 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:54:35.588569 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7318b500-w4ldh"] Nov 02 06:54:35.589143 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:54:35.589106 5916 cpu_manager.go:414] "containerMap: removing container" podUID="48757757-4e3d-404d-8af4-93ed81f60565" containerName="revoke-keys" Nov 02 06:54:35.589143 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:54:35.589121 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48757757-4e3d-404d-8af4-93ed81f60565" containerName="revoke-keys" Nov 02 06:54:35.589215 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:54:35.589156 5916 cpu_manager.go:414] "containerMap: removing container" podUID="cab8160e-1169-4676-b764-92a3435dcc99" containerName="revoke-keys" Nov 02 06:54:35.589215 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:54:35.589162 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cab8160e-1169-4676-b764-92a3435dcc99" containerName="revoke-keys" Nov 02 06:54:35.589278 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:54:35.589241 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="48757757-4e3d-404d-8af4-93ed81f60565" containerName="revoke-keys" Nov 02 06:54:35.589278 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:54:35.589254 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="cab8160e-1169-4676-b764-92a3435dcc99" containerName="revoke-keys" Nov 02 06:54:35.593546 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:54:35.593526 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7318b500-w4ldh" Nov 02 06:54:35.650377 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:54:35.650347 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7318b500-w4ldh"] Nov 02 06:54:35.695618 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:54:35.695583 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 06:54:35.695618 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:54:35.695597 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-68l4r\"" Nov 02 06:54:35.709698 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:54:35.709672 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94db3b14-74f5-4309-bcc8-c0413b2beec2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7318b500-w4ldh\" (UID: \"94db3b14-74f5-4309-bcc8-c0413b2beec2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7318b500-w4ldh" Nov 02 06:54:35.709830 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:54:35.709719 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94db3b14-74f5-4309-bcc8-c0413b2beec2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7318b500-w4ldh\" (UID: \"94db3b14-74f5-4309-bcc8-c0413b2beec2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7318b500-w4ldh" Nov 02 06:54:35.709830 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:54:35.709741 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-779c5\" (UniqueName: \"kubernetes.io/projected/94db3b14-74f5-4309-bcc8-c0413b2beec2-kube-api-access-779c5\") pod \"maas-api-revoke-sub-7318b500-w4ldh\" (UID: \"94db3b14-74f5-4309-bcc8-c0413b2beec2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7318b500-w4ldh" Nov 02 06:54:35.811149 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:54:35.811116 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94db3b14-74f5-4309-bcc8-c0413b2beec2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7318b500-w4ldh\" (UID: \"94db3b14-74f5-4309-bcc8-c0413b2beec2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7318b500-w4ldh" Nov 02 06:54:35.811360 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:54:35.811174 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94db3b14-74f5-4309-bcc8-c0413b2beec2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7318b500-w4ldh\" (UID: \"94db3b14-74f5-4309-bcc8-c0413b2beec2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7318b500-w4ldh" Nov 02 06:54:35.811360 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:54:35.811198 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-779c5\" (UniqueName: \"kubernetes.io/projected/94db3b14-74f5-4309-bcc8-c0413b2beec2-kube-api-access-779c5\") pod \"maas-api-revoke-sub-7318b500-w4ldh\" (UID: \"94db3b14-74f5-4309-bcc8-c0413b2beec2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7318b500-w4ldh" Nov 02 06:54:35.811815 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:54:35.811789 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94db3b14-74f5-4309-bcc8-c0413b2beec2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7318b500-w4ldh\" (UID: \"94db3b14-74f5-4309-bcc8-c0413b2beec2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7318b500-w4ldh" Nov 02 06:54:35.813632 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:54:35.813604 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94db3b14-74f5-4309-bcc8-c0413b2beec2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7318b500-w4ldh\" (UID: \"94db3b14-74f5-4309-bcc8-c0413b2beec2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7318b500-w4ldh" Nov 02 06:54:35.868965 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:54:35.868875 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-779c5\" (UniqueName: \"kubernetes.io/projected/94db3b14-74f5-4309-bcc8-c0413b2beec2-kube-api-access-779c5\") pod \"maas-api-revoke-sub-7318b500-w4ldh\" (UID: \"94db3b14-74f5-4309-bcc8-c0413b2beec2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7318b500-w4ldh" Nov 02 06:54:35.906755 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:54:35.906722 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7318b500-w4ldh" Nov 02 06:54:36.298593 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:54:36.298563 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7318b500-w4ldh"] Nov 02 06:54:36.299426 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:54:36.299400 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94db3b14_74f5_4309_bcc8_c0413b2beec2.slice/crio-59793d4ccf6d58272573c8aa0ffc4ccdba19660d6b32db2d1f81f50e03ac4c8b WatchSource:0}: Error finding container 59793d4ccf6d58272573c8aa0ffc4ccdba19660d6b32db2d1f81f50e03ac4c8b: Status 404 returned error can't find the container with id 59793d4ccf6d58272573c8aa0ffc4ccdba19660d6b32db2d1f81f50e03ac4c8b Nov 02 06:54:37.299964 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:54:37.299917 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7318b500-w4ldh" event={"ID":"94db3b14-74f5-4309-bcc8-c0413b2beec2","Type":"ContainerStarted","Data":"5fc6ee3bab744309d6334dd10dcda78a68cc1317e25722f67d0f92d3d1aa96cb"} Nov 02 06:54:37.300467 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:54:37.300021 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7318b500-w4ldh" event={"ID":"94db3b14-74f5-4309-bcc8-c0413b2beec2","Type":"ContainerStarted","Data":"59793d4ccf6d58272573c8aa0ffc4ccdba19660d6b32db2d1f81f50e03ac4c8b"} Nov 02 06:54:37.417110 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:54:37.417051 5916 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7318b500-w4ldh" podStartSLOduration=2.417033542 podStartE2EDuration="2.417033542s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:54:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:54:37.415490387 +0000 UTC m=+2265.687310782" watchObservedRunningTime="2026-11-02 06:54:37.417033542 +0000 UTC m=+2265.688853938" Nov 02 06:54:38.309192 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:54:38.309153 5916 generic.go:356] "Generic (PLEG): container finished" podID="94db3b14-74f5-4309-bcc8-c0413b2beec2" containerID="5fc6ee3bab744309d6334dd10dcda78a68cc1317e25722f67d0f92d3d1aa96cb" exitCode=0 Nov 02 06:54:38.309622 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:54:38.309248 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7318b500-w4ldh" event={"ID":"94db3b14-74f5-4309-bcc8-c0413b2beec2","Type":"ContainerDied","Data":"5fc6ee3bab744309d6334dd10dcda78a68cc1317e25722f67d0f92d3d1aa96cb"} Nov 02 06:54:39.488627 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:54:39.488601 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7318b500-w4ldh" Nov 02 06:54:39.575700 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:54:39.575573 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94db3b14-74f5-4309-bcc8-c0413b2beec2-kube-api-access-779c5\" (UniqueName: \"kubernetes.io/projected/94db3b14-74f5-4309-bcc8-c0413b2beec2-kube-api-access-779c5\") pod \"94db3b14-74f5-4309-bcc8-c0413b2beec2\" (UID: \"94db3b14-74f5-4309-bcc8-c0413b2beec2\") " Nov 02 06:54:39.576779 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:54:39.575843 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/94db3b14-74f5-4309-bcc8-c0413b2beec2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94db3b14-74f5-4309-bcc8-c0413b2beec2-maas-api-xks-ca\") pod \"94db3b14-74f5-4309-bcc8-c0413b2beec2\" (UID: \"94db3b14-74f5-4309-bcc8-c0413b2beec2\") " Nov 02 06:54:39.576779 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:54:39.575977 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94db3b14-74f5-4309-bcc8-c0413b2beec2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94db3b14-74f5-4309-bcc8-c0413b2beec2-maas-api-openshift-ca\") pod \"94db3b14-74f5-4309-bcc8-c0413b2beec2\" (UID: \"94db3b14-74f5-4309-bcc8-c0413b2beec2\") " Nov 02 06:54:39.599220 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:54:39.576820 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94db3b14-74f5-4309-bcc8-c0413b2beec2-maas-api-openshift-ca" pod "94db3b14-74f5-4309-bcc8-c0413b2beec2" (UID: "94db3b14-74f5-4309-bcc8-c0413b2beec2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:54:39.599220 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:54:39.583780 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94db3b14-74f5-4309-bcc8-c0413b2beec2-kube-api-access-779c5" pod "94db3b14-74f5-4309-bcc8-c0413b2beec2" (UID: "94db3b14-74f5-4309-bcc8-c0413b2beec2"). InnerVolumeSpecName "kube-api-access-779c5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:54:39.599220 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:54:39.598570 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94db3b14-74f5-4309-bcc8-c0413b2beec2-maas-api-xks-ca" pod "94db3b14-74f5-4309-bcc8-c0413b2beec2" (UID: "94db3b14-74f5-4309-bcc8-c0413b2beec2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:54:39.677176 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:54:39.677140 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94db3b14-74f5-4309-bcc8-c0413b2beec2-maas-api-xks-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:54:39.677176 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:54:39.677167 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94db3b14-74f5-4309-bcc8-c0413b2beec2-maas-api-openshift-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:54:39.677176 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:54:39.677177 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-779c5\" (UniqueName: \"kubernetes.io/projected/94db3b14-74f5-4309-bcc8-c0413b2beec2-kube-api-access-779c5\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:54:40.325123 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:54:40.325088 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7318b500-w4ldh" event={"ID":"94db3b14-74f5-4309-bcc8-c0413b2beec2","Type":"ContainerDied","Data":"59793d4ccf6d58272573c8aa0ffc4ccdba19660d6b32db2d1f81f50e03ac4c8b"} Nov 02 06:54:40.325123 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:54:40.325120 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7318b500-w4ldh" Nov 02 06:54:40.325358 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:54:40.325139 5916 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="59793d4ccf6d58272573c8aa0ffc4ccdba19660d6b32db2d1f81f50e03ac4c8b" Nov 02 06:54:46.813702 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:54:46.813665 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-680b7b-r9nqd"] Nov 02 06:54:46.814260 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:54:46.814245 5916 cpu_manager.go:414] "containerMap: removing container" podUID="94db3b14-74f5-4309-bcc8-c0413b2beec2" containerName="revoke-keys" Nov 02 06:54:46.814305 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:54:46.814261 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94db3b14-74f5-4309-bcc8-c0413b2beec2" containerName="revoke-keys" Nov 02 06:54:46.814355 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:54:46.814346 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="94db3b14-74f5-4309-bcc8-c0413b2beec2" containerName="revoke-keys" Nov 02 06:54:46.817154 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:54:46.817138 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-680b7b-r9nqd" Nov 02 06:54:46.871199 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:54:46.871155 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 06:54:46.880248 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:54:46.880171 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33ac8085-f21e-49ee-89ab-c55e4c97dc64-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-680b7b-r9nqd\" (UID: \"33ac8085-f21e-49ee-89ab-c55e4c97dc64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-680b7b-r9nqd" Nov 02 06:54:46.880423 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:54:46.880284 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wz7sj\" (UniqueName: \"kubernetes.io/projected/33ac8085-f21e-49ee-89ab-c55e4c97dc64-kube-api-access-wz7sj\") pod \"maas-api-revoke-keys-e2e-worker-w4-680b7b-r9nqd\" (UID: \"33ac8085-f21e-49ee-89ab-c55e4c97dc64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-680b7b-r9nqd" Nov 02 06:54:46.880534 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:54:46.880505 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33ac8085-f21e-49ee-89ab-c55e4c97dc64-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-680b7b-r9nqd\" (UID: \"33ac8085-f21e-49ee-89ab-c55e4c97dc64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-680b7b-r9nqd" Nov 02 06:54:46.920936 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:54:46.920866 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-68l4r\"" Nov 02 06:54:46.924649 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:54:46.924623 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-680b7b-r9nqd"] Nov 02 06:54:46.981715 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:54:46.981681 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33ac8085-f21e-49ee-89ab-c55e4c97dc64-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-680b7b-r9nqd\" (UID: \"33ac8085-f21e-49ee-89ab-c55e4c97dc64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-680b7b-r9nqd" Nov 02 06:54:46.982001 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:54:46.981974 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wz7sj\" (UniqueName: \"kubernetes.io/projected/33ac8085-f21e-49ee-89ab-c55e4c97dc64-kube-api-access-wz7sj\") pod \"maas-api-revoke-keys-e2e-worker-w4-680b7b-r9nqd\" (UID: \"33ac8085-f21e-49ee-89ab-c55e4c97dc64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-680b7b-r9nqd" Nov 02 06:54:46.982155 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:54:46.982120 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33ac8085-f21e-49ee-89ab-c55e4c97dc64-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-680b7b-r9nqd\" (UID: \"33ac8085-f21e-49ee-89ab-c55e4c97dc64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-680b7b-r9nqd" Nov 02 06:54:46.983023 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:54:46.982999 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33ac8085-f21e-49ee-89ab-c55e4c97dc64-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-680b7b-r9nqd\" (UID: \"33ac8085-f21e-49ee-89ab-c55e4c97dc64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-680b7b-r9nqd" Nov 02 06:54:46.984779 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:54:46.984748 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33ac8085-f21e-49ee-89ab-c55e4c97dc64-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-680b7b-r9nqd\" (UID: \"33ac8085-f21e-49ee-89ab-c55e4c97dc64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-680b7b-r9nqd" Nov 02 06:54:47.039189 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:54:47.039163 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wz7sj\" (UniqueName: \"kubernetes.io/projected/33ac8085-f21e-49ee-89ab-c55e4c97dc64-kube-api-access-wz7sj\") pod \"maas-api-revoke-keys-e2e-worker-w4-680b7b-r9nqd\" (UID: \"33ac8085-f21e-49ee-89ab-c55e4c97dc64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-680b7b-r9nqd" Nov 02 06:54:47.131039 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:54:47.130938 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-680b7b-r9nqd" Nov 02 06:54:47.516580 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:54:47.516549 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-680b7b-r9nqd"] Nov 02 06:54:47.517504 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:54:47.517478 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33ac8085_f21e_49ee_89ab_c55e4c97dc64.slice/crio-a0e773ddd433c903f4c127ecb634b808f8d335fe8899fafc536e2aa9103ac9b6 WatchSource:0}: Error finding container a0e773ddd433c903f4c127ecb634b808f8d335fe8899fafc536e2aa9103ac9b6: Status 404 returned error can't find the container with id a0e773ddd433c903f4c127ecb634b808f8d335fe8899fafc536e2aa9103ac9b6 Nov 02 06:54:47.876022 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:54:47.875993 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-dcf9c3-xt2lx"] Nov 02 06:54:47.924422 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:54:47.924391 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-dcf9c3-xt2lx"] Nov 02 06:54:48.391727 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:54:48.391692 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-680b7b-r9nqd" event={"ID":"33ac8085-f21e-49ee-89ab-c55e4c97dc64","Type":"ContainerStarted","Data":"802cab162f948cd147fde88be449216111acb7b026d34eae16a5a25685fe1b04"} Nov 02 06:54:48.391929 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:54:48.391746 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-680b7b-r9nqd" event={"ID":"33ac8085-f21e-49ee-89ab-c55e4c97dc64","Type":"ContainerStarted","Data":"a0e773ddd433c903f4c127ecb634b808f8d335fe8899fafc536e2aa9103ac9b6"} Nov 02 06:54:48.506871 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:54:48.506817 5916 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-680b7b-r9nqd" podStartSLOduration=2.506803057 podStartE2EDuration="2.506803057s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:54:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:54:48.505297621 +0000 UTC m=+2276.777118016" watchObservedRunningTime="2026-11-02 06:54:48.506803057 +0000 UTC m=+2276.778623451" Nov 02 06:54:49.401130 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:54:49.401095 5916 generic.go:356] "Generic (PLEG): container finished" podID="33ac8085-f21e-49ee-89ab-c55e4c97dc64" containerID="802cab162f948cd147fde88be449216111acb7b026d34eae16a5a25685fe1b04" exitCode=0 Nov 02 06:54:49.401526 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:54:49.401172 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-680b7b-r9nqd" event={"ID":"33ac8085-f21e-49ee-89ab-c55e4c97dc64","Type":"ContainerDied","Data":"802cab162f948cd147fde88be449216111acb7b026d34eae16a5a25685fe1b04"} Nov 02 06:54:49.844532 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:54:49.844451 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18305d70-7fdf-4ffa-8d86-8c97ce8dea4d" path="/var/lib/kubelet/pods/18305d70-7fdf-4ffa-8d86-8c97ce8dea4d/volumes" Nov 02 06:54:49.894331 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:54:49.894299 5916 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p98cq" containerName="extractor" Nov 02 06:54:50.569566 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:54:50.569536 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-680b7b-r9nqd" Nov 02 06:54:50.726409 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:54:50.726319 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33ac8085-f21e-49ee-89ab-c55e4c97dc64-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33ac8085-f21e-49ee-89ab-c55e4c97dc64-maas-api-openshift-ca\") pod \"33ac8085-f21e-49ee-89ab-c55e4c97dc64\" (UID: \"33ac8085-f21e-49ee-89ab-c55e4c97dc64\") " Nov 02 06:54:50.726409 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:54:50.726382 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/33ac8085-f21e-49ee-89ab-c55e4c97dc64-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33ac8085-f21e-49ee-89ab-c55e4c97dc64-maas-api-xks-ca\") pod \"33ac8085-f21e-49ee-89ab-c55e4c97dc64\" (UID: \"33ac8085-f21e-49ee-89ab-c55e4c97dc64\") " Nov 02 06:54:50.726658 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:54:50.726467 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33ac8085-f21e-49ee-89ab-c55e4c97dc64-kube-api-access-wz7sj\" (UniqueName: \"kubernetes.io/projected/33ac8085-f21e-49ee-89ab-c55e4c97dc64-kube-api-access-wz7sj\") pod \"33ac8085-f21e-49ee-89ab-c55e4c97dc64\" (UID: \"33ac8085-f21e-49ee-89ab-c55e4c97dc64\") " Nov 02 06:54:50.726776 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:54:50.726746 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33ac8085-f21e-49ee-89ab-c55e4c97dc64-maas-api-openshift-ca" pod "33ac8085-f21e-49ee-89ab-c55e4c97dc64" (UID: "33ac8085-f21e-49ee-89ab-c55e4c97dc64"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:54:50.727095 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:54:50.727074 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33ac8085-f21e-49ee-89ab-c55e4c97dc64-maas-api-openshift-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:54:50.729640 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:54:50.729598 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33ac8085-f21e-49ee-89ab-c55e4c97dc64-maas-api-xks-ca" pod "33ac8085-f21e-49ee-89ab-c55e4c97dc64" (UID: "33ac8085-f21e-49ee-89ab-c55e4c97dc64"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:54:50.735100 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:54:50.735076 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33ac8085-f21e-49ee-89ab-c55e4c97dc64-kube-api-access-wz7sj" pod "33ac8085-f21e-49ee-89ab-c55e4c97dc64" (UID: "33ac8085-f21e-49ee-89ab-c55e4c97dc64"). InnerVolumeSpecName "kube-api-access-wz7sj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:54:50.828722 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:54:50.828629 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33ac8085-f21e-49ee-89ab-c55e4c97dc64-maas-api-xks-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:54:50.828722 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:54:50.828672 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wz7sj\" (UniqueName: \"kubernetes.io/projected/33ac8085-f21e-49ee-89ab-c55e4c97dc64-kube-api-access-wz7sj\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:54:51.283116 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:54:51.283081 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-680b7b/simulator-w4-680b7b-kserve-6c4f574c56-p5xtc"] Nov 02 06:54:51.416543 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:54:51.416512 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-680b7b-r9nqd" Nov 02 06:54:51.416726 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:54:51.416505 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-680b7b-r9nqd" event={"ID":"33ac8085-f21e-49ee-89ab-c55e4c97dc64","Type":"ContainerDied","Data":"a0e773ddd433c903f4c127ecb634b808f8d335fe8899fafc536e2aa9103ac9b6"} Nov 02 06:54:51.416726 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:54:51.416641 5916 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a0e773ddd433c903f4c127ecb634b808f8d335fe8899fafc536e2aa9103ac9b6" Nov 02 06:54:55.090444 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:54:55.090411 5916 scope.go:122] "RemoveContainer" containerID="984f18719a56ab9ec0ca47d5a43d173aed717c790ef8c08289fcb5af0ae4abc1" Nov 02 06:54:55.114039 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:54:55.113343 5916 scope.go:122] "RemoveContainer" containerID="984508b8d1db03bbd0ceae5b5f7490e08d5bd6c4a66253e78d848c1de9d92cde" Nov 02 06:54:55.124259 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:54:55.124232 5916 scope.go:122] "RemoveContainer" containerID="325eadb393dfd468183817f1c953d320b4e869eedc627c65c983b305687cef09" Nov 02 06:54:55.159857 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:54:55.159829 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-fhb78"] Nov 02 06:54:55.160143 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:54:55.160115 5916 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-fhb78" podUID="933acc24-86ce-4896-8daa-6d36d6ba7cae" containerName="istio-proxy" containerID="cri-o://82d9adc772f5aad1da662164df34432d566c2c35f74746d053c8600827ca565b" gracePeriod=30 Nov 02 06:54:56.227861 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:54:56.227827 5916 patch_prober.go:29] interesting pod/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-fhb78 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 06:54:56.228303 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:54:56.227880 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-fhb78" podUID="933acc24-86ce-4896-8daa-6d36d6ba7cae" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 06:54:58.211698 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:54:58.211662 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-680b7b-867b5cf97c-s84v9"] Nov 02 06:54:58.318022 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:54:58.317987 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-680b7b-775c58cc67-6d6rn"] Nov 02 06:55:00.419964 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:00.419930 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-fhb78" Nov 02 06:55:00.486090 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:00.486017 5916 generic.go:356] "Generic (PLEG): container finished" podID="933acc24-86ce-4896-8daa-6d36d6ba7cae" containerID="82d9adc772f5aad1da662164df34432d566c2c35f74746d053c8600827ca565b" exitCode=0 Nov 02 06:55:00.486090 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:00.486059 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-fhb78" Nov 02 06:55:00.486257 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:00.486109 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-fhb78" event={"ID":"933acc24-86ce-4896-8daa-6d36d6ba7cae","Type":"ContainerDied","Data":"82d9adc772f5aad1da662164df34432d566c2c35f74746d053c8600827ca565b"} Nov 02 06:55:00.486257 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:00.486164 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-fhb78" event={"ID":"933acc24-86ce-4896-8daa-6d36d6ba7cae","Type":"ContainerDied","Data":"c98800f64e8d6065fd240491b5f9eddcf816d58f2147026d07c909dc621a5cf8"} Nov 02 06:55:00.486257 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:00.486192 5916 scope.go:122] "RemoveContainer" containerID="82d9adc772f5aad1da662164df34432d566c2c35f74746d053c8600827ca565b" Nov 02 06:55:00.496559 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:00.496538 5916 scope.go:122] "RemoveContainer" containerID="82d9adc772f5aad1da662164df34432d566c2c35f74746d053c8600827ca565b" Nov 02 06:55:00.496807 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:55:00.496789 5916 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82d9adc772f5aad1da662164df34432d566c2c35f74746d053c8600827ca565b\": container with ID starting with 82d9adc772f5aad1da662164df34432d566c2c35f74746d053c8600827ca565b not found: ID does not exist" containerID="82d9adc772f5aad1da662164df34432d566c2c35f74746d053c8600827ca565b" Nov 02 06:55:00.496863 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:00.496815 5916 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82d9adc772f5aad1da662164df34432d566c2c35f74746d053c8600827ca565b"} err="failed to get container status \"82d9adc772f5aad1da662164df34432d566c2c35f74746d053c8600827ca565b\": rpc error: code = NotFound desc = could not find container \"82d9adc772f5aad1da662164df34432d566c2c35f74746d053c8600827ca565b\": container with ID starting with 82d9adc772f5aad1da662164df34432d566c2c35f74746d053c8600827ca565b not found: ID does not exist" Nov 02 06:55:00.551125 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:00.551101 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/933acc24-86ce-4896-8daa-6d36d6ba7cae-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/933acc24-86ce-4896-8daa-6d36d6ba7cae-istio-podinfo\") pod \"933acc24-86ce-4896-8daa-6d36d6ba7cae\" (UID: \"933acc24-86ce-4896-8daa-6d36d6ba7cae\") " Nov 02 06:55:00.551236 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:00.551140 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/933acc24-86ce-4896-8daa-6d36d6ba7cae-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/933acc24-86ce-4896-8daa-6d36d6ba7cae-istiod-ca-cert\") pod \"933acc24-86ce-4896-8daa-6d36d6ba7cae\" (UID: \"933acc24-86ce-4896-8daa-6d36d6ba7cae\") " Nov 02 06:55:00.551236 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:00.551195 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/933acc24-86ce-4896-8daa-6d36d6ba7cae-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/933acc24-86ce-4896-8daa-6d36d6ba7cae-workload-socket\") pod \"933acc24-86ce-4896-8daa-6d36d6ba7cae\" (UID: \"933acc24-86ce-4896-8daa-6d36d6ba7cae\") " Nov 02 06:55:00.551236 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:00.551215 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/933acc24-86ce-4896-8daa-6d36d6ba7cae-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/933acc24-86ce-4896-8daa-6d36d6ba7cae-istio-data\") pod \"933acc24-86ce-4896-8daa-6d36d6ba7cae\" (UID: \"933acc24-86ce-4896-8daa-6d36d6ba7cae\") " Nov 02 06:55:00.551345 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:00.551248 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/933acc24-86ce-4896-8daa-6d36d6ba7cae-kube-api-access-xx472\" (UniqueName: \"kubernetes.io/projected/933acc24-86ce-4896-8daa-6d36d6ba7cae-kube-api-access-xx472\") pod \"933acc24-86ce-4896-8daa-6d36d6ba7cae\" (UID: \"933acc24-86ce-4896-8daa-6d36d6ba7cae\") " Nov 02 06:55:00.551345 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:00.551300 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/933acc24-86ce-4896-8daa-6d36d6ba7cae-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/933acc24-86ce-4896-8daa-6d36d6ba7cae-credential-socket\") pod \"933acc24-86ce-4896-8daa-6d36d6ba7cae\" (UID: \"933acc24-86ce-4896-8daa-6d36d6ba7cae\") " Nov 02 06:55:00.551433 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:00.551347 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/933acc24-86ce-4896-8daa-6d36d6ba7cae-istio-token\" (UniqueName: \"kubernetes.io/projected/933acc24-86ce-4896-8daa-6d36d6ba7cae-istio-token\") pod \"933acc24-86ce-4896-8daa-6d36d6ba7cae\" (UID: \"933acc24-86ce-4896-8daa-6d36d6ba7cae\") " Nov 02 06:55:00.551433 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:00.551397 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/933acc24-86ce-4896-8daa-6d36d6ba7cae-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/933acc24-86ce-4896-8daa-6d36d6ba7cae-istio-envoy\") pod \"933acc24-86ce-4896-8daa-6d36d6ba7cae\" (UID: \"933acc24-86ce-4896-8daa-6d36d6ba7cae\") " Nov 02 06:55:00.551539 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:00.551435 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/933acc24-86ce-4896-8daa-6d36d6ba7cae-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/933acc24-86ce-4896-8daa-6d36d6ba7cae-workload-certs\") pod \"933acc24-86ce-4896-8daa-6d36d6ba7cae\" (UID: \"933acc24-86ce-4896-8daa-6d36d6ba7cae\") " Nov 02 06:55:00.551539 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:00.551476 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/933acc24-86ce-4896-8daa-6d36d6ba7cae-istio-data" pod "933acc24-86ce-4896-8daa-6d36d6ba7cae" (UID: "933acc24-86ce-4896-8daa-6d36d6ba7cae"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:55:00.551715 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:00.551692 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/933acc24-86ce-4896-8daa-6d36d6ba7cae-istiod-ca-cert" pod "933acc24-86ce-4896-8daa-6d36d6ba7cae" (UID: "933acc24-86ce-4896-8daa-6d36d6ba7cae"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:55:00.551805 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:00.551706 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/933acc24-86ce-4896-8daa-6d36d6ba7cae-credential-socket" pod "933acc24-86ce-4896-8daa-6d36d6ba7cae" (UID: "933acc24-86ce-4896-8daa-6d36d6ba7cae"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:55:00.551865 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:00.551826 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/933acc24-86ce-4896-8daa-6d36d6ba7cae-workload-socket" pod "933acc24-86ce-4896-8daa-6d36d6ba7cae" (UID: "933acc24-86ce-4896-8daa-6d36d6ba7cae"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:55:00.551943 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:00.551886 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/933acc24-86ce-4896-8daa-6d36d6ba7cae-workload-certs" pod "933acc24-86ce-4896-8daa-6d36d6ba7cae" (UID: "933acc24-86ce-4896-8daa-6d36d6ba7cae"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:55:00.552064 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:00.552048 5916 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/933acc24-86ce-4896-8daa-6d36d6ba7cae-istiod-ca-cert\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:55:00.552119 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:00.552071 5916 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/933acc24-86ce-4896-8daa-6d36d6ba7cae-workload-socket\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:55:00.552119 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:00.552084 5916 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/933acc24-86ce-4896-8daa-6d36d6ba7cae-istio-data\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:55:00.552119 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:00.552098 5916 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/933acc24-86ce-4896-8daa-6d36d6ba7cae-credential-socket\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:55:00.552119 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:00.552111 5916 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/933acc24-86ce-4896-8daa-6d36d6ba7cae-workload-certs\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:55:00.553477 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:00.553456 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/933acc24-86ce-4896-8daa-6d36d6ba7cae-kube-api-access-xx472" pod "933acc24-86ce-4896-8daa-6d36d6ba7cae" (UID: "933acc24-86ce-4896-8daa-6d36d6ba7cae"). InnerVolumeSpecName "kube-api-access-xx472". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:55:00.553564 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:00.553486 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/933acc24-86ce-4896-8daa-6d36d6ba7cae-istio-token" pod "933acc24-86ce-4896-8daa-6d36d6ba7cae" (UID: "933acc24-86ce-4896-8daa-6d36d6ba7cae"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:55:00.553634 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:00.553613 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/933acc24-86ce-4896-8daa-6d36d6ba7cae-istio-podinfo" pod "933acc24-86ce-4896-8daa-6d36d6ba7cae" (UID: "933acc24-86ce-4896-8daa-6d36d6ba7cae"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 06:55:00.553695 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:00.553655 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/933acc24-86ce-4896-8daa-6d36d6ba7cae-istio-envoy" pod "933acc24-86ce-4896-8daa-6d36d6ba7cae" (UID: "933acc24-86ce-4896-8daa-6d36d6ba7cae"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:55:00.653770 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:00.653733 5916 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/933acc24-86ce-4896-8daa-6d36d6ba7cae-istio-envoy\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:55:00.653770 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:00.653760 5916 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/933acc24-86ce-4896-8daa-6d36d6ba7cae-istio-podinfo\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:55:00.653770 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:00.653770 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xx472\" (UniqueName: \"kubernetes.io/projected/933acc24-86ce-4896-8daa-6d36d6ba7cae-kube-api-access-xx472\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:55:00.653770 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:00.653779 5916 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/933acc24-86ce-4896-8daa-6d36d6ba7cae-istio-token\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:55:00.908878 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:00.908847 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-fhb78"] Nov 02 06:55:00.911981 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:00.911957 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-fhb78"] Nov 02 06:55:01.202026 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:01.201993 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-fb1398-54468d6bf5-wpgxd"] Nov 02 06:55:01.202253 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:01.202227 5916 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-fb1398-54468d6bf5-wpgxd" podUID="50326ab7-9415-4017-ab7a-a6b04c2d222f" containerName="maas-api" containerID="cri-o://2bc46a298a74341a8dbca8e89f8991eec247a202535d2d1a649b3454e14aba2f" gracePeriod=30 Nov 02 06:55:01.502570 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:01.502526 5916 generic.go:356] "Generic (PLEG): container finished" podID="50326ab7-9415-4017-ab7a-a6b04c2d222f" containerID="2bc46a298a74341a8dbca8e89f8991eec247a202535d2d1a649b3454e14aba2f" exitCode=0 Nov 02 06:55:01.503037 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:01.502620 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-fb1398-54468d6bf5-wpgxd" event={"ID":"50326ab7-9415-4017-ab7a-a6b04c2d222f","Type":"ContainerDied","Data":"2bc46a298a74341a8dbca8e89f8991eec247a202535d2d1a649b3454e14aba2f"} Nov 02 06:55:01.518402 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:01.518359 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-fb1398-54468d6bf5-wpgxd" Nov 02 06:55:01.665264 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:01.665226 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50326ab7-9415-4017-ab7a-a6b04c2d222f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50326ab7-9415-4017-ab7a-a6b04c2d222f-openshift-service-ca\") pod \"50326ab7-9415-4017-ab7a-a6b04c2d222f\" (UID: \"50326ab7-9415-4017-ab7a-a6b04c2d222f\") " Nov 02 06:55:01.665458 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:01.665424 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/50326ab7-9415-4017-ab7a-a6b04c2d222f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/50326ab7-9415-4017-ab7a-a6b04c2d222f-metrics-certs\") pod \"50326ab7-9415-4017-ab7a-a6b04c2d222f\" (UID: \"50326ab7-9415-4017-ab7a-a6b04c2d222f\") " Nov 02 06:55:01.665531 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:01.665483 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50326ab7-9415-4017-ab7a-a6b04c2d222f-kube-api-access-8tcmm\" (UniqueName: \"kubernetes.io/projected/50326ab7-9415-4017-ab7a-a6b04c2d222f-kube-api-access-8tcmm\") pod \"50326ab7-9415-4017-ab7a-a6b04c2d222f\" (UID: \"50326ab7-9415-4017-ab7a-a6b04c2d222f\") " Nov 02 06:55:01.665592 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:01.665540 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/50326ab7-9415-4017-ab7a-a6b04c2d222f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50326ab7-9415-4017-ab7a-a6b04c2d222f-maas-api-tls\") pod \"50326ab7-9415-4017-ab7a-a6b04c2d222f\" (UID: \"50326ab7-9415-4017-ab7a-a6b04c2d222f\") " Nov 02 06:55:01.665592 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:01.665542 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50326ab7-9415-4017-ab7a-a6b04c2d222f-openshift-service-ca" pod "50326ab7-9415-4017-ab7a-a6b04c2d222f" (UID: "50326ab7-9415-4017-ab7a-a6b04c2d222f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:55:01.666167 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:01.666125 5916 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50326ab7-9415-4017-ab7a-a6b04c2d222f-openshift-service-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:55:01.668078 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:01.668049 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50326ab7-9415-4017-ab7a-a6b04c2d222f-maas-api-tls" pod "50326ab7-9415-4017-ab7a-a6b04c2d222f" (UID: "50326ab7-9415-4017-ab7a-a6b04c2d222f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:55:01.668553 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:01.668527 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50326ab7-9415-4017-ab7a-a6b04c2d222f-metrics-certs" pod "50326ab7-9415-4017-ab7a-a6b04c2d222f" (UID: "50326ab7-9415-4017-ab7a-a6b04c2d222f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:55:01.669235 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:01.669211 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50326ab7-9415-4017-ab7a-a6b04c2d222f-kube-api-access-8tcmm" pod "50326ab7-9415-4017-ab7a-a6b04c2d222f" (UID: "50326ab7-9415-4017-ab7a-a6b04c2d222f"). InnerVolumeSpecName "kube-api-access-8tcmm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:55:01.771992 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:01.769676 5916 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50326ab7-9415-4017-ab7a-a6b04c2d222f-metrics-certs\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:55:01.771992 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:01.769713 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8tcmm\" (UniqueName: \"kubernetes.io/projected/50326ab7-9415-4017-ab7a-a6b04c2d222f-kube-api-access-8tcmm\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:55:01.771992 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:01.769729 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50326ab7-9415-4017-ab7a-a6b04c2d222f-maas-api-tls\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:55:01.843471 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:01.843440 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="933acc24-86ce-4896-8daa-6d36d6ba7cae" path="/var/lib/kubelet/pods/933acc24-86ce-4896-8daa-6d36d6ba7cae/volumes" Nov 02 06:55:02.515634 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:02.515596 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-fb1398-54468d6bf5-wpgxd" Nov 02 06:55:02.515634 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:02.515601 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-fb1398-54468d6bf5-wpgxd" event={"ID":"50326ab7-9415-4017-ab7a-a6b04c2d222f","Type":"ContainerDied","Data":"b07844dbf8226522c69852688bf384879b4142e8a8a1b06935126ad08cdc4198"} Nov 02 06:55:02.516303 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:02.515697 5916 scope.go:122] "RemoveContainer" containerID="2bc46a298a74341a8dbca8e89f8991eec247a202535d2d1a649b3454e14aba2f" Nov 02 06:55:02.681946 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:02.681899 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-fb1398-54468d6bf5-wpgxd"] Nov 02 06:55:02.731272 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:02.731229 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-fb1398-54468d6bf5-wpgxd"] Nov 02 06:55:03.212058 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:03.211997 5916 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-680b7b-867b5cf97c-s84v9" podUID="144a4e28-24fb-467e-8e2a-2e26736c9589" containerName="payload-processing" containerID="cri-o://7121fef955574f68cbc728df2965193b959aad9917544b2c19f134ba0cbd6ea3" gracePeriod=25 Nov 02 06:55:03.279957 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:03.279901 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-fb1398-gpj8g"] Nov 02 06:55:03.282853 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:03.282830 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-fb1398-gpj8g"] Nov 02 06:55:03.318482 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:03.318444 5916 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-680b7b-775c58cc67-6d6rn" podUID="90130906-592a-4243-9697-6daa79939293" containerName="payload-pre-processing" containerID="cri-o://7dbfa1661aa93f68b1c6bd6df01d8439677c9f686228ff490143ba51750c882d" gracePeriod=25 Nov 02 06:55:03.842266 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:03.842229 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50326ab7-9415-4017-ab7a-a6b04c2d222f" path="/var/lib/kubelet/pods/50326ab7-9415-4017-ab7a-a6b04c2d222f/volumes" Nov 02 06:55:03.842678 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:03.842665 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c7b54e7-87ea-48d5-b868-7a81da8f2099" path="/var/lib/kubelet/pods/6c7b54e7-87ea-48d5-b868-7a81da8f2099/volumes" Nov 02 06:55:06.303397 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:06.303358 5916 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-680b7b/simulator-w4-680b7b-kserve-6c4f574c56-p5xtc" podUID="be8a1039-5934-4885-93fe-070575f936ec" containerName="main" containerID="cri-o://a928abd80efbbfa38f816d3f018ed4f0c9a479d962794cd394ccba05a4b2fa80" gracePeriod=45 Nov 02 06:55:06.549689 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:06.549659 5916 generic.go:356] "Generic (PLEG): container finished" podID="be8a1039-5934-4885-93fe-070575f936ec" containerID="a928abd80efbbfa38f816d3f018ed4f0c9a479d962794cd394ccba05a4b2fa80" exitCode=0 Nov 02 06:55:06.549863 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:06.549730 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-680b7b/simulator-w4-680b7b-kserve-6c4f574c56-p5xtc" event={"ID":"be8a1039-5934-4885-93fe-070575f936ec","Type":"ContainerDied","Data":"a928abd80efbbfa38f816d3f018ed4f0c9a479d962794cd394ccba05a4b2fa80"} Nov 02 06:55:06.576019 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:06.575997 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-680b7b/simulator-w4-680b7b-kserve-6c4f574c56-p5xtc" Nov 02 06:55:06.623711 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:06.623686 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be8a1039-5934-4885-93fe-070575f936ec-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be8a1039-5934-4885-93fe-070575f936ec-tmp-dir\") pod \"be8a1039-5934-4885-93fe-070575f936ec\" (UID: \"be8a1039-5934-4885-93fe-070575f936ec\") " Nov 02 06:55:06.623846 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:06.623734 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be8a1039-5934-4885-93fe-070575f936ec-home\" (UniqueName: \"kubernetes.io/empty-dir/be8a1039-5934-4885-93fe-070575f936ec-home\") pod \"be8a1039-5934-4885-93fe-070575f936ec\" (UID: \"be8a1039-5934-4885-93fe-070575f936ec\") " Nov 02 06:55:06.623846 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:06.623757 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be8a1039-5934-4885-93fe-070575f936ec-dshm\" (UniqueName: \"kubernetes.io/empty-dir/be8a1039-5934-4885-93fe-070575f936ec-dshm\") pod \"be8a1039-5934-4885-93fe-070575f936ec\" (UID: \"be8a1039-5934-4885-93fe-070575f936ec\") " Nov 02 06:55:06.623846 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:06.623784 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/be8a1039-5934-4885-93fe-070575f936ec-tls-certs\" (UniqueName: \"kubernetes.io/secret/be8a1039-5934-4885-93fe-070575f936ec-tls-certs\") pod \"be8a1039-5934-4885-93fe-070575f936ec\" (UID: \"be8a1039-5934-4885-93fe-070575f936ec\") " Nov 02 06:55:06.623846 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:06.623824 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be8a1039-5934-4885-93fe-070575f936ec-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be8a1039-5934-4885-93fe-070575f936ec-model-cache\") pod \"be8a1039-5934-4885-93fe-070575f936ec\" (UID: \"be8a1039-5934-4885-93fe-070575f936ec\") " Nov 02 06:55:06.624082 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:06.623882 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be8a1039-5934-4885-93fe-070575f936ec-kube-api-access-kvvxp\" (UniqueName: \"kubernetes.io/projected/be8a1039-5934-4885-93fe-070575f936ec-kube-api-access-kvvxp\") pod \"be8a1039-5934-4885-93fe-070575f936ec\" (UID: \"be8a1039-5934-4885-93fe-070575f936ec\") " Nov 02 06:55:06.624082 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:06.624027 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be8a1039-5934-4885-93fe-070575f936ec-home" pod "be8a1039-5934-4885-93fe-070575f936ec" (UID: "be8a1039-5934-4885-93fe-070575f936ec"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:55:06.624082 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:06.624039 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be8a1039-5934-4885-93fe-070575f936ec-tmp-dir" pod "be8a1039-5934-4885-93fe-070575f936ec" (UID: "be8a1039-5934-4885-93fe-070575f936ec"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:55:06.624259 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:06.624137 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be8a1039-5934-4885-93fe-070575f936ec-model-cache" pod "be8a1039-5934-4885-93fe-070575f936ec" (UID: "be8a1039-5934-4885-93fe-070575f936ec"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:55:06.624456 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:06.624437 5916 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be8a1039-5934-4885-93fe-070575f936ec-model-cache\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:55:06.624456 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:06.624457 5916 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be8a1039-5934-4885-93fe-070575f936ec-tmp-dir\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:55:06.624614 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:06.624468 5916 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be8a1039-5934-4885-93fe-070575f936ec-home\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:55:06.626145 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:06.626114 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be8a1039-5934-4885-93fe-070575f936ec-kube-api-access-kvvxp" pod "be8a1039-5934-4885-93fe-070575f936ec" (UID: "be8a1039-5934-4885-93fe-070575f936ec"). InnerVolumeSpecName "kube-api-access-kvvxp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:55:06.626259 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:06.626157 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be8a1039-5934-4885-93fe-070575f936ec-tls-certs" pod "be8a1039-5934-4885-93fe-070575f936ec" (UID: "be8a1039-5934-4885-93fe-070575f936ec"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:55:06.626259 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:06.626168 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be8a1039-5934-4885-93fe-070575f936ec-dshm" pod "be8a1039-5934-4885-93fe-070575f936ec" (UID: "be8a1039-5934-4885-93fe-070575f936ec"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:55:06.725498 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:06.725465 5916 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be8a1039-5934-4885-93fe-070575f936ec-dshm\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:55:06.725498 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:06.725501 5916 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be8a1039-5934-4885-93fe-070575f936ec-tls-certs\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:55:06.725776 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:06.725518 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvvxp\" (UniqueName: \"kubernetes.io/projected/be8a1039-5934-4885-93fe-070575f936ec-kube-api-access-kvvxp\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:55:07.265508 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:07.265467 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-73e2cf7f-zjks2"] Nov 02 06:55:07.266726 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:07.266680 5916 cpu_manager.go:414] "containerMap: removing container" podUID="be8a1039-5934-4885-93fe-070575f936ec" containerName="main" Nov 02 06:55:07.266726 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:07.266706 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be8a1039-5934-4885-93fe-070575f936ec" containerName="main" Nov 02 06:55:07.266936 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:07.266753 5916 cpu_manager.go:414] "containerMap: removing container" podUID="33ac8085-f21e-49ee-89ab-c55e4c97dc64" containerName="revoke-keys" Nov 02 06:55:07.266936 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:07.266761 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33ac8085-f21e-49ee-89ab-c55e4c97dc64" containerName="revoke-keys" Nov 02 06:55:07.266936 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:07.266788 5916 cpu_manager.go:414] "containerMap: removing container" podUID="933acc24-86ce-4896-8daa-6d36d6ba7cae" containerName="istio-proxy" Nov 02 06:55:07.266936 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:07.266797 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="933acc24-86ce-4896-8daa-6d36d6ba7cae" containerName="istio-proxy" Nov 02 06:55:07.266936 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:07.266875 5916 cpu_manager.go:414] "containerMap: removing container" podUID="50326ab7-9415-4017-ab7a-a6b04c2d222f" containerName="maas-api" Nov 02 06:55:07.266936 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:07.266882 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50326ab7-9415-4017-ab7a-a6b04c2d222f" containerName="maas-api" Nov 02 06:55:07.267217 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:07.267156 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="50326ab7-9415-4017-ab7a-a6b04c2d222f" containerName="maas-api" Nov 02 06:55:07.267217 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:07.267183 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="be8a1039-5934-4885-93fe-070575f936ec" containerName="main" Nov 02 06:55:07.267217 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:07.267202 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="33ac8085-f21e-49ee-89ab-c55e4c97dc64" containerName="revoke-keys" Nov 02 06:55:07.267217 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:07.267213 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="933acc24-86ce-4896-8daa-6d36d6ba7cae" containerName="istio-proxy" Nov 02 06:55:07.275160 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:07.275136 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73e2cf7f-zjks2" Nov 02 06:55:07.311879 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:07.311846 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9a43fa3-7tt7w"] Nov 02 06:55:07.315884 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:07.315864 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9a43fa3-7tt7w" Nov 02 06:55:07.323310 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:07.323286 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-73e2cf7f-zjks2"] Nov 02 06:55:07.330984 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:07.330036 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 06:55:07.358385 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:07.358355 5916 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 06:55:07.358385 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:07.358379 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-68l4r\"" Nov 02 06:55:07.358598 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:07.358435 5916 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 06:55:07.417156 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:07.417122 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9a43fa3-7tt7w"] Nov 02 06:55:07.435237 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:07.435189 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b23fcf38-1c33-4668-890c-436e1b54b7a4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e9a43fa3-7tt7w\" (UID: \"b23fcf38-1c33-4668-890c-436e1b54b7a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9a43fa3-7tt7w" Nov 02 06:55:07.435437 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:07.435281 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39513a10-9b43-4963-8b4b-265187974870-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-73e2cf7f-zjks2\" (UID: \"39513a10-9b43-4963-8b4b-265187974870\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73e2cf7f-zjks2" Nov 02 06:55:07.435437 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:07.435415 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39513a10-9b43-4963-8b4b-265187974870-maas-api-xks-ca\") pod \"maas-api-revoke-sub-73e2cf7f-zjks2\" (UID: \"39513a10-9b43-4963-8b4b-265187974870\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73e2cf7f-zjks2" Nov 02 06:55:07.435561 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:07.435478 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8r4dt\" (UniqueName: \"kubernetes.io/projected/39513a10-9b43-4963-8b4b-265187974870-kube-api-access-8r4dt\") pod \"maas-api-revoke-sub-73e2cf7f-zjks2\" (UID: \"39513a10-9b43-4963-8b4b-265187974870\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73e2cf7f-zjks2" Nov 02 06:55:07.435561 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:07.435527 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b23fcf38-1c33-4668-890c-436e1b54b7a4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e9a43fa3-7tt7w\" (UID: \"b23fcf38-1c33-4668-890c-436e1b54b7a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9a43fa3-7tt7w" Nov 02 06:55:07.435561 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:07.435551 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82zqj\" (UniqueName: \"kubernetes.io/projected/b23fcf38-1c33-4668-890c-436e1b54b7a4-kube-api-access-82zqj\") pod \"maas-api-revoke-sub-e9a43fa3-7tt7w\" (UID: \"b23fcf38-1c33-4668-890c-436e1b54b7a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9a43fa3-7tt7w" Nov 02 06:55:07.537120 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:07.537035 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39513a10-9b43-4963-8b4b-265187974870-maas-api-xks-ca\") pod \"maas-api-revoke-sub-73e2cf7f-zjks2\" (UID: \"39513a10-9b43-4963-8b4b-265187974870\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73e2cf7f-zjks2" Nov 02 06:55:07.537120 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:07.537098 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8r4dt\" (UniqueName: \"kubernetes.io/projected/39513a10-9b43-4963-8b4b-265187974870-kube-api-access-8r4dt\") pod \"maas-api-revoke-sub-73e2cf7f-zjks2\" (UID: \"39513a10-9b43-4963-8b4b-265187974870\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73e2cf7f-zjks2" Nov 02 06:55:07.537353 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:07.537139 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b23fcf38-1c33-4668-890c-436e1b54b7a4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e9a43fa3-7tt7w\" (UID: \"b23fcf38-1c33-4668-890c-436e1b54b7a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9a43fa3-7tt7w" Nov 02 06:55:07.537353 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:07.537160 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82zqj\" (UniqueName: \"kubernetes.io/projected/b23fcf38-1c33-4668-890c-436e1b54b7a4-kube-api-access-82zqj\") pod \"maas-api-revoke-sub-e9a43fa3-7tt7w\" (UID: \"b23fcf38-1c33-4668-890c-436e1b54b7a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9a43fa3-7tt7w" Nov 02 06:55:07.537353 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:07.537214 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b23fcf38-1c33-4668-890c-436e1b54b7a4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e9a43fa3-7tt7w\" (UID: \"b23fcf38-1c33-4668-890c-436e1b54b7a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9a43fa3-7tt7w" Nov 02 06:55:07.537353 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:07.537264 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39513a10-9b43-4963-8b4b-265187974870-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-73e2cf7f-zjks2\" (UID: \"39513a10-9b43-4963-8b4b-265187974870\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73e2cf7f-zjks2" Nov 02 06:55:07.538131 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:07.538098 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39513a10-9b43-4963-8b4b-265187974870-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-73e2cf7f-zjks2\" (UID: \"39513a10-9b43-4963-8b4b-265187974870\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73e2cf7f-zjks2" Nov 02 06:55:07.538531 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:07.538505 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b23fcf38-1c33-4668-890c-436e1b54b7a4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e9a43fa3-7tt7w\" (UID: \"b23fcf38-1c33-4668-890c-436e1b54b7a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9a43fa3-7tt7w" Nov 02 06:55:07.539713 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:07.539691 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39513a10-9b43-4963-8b4b-265187974870-maas-api-xks-ca\") pod \"maas-api-revoke-sub-73e2cf7f-zjks2\" (UID: \"39513a10-9b43-4963-8b4b-265187974870\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73e2cf7f-zjks2" Nov 02 06:55:07.540778 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:07.540757 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b23fcf38-1c33-4668-890c-436e1b54b7a4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e9a43fa3-7tt7w\" (UID: \"b23fcf38-1c33-4668-890c-436e1b54b7a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9a43fa3-7tt7w" Nov 02 06:55:07.560969 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:07.560935 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-680b7b/simulator-w4-680b7b-kserve-6c4f574c56-p5xtc" event={"ID":"be8a1039-5934-4885-93fe-070575f936ec","Type":"ContainerDied","Data":"afbcb7dee987643a452dc61651bc8586babebf5bd24118a68a970ee48c611716"} Nov 02 06:55:07.561082 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:07.560973 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-680b7b/simulator-w4-680b7b-kserve-6c4f574c56-p5xtc" Nov 02 06:55:07.561082 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:07.560999 5916 scope.go:122] "RemoveContainer" containerID="a928abd80efbbfa38f816d3f018ed4f0c9a479d962794cd394ccba05a4b2fa80" Nov 02 06:55:07.598358 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:07.598334 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82zqj\" (UniqueName: \"kubernetes.io/projected/b23fcf38-1c33-4668-890c-436e1b54b7a4-kube-api-access-82zqj\") pod \"maas-api-revoke-sub-e9a43fa3-7tt7w\" (UID: \"b23fcf38-1c33-4668-890c-436e1b54b7a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9a43fa3-7tt7w" Nov 02 06:55:07.598789 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:07.598770 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8r4dt\" (UniqueName: \"kubernetes.io/projected/39513a10-9b43-4963-8b4b-265187974870-kube-api-access-8r4dt\") pod \"maas-api-revoke-sub-73e2cf7f-zjks2\" (UID: \"39513a10-9b43-4963-8b4b-265187974870\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73e2cf7f-zjks2" Nov 02 06:55:07.635406 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:07.635386 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9a43fa3-7tt7w" Nov 02 06:55:07.732917 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:07.732877 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-680b7b/simulator-w4-680b7b-kserve-6c4f574c56-p5xtc"] Nov 02 06:55:07.737774 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:07.737742 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-680b7b/simulator-w4-680b7b-kserve-6c4f574c56-p5xtc"] Nov 02 06:55:07.844348 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:07.844263 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be8a1039-5934-4885-93fe-070575f936ec" path="/var/lib/kubelet/pods/be8a1039-5934-4885-93fe-070575f936ec/volumes" Nov 02 06:55:07.889602 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:07.889574 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73e2cf7f-zjks2" Nov 02 06:55:08.020596 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:08.020562 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9a43fa3-7tt7w"] Nov 02 06:55:08.024559 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:55:08.024524 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb23fcf38_1c33_4668_890c_436e1b54b7a4.slice/crio-9534b546bca349e84a2e7e3ed2c5b7b151a0076dfc5ee005174237c384e67af9 WatchSource:0}: Error finding container 9534b546bca349e84a2e7e3ed2c5b7b151a0076dfc5ee005174237c384e67af9: Status 404 returned error can't find the container with id 9534b546bca349e84a2e7e3ed2c5b7b151a0076dfc5ee005174237c384e67af9 Nov 02 06:55:08.281847 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:08.281819 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-73e2cf7f-zjks2"] Nov 02 06:55:08.282602 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:55:08.282566 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39513a10_9b43_4963_8b4b_265187974870.slice/crio-33de7292f812a720e910b418170a255d2941a103685e277717d22ae1896c8d98 WatchSource:0}: Error finding container 33de7292f812a720e910b418170a255d2941a103685e277717d22ae1896c8d98: Status 404 returned error can't find the container with id 33de7292f812a720e910b418170a255d2941a103685e277717d22ae1896c8d98 Nov 02 06:55:08.574393 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:08.574362 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9a43fa3-7tt7w" event={"ID":"b23fcf38-1c33-4668-890c-436e1b54b7a4","Type":"ContainerStarted","Data":"9f736606ebb823adf23b5f5075bc38b3de3a188e6592a490290732db3f028e00"} Nov 02 06:55:08.575373 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:08.574416 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9a43fa3-7tt7w" event={"ID":"b23fcf38-1c33-4668-890c-436e1b54b7a4","Type":"ContainerStarted","Data":"9534b546bca349e84a2e7e3ed2c5b7b151a0076dfc5ee005174237c384e67af9"} Nov 02 06:55:08.576148 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:08.576127 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73e2cf7f-zjks2" event={"ID":"39513a10-9b43-4963-8b4b-265187974870","Type":"ContainerStarted","Data":"9db1cab1f81d6480af2d5503bce80d922517edb60cf9e1ed18ae0109fad92035"} Nov 02 06:55:08.576252 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:08.576175 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73e2cf7f-zjks2" event={"ID":"39513a10-9b43-4963-8b4b-265187974870","Type":"ContainerStarted","Data":"33de7292f812a720e910b418170a255d2941a103685e277717d22ae1896c8d98"} Nov 02 06:55:08.689986 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:08.689934 5916 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9a43fa3-7tt7w" podStartSLOduration=1.6899197099999999 podStartE2EDuration="1.68991971s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:55:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:55:08.688147396 +0000 UTC m=+2296.959967790" watchObservedRunningTime="2026-11-02 06:55:08.68991971 +0000 UTC m=+2296.961740097" Nov 02 06:55:08.803223 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:08.803116 5916 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73e2cf7f-zjks2" podStartSLOduration=1.803102665 podStartE2EDuration="1.803102665s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:55:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:55:08.801752597 +0000 UTC m=+2297.073572992" watchObservedRunningTime="2026-11-02 06:55:08.803102665 +0000 UTC m=+2297.074923059" Nov 02 06:55:09.584720 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:09.584629 5916 generic.go:356] "Generic (PLEG): container finished" podID="b23fcf38-1c33-4668-890c-436e1b54b7a4" containerID="9f736606ebb823adf23b5f5075bc38b3de3a188e6592a490290732db3f028e00" exitCode=0 Nov 02 06:55:09.584720 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:09.584706 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9a43fa3-7tt7w" event={"ID":"b23fcf38-1c33-4668-890c-436e1b54b7a4","Type":"ContainerDied","Data":"9f736606ebb823adf23b5f5075bc38b3de3a188e6592a490290732db3f028e00"} Nov 02 06:55:09.586321 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:09.586299 5916 generic.go:356] "Generic (PLEG): container finished" podID="39513a10-9b43-4963-8b4b-265187974870" containerID="9db1cab1f81d6480af2d5503bce80d922517edb60cf9e1ed18ae0109fad92035" exitCode=0 Nov 02 06:55:09.586430 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:09.586378 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73e2cf7f-zjks2" event={"ID":"39513a10-9b43-4963-8b4b-265187974870","Type":"ContainerDied","Data":"9db1cab1f81d6480af2d5503bce80d922517edb60cf9e1ed18ae0109fad92035"} Nov 02 06:55:10.756350 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:10.756327 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73e2cf7f-zjks2" Nov 02 06:55:10.776716 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:10.776681 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/39513a10-9b43-4963-8b4b-265187974870-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39513a10-9b43-4963-8b4b-265187974870-maas-api-xks-ca\") pod \"39513a10-9b43-4963-8b4b-265187974870\" (UID: \"39513a10-9b43-4963-8b4b-265187974870\") " Nov 02 06:55:10.776855 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:10.776731 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39513a10-9b43-4963-8b4b-265187974870-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39513a10-9b43-4963-8b4b-265187974870-maas-api-openshift-ca\") pod \"39513a10-9b43-4963-8b4b-265187974870\" (UID: \"39513a10-9b43-4963-8b4b-265187974870\") " Nov 02 06:55:10.776855 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:10.776814 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39513a10-9b43-4963-8b4b-265187974870-kube-api-access-8r4dt\" (UniqueName: \"kubernetes.io/projected/39513a10-9b43-4963-8b4b-265187974870-kube-api-access-8r4dt\") pod \"39513a10-9b43-4963-8b4b-265187974870\" (UID: \"39513a10-9b43-4963-8b4b-265187974870\") " Nov 02 06:55:10.777350 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:10.777306 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39513a10-9b43-4963-8b4b-265187974870-maas-api-openshift-ca" pod "39513a10-9b43-4963-8b4b-265187974870" (UID: "39513a10-9b43-4963-8b4b-265187974870"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:55:10.777466 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:10.777443 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39513a10-9b43-4963-8b4b-265187974870-maas-api-openshift-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:55:10.779090 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:10.779060 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39513a10-9b43-4963-8b4b-265187974870-maas-api-xks-ca" pod "39513a10-9b43-4963-8b4b-265187974870" (UID: "39513a10-9b43-4963-8b4b-265187974870"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:55:10.779187 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:10.779164 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39513a10-9b43-4963-8b4b-265187974870-kube-api-access-8r4dt" pod "39513a10-9b43-4963-8b4b-265187974870" (UID: "39513a10-9b43-4963-8b4b-265187974870"). InnerVolumeSpecName "kube-api-access-8r4dt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:55:10.796333 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:10.796310 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9a43fa3-7tt7w" Nov 02 06:55:10.882842 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:10.880839 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39513a10-9b43-4963-8b4b-265187974870-maas-api-xks-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:55:10.882842 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:10.880879 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8r4dt\" (UniqueName: \"kubernetes.io/projected/39513a10-9b43-4963-8b4b-265187974870-kube-api-access-8r4dt\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:55:10.981367 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:10.981333 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b23fcf38-1c33-4668-890c-436e1b54b7a4-kube-api-access-82zqj\" (UniqueName: \"kubernetes.io/projected/b23fcf38-1c33-4668-890c-436e1b54b7a4-kube-api-access-82zqj\") pod \"b23fcf38-1c33-4668-890c-436e1b54b7a4\" (UID: \"b23fcf38-1c33-4668-890c-436e1b54b7a4\") " Nov 02 06:55:10.981544 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:10.981514 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b23fcf38-1c33-4668-890c-436e1b54b7a4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b23fcf38-1c33-4668-890c-436e1b54b7a4-maas-api-xks-ca\") pod \"b23fcf38-1c33-4668-890c-436e1b54b7a4\" (UID: \"b23fcf38-1c33-4668-890c-436e1b54b7a4\") " Nov 02 06:55:10.981687 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:10.981670 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b23fcf38-1c33-4668-890c-436e1b54b7a4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b23fcf38-1c33-4668-890c-436e1b54b7a4-maas-api-openshift-ca\") pod \"b23fcf38-1c33-4668-890c-436e1b54b7a4\" (UID: \"b23fcf38-1c33-4668-890c-436e1b54b7a4\") " Nov 02 06:55:10.982047 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:10.982012 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b23fcf38-1c33-4668-890c-436e1b54b7a4-maas-api-openshift-ca" pod "b23fcf38-1c33-4668-890c-436e1b54b7a4" (UID: "b23fcf38-1c33-4668-890c-436e1b54b7a4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:55:10.982378 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:10.982331 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b23fcf38-1c33-4668-890c-436e1b54b7a4-maas-api-openshift-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:55:10.983989 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:10.983963 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b23fcf38-1c33-4668-890c-436e1b54b7a4-maas-api-xks-ca" pod "b23fcf38-1c33-4668-890c-436e1b54b7a4" (UID: "b23fcf38-1c33-4668-890c-436e1b54b7a4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:55:10.984101 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:10.984000 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b23fcf38-1c33-4668-890c-436e1b54b7a4-kube-api-access-82zqj" pod "b23fcf38-1c33-4668-890c-436e1b54b7a4" (UID: "b23fcf38-1c33-4668-890c-436e1b54b7a4"). InnerVolumeSpecName "kube-api-access-82zqj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:55:11.083403 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:11.083368 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b23fcf38-1c33-4668-890c-436e1b54b7a4-maas-api-xks-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:55:11.083403 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:11.083395 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82zqj\" (UniqueName: \"kubernetes.io/projected/b23fcf38-1c33-4668-890c-436e1b54b7a4-kube-api-access-82zqj\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:55:11.602781 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:11.602740 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9a43fa3-7tt7w" Nov 02 06:55:11.602781 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:11.602762 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9a43fa3-7tt7w" event={"ID":"b23fcf38-1c33-4668-890c-436e1b54b7a4","Type":"ContainerDied","Data":"9534b546bca349e84a2e7e3ed2c5b7b151a0076dfc5ee005174237c384e67af9"} Nov 02 06:55:11.603052 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:11.602832 5916 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9534b546bca349e84a2e7e3ed2c5b7b151a0076dfc5ee005174237c384e67af9" Nov 02 06:55:11.604406 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:11.604378 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73e2cf7f-zjks2" Nov 02 06:55:11.604556 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:11.604429 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73e2cf7f-zjks2" event={"ID":"39513a10-9b43-4963-8b4b-265187974870","Type":"ContainerDied","Data":"33de7292f812a720e910b418170a255d2941a103685e277717d22ae1896c8d98"} Nov 02 06:55:11.604556 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:11.604478 5916 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="33de7292f812a720e910b418170a255d2941a103685e277717d22ae1896c8d98" Nov 02 06:55:11.627141 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:55:11.627116 5916 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39513a10_9b43_4963_8b4b_265187974870.slice\": RecentStats: unable to find data in memory cache]" Nov 02 06:55:11.789314 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:11.789271 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-680b7b-867b5cf97c-s84v9" podUID="144a4e28-24fb-467e-8e2a-2e26736c9589" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 06:55:11.789314 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:11.789299 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-680b7b-775c58cc67-6d6rn" podUID="90130906-592a-4243-9697-6daa79939293" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 06:55:14.987930 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:14.987616 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6ff85e/premium-w6-6ff85e-kserve-59848ff6d4-szbxx" podUID="eb386976-f898-4f1d-b27c-77bccc735a3b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:55:19.894283 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:55:19.894243 5916 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p98cq" containerName="extractor" Nov 02 06:55:21.789377 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:21.789335 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-680b7b-775c58cc67-6d6rn" podUID="90130906-592a-4243-9697-6daa79939293" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 06:55:21.789769 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:21.789392 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-680b7b-867b5cf97c-s84v9" podUID="144a4e28-24fb-467e-8e2a-2e26736c9589" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 06:55:23.507693 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:23.507338 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-680b7b-867b5cf97c-s84v9" Nov 02 06:55:23.584706 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:23.584683 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-680b7b-775c58cc67-6d6rn" Nov 02 06:55:23.608046 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:23.608025 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/144a4e28-24fb-467e-8e2a-2e26736c9589-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/144a4e28-24fb-467e-8e2a-2e26736c9589-plugins-config-volume\") pod \"144a4e28-24fb-467e-8e2a-2e26736c9589\" (UID: \"144a4e28-24fb-467e-8e2a-2e26736c9589\") " Nov 02 06:55:23.608165 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:23.608151 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/144a4e28-24fb-467e-8e2a-2e26736c9589-kube-api-access-gvq2d\" (UniqueName: \"kubernetes.io/projected/144a4e28-24fb-467e-8e2a-2e26736c9589-kube-api-access-gvq2d\") pod \"144a4e28-24fb-467e-8e2a-2e26736c9589\" (UID: \"144a4e28-24fb-467e-8e2a-2e26736c9589\") " Nov 02 06:55:23.608366 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:23.608345 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/144a4e28-24fb-467e-8e2a-2e26736c9589-plugins-config-volume" pod "144a4e28-24fb-467e-8e2a-2e26736c9589" (UID: "144a4e28-24fb-467e-8e2a-2e26736c9589"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:55:23.608621 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:23.608601 5916 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/144a4e28-24fb-467e-8e2a-2e26736c9589-plugins-config-volume\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:55:23.610189 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:23.610169 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/144a4e28-24fb-467e-8e2a-2e26736c9589-kube-api-access-gvq2d" pod "144a4e28-24fb-467e-8e2a-2e26736c9589" (UID: "144a4e28-24fb-467e-8e2a-2e26736c9589"). InnerVolumeSpecName "kube-api-access-gvq2d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:55:23.709779 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:23.709748 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90130906-592a-4243-9697-6daa79939293-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90130906-592a-4243-9697-6daa79939293-plugins-config-volume\") pod \"90130906-592a-4243-9697-6daa79939293\" (UID: \"90130906-592a-4243-9697-6daa79939293\") " Nov 02 06:55:23.709959 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:23.709795 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90130906-592a-4243-9697-6daa79939293-kube-api-access-rxd4m\" (UniqueName: \"kubernetes.io/projected/90130906-592a-4243-9697-6daa79939293-kube-api-access-rxd4m\") pod \"90130906-592a-4243-9697-6daa79939293\" (UID: \"90130906-592a-4243-9697-6daa79939293\") " Nov 02 06:55:23.710106 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:23.710090 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvq2d\" (UniqueName: \"kubernetes.io/projected/144a4e28-24fb-467e-8e2a-2e26736c9589-kube-api-access-gvq2d\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:55:23.710149 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:23.710126 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90130906-592a-4243-9697-6daa79939293-plugins-config-volume" pod "90130906-592a-4243-9697-6daa79939293" (UID: "90130906-592a-4243-9697-6daa79939293"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:55:23.711858 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:23.711837 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90130906-592a-4243-9697-6daa79939293-kube-api-access-rxd4m" pod "90130906-592a-4243-9697-6daa79939293" (UID: "90130906-592a-4243-9697-6daa79939293"). InnerVolumeSpecName "kube-api-access-rxd4m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:55:23.811379 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:23.811295 5916 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90130906-592a-4243-9697-6daa79939293-plugins-config-volume\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:55:23.811379 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:23.811329 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxd4m\" (UniqueName: \"kubernetes.io/projected/90130906-592a-4243-9697-6daa79939293-kube-api-access-rxd4m\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:55:23.827310 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:23.827279 5916 generic.go:356] "Generic (PLEG): container finished" podID="90130906-592a-4243-9697-6daa79939293" containerID="7dbfa1661aa93f68b1c6bd6df01d8439677c9f686228ff490143ba51750c882d" exitCode=0 Nov 02 06:55:23.827482 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:23.827336 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-680b7b-775c58cc67-6d6rn" Nov 02 06:55:23.827482 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:23.827363 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-680b7b-775c58cc67-6d6rn" event={"ID":"90130906-592a-4243-9697-6daa79939293","Type":"ContainerDied","Data":"7dbfa1661aa93f68b1c6bd6df01d8439677c9f686228ff490143ba51750c882d"} Nov 02 06:55:23.827482 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:23.827424 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-680b7b-775c58cc67-6d6rn" event={"ID":"90130906-592a-4243-9697-6daa79939293","Type":"ContainerDied","Data":"f2dacff3ffb0c1b7be64a47061bae6aa76cec87270047e1459d0c51527a53992"} Nov 02 06:55:23.827482 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:23.827455 5916 scope.go:122] "RemoveContainer" containerID="7dbfa1661aa93f68b1c6bd6df01d8439677c9f686228ff490143ba51750c882d" Nov 02 06:55:23.828789 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:23.828769 5916 generic.go:356] "Generic (PLEG): container finished" podID="144a4e28-24fb-467e-8e2a-2e26736c9589" containerID="7121fef955574f68cbc728df2965193b959aad9917544b2c19f134ba0cbd6ea3" exitCode=0 Nov 02 06:55:23.828889 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:23.828818 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-680b7b-867b5cf97c-s84v9" Nov 02 06:55:23.828889 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:23.828852 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-680b7b-867b5cf97c-s84v9" event={"ID":"144a4e28-24fb-467e-8e2a-2e26736c9589","Type":"ContainerDied","Data":"7121fef955574f68cbc728df2965193b959aad9917544b2c19f134ba0cbd6ea3"} Nov 02 06:55:23.829018 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:23.828915 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-680b7b-867b5cf97c-s84v9" event={"ID":"144a4e28-24fb-467e-8e2a-2e26736c9589","Type":"ContainerDied","Data":"57a6d8128a397763aedffd824fcc9b55dcf0089590725411792d143b24254ee6"} Nov 02 06:55:23.841189 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:23.841171 5916 scope.go:122] "RemoveContainer" containerID="7dbfa1661aa93f68b1c6bd6df01d8439677c9f686228ff490143ba51750c882d" Nov 02 06:55:23.841458 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:55:23.841441 5916 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7dbfa1661aa93f68b1c6bd6df01d8439677c9f686228ff490143ba51750c882d\": container with ID starting with 7dbfa1661aa93f68b1c6bd6df01d8439677c9f686228ff490143ba51750c882d not found: ID does not exist" containerID="7dbfa1661aa93f68b1c6bd6df01d8439677c9f686228ff490143ba51750c882d" Nov 02 06:55:23.841516 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:23.841465 5916 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7dbfa1661aa93f68b1c6bd6df01d8439677c9f686228ff490143ba51750c882d"} err="failed to get container status \"7dbfa1661aa93f68b1c6bd6df01d8439677c9f686228ff490143ba51750c882d\": rpc error: code = NotFound desc = could not find container \"7dbfa1661aa93f68b1c6bd6df01d8439677c9f686228ff490143ba51750c882d\": container with ID starting with 7dbfa1661aa93f68b1c6bd6df01d8439677c9f686228ff490143ba51750c882d not found: ID does not exist" Nov 02 06:55:23.841516 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:23.841490 5916 scope.go:122] "RemoveContainer" containerID="7121fef955574f68cbc728df2965193b959aad9917544b2c19f134ba0cbd6ea3" Nov 02 06:55:23.852161 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:23.852141 5916 scope.go:122] "RemoveContainer" containerID="7121fef955574f68cbc728df2965193b959aad9917544b2c19f134ba0cbd6ea3" Nov 02 06:55:23.852427 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:55:23.852408 5916 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7121fef955574f68cbc728df2965193b959aad9917544b2c19f134ba0cbd6ea3\": container with ID starting with 7121fef955574f68cbc728df2965193b959aad9917544b2c19f134ba0cbd6ea3 not found: ID does not exist" containerID="7121fef955574f68cbc728df2965193b959aad9917544b2c19f134ba0cbd6ea3" Nov 02 06:55:23.852476 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:23.852435 5916 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7121fef955574f68cbc728df2965193b959aad9917544b2c19f134ba0cbd6ea3"} err="failed to get container status \"7121fef955574f68cbc728df2965193b959aad9917544b2c19f134ba0cbd6ea3\": rpc error: code = NotFound desc = could not find container \"7121fef955574f68cbc728df2965193b959aad9917544b2c19f134ba0cbd6ea3\": container with ID starting with 7121fef955574f68cbc728df2965193b959aad9917544b2c19f134ba0cbd6ea3 not found: ID does not exist" Nov 02 06:55:23.994736 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:23.994699 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-680b7b-867b5cf97c-s84v9"] Nov 02 06:55:24.004893 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:23.999178 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-680b7b-867b5cf97c-s84v9"] Nov 02 06:55:24.106418 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:24.106328 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-680b7b-775c58cc67-6d6rn"] Nov 02 06:55:24.109939 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:24.109901 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-680b7b-775c58cc67-6d6rn"] Nov 02 06:55:25.844240 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:25.844205 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="144a4e28-24fb-467e-8e2a-2e26736c9589" path="/var/lib/kubelet/pods/144a4e28-24fb-467e-8e2a-2e26736c9589/volumes" Nov 02 06:55:25.844769 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:25.844753 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90130906-592a-4243-9697-6daa79939293" path="/var/lib/kubelet/pods/90130906-592a-4243-9697-6daa79939293/volumes" Nov 02 06:55:49.894272 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:55:49.894237 5916 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p98cq" containerName="extractor" Nov 02 06:55:55.176352 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:55.176319 5916 scope.go:122] "RemoveContainer" containerID="7000134fb2522a45ba6f403832aa45064ed396ffb88e88492e88148e5737d2d5" Nov 02 06:55:55.186282 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:55.186259 5916 scope.go:122] "RemoveContainer" containerID="f16ff7aeb9afbffd78e1c7a8441621d241a0489fd22c6b750664ff560215bfe9" Nov 02 06:55:55.195358 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:55:55.195338 5916 scope.go:122] "RemoveContainer" containerID="da58e7dc8cfbfb14d03adbd957ca48b64ee92c0b0f5c3c798cb736b73e866541" Nov 02 06:56:05.295592 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:56:05.295508 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-680b7b-r9nqd"] Nov 02 06:56:05.342636 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:56:05.342601 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-680b7b-r9nqd"] Nov 02 06:56:05.842859 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:56:05.842826 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33ac8085-f21e-49ee-89ab-c55e4c97dc64" path="/var/lib/kubelet/pods/33ac8085-f21e-49ee-89ab-c55e4c97dc64/volumes" Nov 02 06:56:19.893476 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:56:19.893439 5916 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p98cq" containerName="extractor" Nov 02 06:56:31.765156 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:56:31.765116 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f42795-92jpm"] Nov 02 06:56:31.765867 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:56:31.765847 5916 cpu_manager.go:414] "containerMap: removing container" podUID="90130906-592a-4243-9697-6daa79939293" containerName="payload-pre-processing" Nov 02 06:56:31.765949 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:56:31.765869 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90130906-592a-4243-9697-6daa79939293" containerName="payload-pre-processing" Nov 02 06:56:31.765949 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:56:31.765881 5916 cpu_manager.go:414] "containerMap: removing container" podUID="39513a10-9b43-4963-8b4b-265187974870" containerName="revoke-keys" Nov 02 06:56:31.765949 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:56:31.765889 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39513a10-9b43-4963-8b4b-265187974870" containerName="revoke-keys" Nov 02 06:56:31.765949 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:56:31.765919 5916 cpu_manager.go:414] "containerMap: removing container" podUID="b23fcf38-1c33-4668-890c-436e1b54b7a4" containerName="revoke-keys" Nov 02 06:56:31.765949 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:56:31.765927 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b23fcf38-1c33-4668-890c-436e1b54b7a4" containerName="revoke-keys" Nov 02 06:56:31.765949 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:56:31.765942 5916 cpu_manager.go:414] "containerMap: removing container" podUID="144a4e28-24fb-467e-8e2a-2e26736c9589" containerName="payload-processing" Nov 02 06:56:31.765949 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:56:31.765948 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="144a4e28-24fb-467e-8e2a-2e26736c9589" containerName="payload-processing" Nov 02 06:56:31.766167 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:56:31.766061 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="90130906-592a-4243-9697-6daa79939293" containerName="payload-pre-processing" Nov 02 06:56:31.766167 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:56:31.766074 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="39513a10-9b43-4963-8b4b-265187974870" containerName="revoke-keys" Nov 02 06:56:31.766167 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:56:31.766082 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="b23fcf38-1c33-4668-890c-436e1b54b7a4" containerName="revoke-keys" Nov 02 06:56:31.766167 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:56:31.766094 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="144a4e28-24fb-467e-8e2a-2e26736c9589" containerName="payload-processing" Nov 02 06:56:31.770032 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:56:31.770004 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f42795-92jpm" Nov 02 06:56:31.823094 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:56:31.823056 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 06:56:31.827435 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:56:31.827411 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f42795-92jpm"] Nov 02 06:56:31.872441 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:56:31.872413 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-68l4r\"" Nov 02 06:56:31.872648 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:56:31.872481 5916 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 06:56:31.872648 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:56:31.872550 5916 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 06:56:31.931207 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:56:31.931169 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43edf66d-033d-4274-9356-8cebe7f84b4c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f42795-92jpm\" (UID: \"43edf66d-033d-4274-9356-8cebe7f84b4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f42795-92jpm" Nov 02 06:56:31.931383 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:56:31.931363 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43edf66d-033d-4274-9356-8cebe7f84b4c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f42795-92jpm\" (UID: \"43edf66d-033d-4274-9356-8cebe7f84b4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f42795-92jpm" Nov 02 06:56:31.931452 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:56:31.931404 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skf2f\" (UniqueName: \"kubernetes.io/projected/43edf66d-033d-4274-9356-8cebe7f84b4c-kube-api-access-skf2f\") pod \"maas-api-revoke-keys-e2e-api-a-f42795-92jpm\" (UID: \"43edf66d-033d-4274-9356-8cebe7f84b4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f42795-92jpm" Nov 02 06:56:32.033140 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:56:32.033061 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43edf66d-033d-4274-9356-8cebe7f84b4c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f42795-92jpm\" (UID: \"43edf66d-033d-4274-9356-8cebe7f84b4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f42795-92jpm" Nov 02 06:56:32.033140 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:56:32.033117 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skf2f\" (UniqueName: \"kubernetes.io/projected/43edf66d-033d-4274-9356-8cebe7f84b4c-kube-api-access-skf2f\") pod \"maas-api-revoke-keys-e2e-api-a-f42795-92jpm\" (UID: \"43edf66d-033d-4274-9356-8cebe7f84b4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f42795-92jpm" Nov 02 06:56:32.033359 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:56:32.033179 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43edf66d-033d-4274-9356-8cebe7f84b4c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f42795-92jpm\" (UID: \"43edf66d-033d-4274-9356-8cebe7f84b4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f42795-92jpm" Nov 02 06:56:32.033974 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:56:32.033948 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43edf66d-033d-4274-9356-8cebe7f84b4c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f42795-92jpm\" (UID: \"43edf66d-033d-4274-9356-8cebe7f84b4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f42795-92jpm" Nov 02 06:56:32.035261 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:56:32.035240 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43edf66d-033d-4274-9356-8cebe7f84b4c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f42795-92jpm\" (UID: \"43edf66d-033d-4274-9356-8cebe7f84b4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f42795-92jpm" Nov 02 06:56:32.089937 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:56:32.089893 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-skf2f\" (UniqueName: \"kubernetes.io/projected/43edf66d-033d-4274-9356-8cebe7f84b4c-kube-api-access-skf2f\") pod \"maas-api-revoke-keys-e2e-api-a-f42795-92jpm\" (UID: \"43edf66d-033d-4274-9356-8cebe7f84b4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f42795-92jpm" Nov 02 06:56:32.384043 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:56:32.384017 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f42795-92jpm" Nov 02 06:56:32.762566 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:56:32.762543 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f42795-92jpm"] Nov 02 06:56:32.764896 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:56:32.764867 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43edf66d_033d_4274_9356_8cebe7f84b4c.slice/crio-6046e1ac303da0c835c8342830f824e968ab67354d73eca62cd7473558e625fd WatchSource:0}: Error finding container 6046e1ac303da0c835c8342830f824e968ab67354d73eca62cd7473558e625fd: Status 404 returned error can't find the container with id 6046e1ac303da0c835c8342830f824e968ab67354d73eca62cd7473558e625fd Nov 02 06:56:33.428700 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:56:33.428664 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f42795-92jpm" event={"ID":"43edf66d-033d-4274-9356-8cebe7f84b4c","Type":"ContainerStarted","Data":"8da2b6040ea1cb63e340397628762e5772f58e5906372dc38798364ae84b2470"} Nov 02 06:56:33.429132 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:56:33.428725 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f42795-92jpm" event={"ID":"43edf66d-033d-4274-9356-8cebe7f84b4c","Type":"ContainerStarted","Data":"6046e1ac303da0c835c8342830f824e968ab67354d73eca62cd7473558e625fd"} Nov 02 06:56:33.551351 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:56:33.551291 5916 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f42795-92jpm" podStartSLOduration=2.551275881 podStartE2EDuration="2.551275881s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:56:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:56:33.549177919 +0000 UTC m=+2381.820998314" watchObservedRunningTime="2026-11-02 06:56:33.551275881 +0000 UTC m=+2381.823096276" Nov 02 06:56:34.437661 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:56:34.437622 5916 generic.go:356] "Generic (PLEG): container finished" podID="43edf66d-033d-4274-9356-8cebe7f84b4c" containerID="8da2b6040ea1cb63e340397628762e5772f58e5906372dc38798364ae84b2470" exitCode=0 Nov 02 06:56:34.438070 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:56:34.437679 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f42795-92jpm" event={"ID":"43edf66d-033d-4274-9356-8cebe7f84b4c","Type":"ContainerDied","Data":"8da2b6040ea1cb63e340397628762e5772f58e5906372dc38798364ae84b2470"} Nov 02 06:56:35.607021 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:56:35.606995 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f42795-92jpm" Nov 02 06:56:35.785133 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:56:35.785039 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43edf66d-033d-4274-9356-8cebe7f84b4c-kube-api-access-skf2f\" (UniqueName: \"kubernetes.io/projected/43edf66d-033d-4274-9356-8cebe7f84b4c-kube-api-access-skf2f\") pod \"43edf66d-033d-4274-9356-8cebe7f84b4c\" (UID: \"43edf66d-033d-4274-9356-8cebe7f84b4c\") " Nov 02 06:56:35.785133 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:56:35.785085 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43edf66d-033d-4274-9356-8cebe7f84b4c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43edf66d-033d-4274-9356-8cebe7f84b4c-maas-api-xks-ca\") pod \"43edf66d-033d-4274-9356-8cebe7f84b4c\" (UID: \"43edf66d-033d-4274-9356-8cebe7f84b4c\") " Nov 02 06:56:35.785370 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:56:35.785233 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43edf66d-033d-4274-9356-8cebe7f84b4c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43edf66d-033d-4274-9356-8cebe7f84b4c-maas-api-openshift-ca\") pod \"43edf66d-033d-4274-9356-8cebe7f84b4c\" (UID: \"43edf66d-033d-4274-9356-8cebe7f84b4c\") " Nov 02 06:56:35.785571 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:56:35.785545 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43edf66d-033d-4274-9356-8cebe7f84b4c-maas-api-openshift-ca" pod "43edf66d-033d-4274-9356-8cebe7f84b4c" (UID: "43edf66d-033d-4274-9356-8cebe7f84b4c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:56:35.785818 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:56:35.785800 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43edf66d-033d-4274-9356-8cebe7f84b4c-maas-api-openshift-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:56:35.787236 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:56:35.787214 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43edf66d-033d-4274-9356-8cebe7f84b4c-maas-api-xks-ca" pod "43edf66d-033d-4274-9356-8cebe7f84b4c" (UID: "43edf66d-033d-4274-9356-8cebe7f84b4c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:56:35.787334 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:56:35.787309 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43edf66d-033d-4274-9356-8cebe7f84b4c-kube-api-access-skf2f" pod "43edf66d-033d-4274-9356-8cebe7f84b4c" (UID: "43edf66d-033d-4274-9356-8cebe7f84b4c"). InnerVolumeSpecName "kube-api-access-skf2f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:56:35.886487 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:56:35.886459 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-skf2f\" (UniqueName: \"kubernetes.io/projected/43edf66d-033d-4274-9356-8cebe7f84b4c-kube-api-access-skf2f\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:56:35.886487 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:56:35.886487 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43edf66d-033d-4274-9356-8cebe7f84b4c-maas-api-xks-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:56:36.453056 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:56:36.453028 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f42795-92jpm" Nov 02 06:56:36.453248 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:56:36.453067 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f42795-92jpm" event={"ID":"43edf66d-033d-4274-9356-8cebe7f84b4c","Type":"ContainerDied","Data":"6046e1ac303da0c835c8342830f824e968ab67354d73eca62cd7473558e625fd"} Nov 02 06:56:36.453248 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:56:36.453120 5916 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6046e1ac303da0c835c8342830f824e968ab67354d73eca62cd7473558e625fd" Nov 02 06:56:49.894731 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:56:49.894690 5916 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p98cq" containerName="extractor" Nov 02 06:56:53.211413 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:56:53.211383 5916 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-csfhg_17dee72b-2f47-4016-8ffe-3a89934e66ee/console-operator/2.log" Nov 02 06:56:53.233144 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:56:53.233114 5916 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-csfhg_17dee72b-2f47-4016-8ffe-3a89934e66ee/console-operator/2.log" Nov 02 06:56:55.258411 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:56:55.258375 5916 scope.go:122] "RemoveContainer" containerID="bf05711a7cfc850bdfaea5762e9789763e72bc5a0f6aac3302d539ee7f1c40d6" Nov 02 06:56:55.270000 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:56:55.269981 5916 scope.go:122] "RemoveContainer" containerID="a27965f3572e4513d105e04cd8f356867f4769ca651b831d1ac47067c26f1e48" Nov 02 06:57:19.893622 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:57:19.893584 5916 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p98cq" containerName="extractor" Nov 02 06:57:48.050508 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:57:48.050420 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27db47d9-kzr5h"] Nov 02 06:57:48.099851 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:57:48.099814 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1f85e4b-kfhtc"] Nov 02 06:57:48.102918 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:57:48.102889 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27db47d9-kzr5h"] Nov 02 06:57:48.105328 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:57:48.105306 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1f85e4b-kfhtc"] Nov 02 06:57:49.842596 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:57:49.842560 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04e1849e-57d9-4230-b35c-62a76cbd90bb" path="/var/lib/kubelet/pods/04e1849e-57d9-4230-b35c-62a76cbd90bb/volumes" Nov 02 06:57:49.843149 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:57:49.843043 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb70237e-4074-4947-a12f-8e45c9e5ecaf" path="/var/lib/kubelet/pods/bb70237e-4074-4947-a12f-8e45c9e5ecaf/volumes" Nov 02 06:57:49.894390 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:57:49.894359 5916 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p98cq" containerName="extractor" Nov 02 06:57:55.300243 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:57:55.300209 5916 scope.go:122] "RemoveContainer" containerID="e8c0446547f624181f5a2d523b52c1a4c4c924dd6eebec7c3d88eb2fdd3033e0" Nov 02 06:57:55.313457 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:57:55.313434 5916 scope.go:122] "RemoveContainer" containerID="09728591c0eedbce9a14c79e9217b32ab93838d246e60ee190740f2223c5f4de" Nov 02 06:58:01.793166 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:58:01.793127 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f42795-92jpm"] Nov 02 06:58:01.844502 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:58:01.844471 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f42795-92jpm"] Nov 02 06:58:01.890458 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:58:01.890420 5916 status_manager.go:1045] "Failed to get status for pod" err="pods \"maas-api-revoke-keys-e2e-api-a-f42795-92jpm\" is forbidden: User \"system:node:ip-10-0-130-93.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"odh-ai-gateway-infra\": no relationship found between node 'ip-10-0-130-93.us-west-2.compute.internal' and this object" podUID="43edf66d-033d-4274-9356-8cebe7f84b4c" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f42795-92jpm" Nov 02 06:58:03.843687 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:58:03.843656 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43edf66d-033d-4274-9356-8cebe7f84b4c" path="/var/lib/kubelet/pods/43edf66d-033d-4274-9356-8cebe7f84b4c/volumes" Nov 02 06:58:09.387310 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:58:09.387267 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e6fde6-zqqg9"] Nov 02 06:58:09.388109 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:58:09.388091 5916 cpu_manager.go:414] "containerMap: removing container" podUID="43edf66d-033d-4274-9356-8cebe7f84b4c" containerName="revoke-keys" Nov 02 06:58:09.388165 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:58:09.388111 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43edf66d-033d-4274-9356-8cebe7f84b4c" containerName="revoke-keys" Nov 02 06:58:09.388211 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:58:09.388206 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="43edf66d-033d-4274-9356-8cebe7f84b4c" containerName="revoke-keys" Nov 02 06:58:09.393275 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:58:09.393257 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e6fde6-zqqg9" Nov 02 06:58:09.446826 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:58:09.446794 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e6fde6-zqqg9"] Nov 02 06:58:09.492449 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:58:09.492414 5916 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 06:58:09.492603 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:58:09.492464 5916 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 06:58:09.494931 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:58:09.494898 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 06:58:09.495049 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:58:09.495001 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-68l4r\"" Nov 02 06:58:09.511467 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:58:09.511447 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c7b7cb7-e43b-4eef-bbe1-2fd396e6d6b3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-e6fde6-zqqg9\" (UID: \"4c7b7cb7-e43b-4eef-bbe1-2fd396e6d6b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e6fde6-zqqg9" Nov 02 06:58:09.511607 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:58:09.511589 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjbqp\" (UniqueName: \"kubernetes.io/projected/4c7b7cb7-e43b-4eef-bbe1-2fd396e6d6b3-kube-api-access-fjbqp\") pod \"maas-api-revoke-keys-e2e-api-b-e6fde6-zqqg9\" (UID: \"4c7b7cb7-e43b-4eef-bbe1-2fd396e6d6b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e6fde6-zqqg9" Nov 02 06:58:09.511657 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:58:09.511644 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c7b7cb7-e43b-4eef-bbe1-2fd396e6d6b3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-e6fde6-zqqg9\" (UID: \"4c7b7cb7-e43b-4eef-bbe1-2fd396e6d6b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e6fde6-zqqg9" Nov 02 06:58:09.612717 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:58:09.612682 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c7b7cb7-e43b-4eef-bbe1-2fd396e6d6b3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-e6fde6-zqqg9\" (UID: \"4c7b7cb7-e43b-4eef-bbe1-2fd396e6d6b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e6fde6-zqqg9" Nov 02 06:58:09.612877 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:58:09.612833 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjbqp\" (UniqueName: \"kubernetes.io/projected/4c7b7cb7-e43b-4eef-bbe1-2fd396e6d6b3-kube-api-access-fjbqp\") pod \"maas-api-revoke-keys-e2e-api-b-e6fde6-zqqg9\" (UID: \"4c7b7cb7-e43b-4eef-bbe1-2fd396e6d6b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e6fde6-zqqg9" Nov 02 06:58:09.612945 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:58:09.612881 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c7b7cb7-e43b-4eef-bbe1-2fd396e6d6b3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-e6fde6-zqqg9\" (UID: \"4c7b7cb7-e43b-4eef-bbe1-2fd396e6d6b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e6fde6-zqqg9" Nov 02 06:58:09.613577 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:58:09.613552 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c7b7cb7-e43b-4eef-bbe1-2fd396e6d6b3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-e6fde6-zqqg9\" (UID: \"4c7b7cb7-e43b-4eef-bbe1-2fd396e6d6b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e6fde6-zqqg9" Nov 02 06:58:09.615090 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:58:09.615069 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c7b7cb7-e43b-4eef-bbe1-2fd396e6d6b3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-e6fde6-zqqg9\" (UID: \"4c7b7cb7-e43b-4eef-bbe1-2fd396e6d6b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e6fde6-zqqg9" Nov 02 06:58:09.671244 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:58:09.671185 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjbqp\" (UniqueName: \"kubernetes.io/projected/4c7b7cb7-e43b-4eef-bbe1-2fd396e6d6b3-kube-api-access-fjbqp\") pod \"maas-api-revoke-keys-e2e-api-b-e6fde6-zqqg9\" (UID: \"4c7b7cb7-e43b-4eef-bbe1-2fd396e6d6b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e6fde6-zqqg9" Nov 02 06:58:09.707085 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:58:09.707060 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e6fde6-zqqg9" Nov 02 06:58:10.080155 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:58:10.080128 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e6fde6-zqqg9"] Nov 02 06:58:10.081400 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:58:10.081374 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c7b7cb7_e43b_4eef_bbe1_2fd396e6d6b3.slice/crio-d2c0e0732bb4bb304908e84fc9463b9261cd3b3d625e6c4c5cecf6df725d3531 WatchSource:0}: Error finding container d2c0e0732bb4bb304908e84fc9463b9261cd3b3d625e6c4c5cecf6df725d3531: Status 404 returned error can't find the container with id d2c0e0732bb4bb304908e84fc9463b9261cd3b3d625e6c4c5cecf6df725d3531 Nov 02 06:58:10.173192 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:58:10.173155 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e6fde6-zqqg9" event={"ID":"4c7b7cb7-e43b-4eef-bbe1-2fd396e6d6b3","Type":"ContainerStarted","Data":"d2c0e0732bb4bb304908e84fc9463b9261cd3b3d625e6c4c5cecf6df725d3531"} Nov 02 06:58:11.181055 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:58:11.180965 5916 generic.go:356] "Generic (PLEG): container finished" podID="4c7b7cb7-e43b-4eef-bbe1-2fd396e6d6b3" containerID="1ec051afde7eff79711cb9ac5990a09a0df215d1ac0145647336e8ccf7dd4d79" exitCode=0 Nov 02 06:58:11.181457 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:58:11.181049 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e6fde6-zqqg9" event={"ID":"4c7b7cb7-e43b-4eef-bbe1-2fd396e6d6b3","Type":"ContainerDied","Data":"1ec051afde7eff79711cb9ac5990a09a0df215d1ac0145647336e8ccf7dd4d79"} Nov 02 06:58:12.327731 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:58:12.327703 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e6fde6-zqqg9" Nov 02 06:58:12.444879 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:58:12.444786 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c7b7cb7-e43b-4eef-bbe1-2fd396e6d6b3-kube-api-access-fjbqp\" (UniqueName: \"kubernetes.io/projected/4c7b7cb7-e43b-4eef-bbe1-2fd396e6d6b3-kube-api-access-fjbqp\") pod \"4c7b7cb7-e43b-4eef-bbe1-2fd396e6d6b3\" (UID: \"4c7b7cb7-e43b-4eef-bbe1-2fd396e6d6b3\") " Nov 02 06:58:12.444879 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:58:12.444824 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c7b7cb7-e43b-4eef-bbe1-2fd396e6d6b3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c7b7cb7-e43b-4eef-bbe1-2fd396e6d6b3-maas-api-openshift-ca\") pod \"4c7b7cb7-e43b-4eef-bbe1-2fd396e6d6b3\" (UID: \"4c7b7cb7-e43b-4eef-bbe1-2fd396e6d6b3\") " Nov 02 06:58:12.444879 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:58:12.444880 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c7b7cb7-e43b-4eef-bbe1-2fd396e6d6b3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c7b7cb7-e43b-4eef-bbe1-2fd396e6d6b3-maas-api-xks-ca\") pod \"4c7b7cb7-e43b-4eef-bbe1-2fd396e6d6b3\" (UID: \"4c7b7cb7-e43b-4eef-bbe1-2fd396e6d6b3\") " Nov 02 06:58:12.445293 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:58:12.445262 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c7b7cb7-e43b-4eef-bbe1-2fd396e6d6b3-maas-api-openshift-ca" pod "4c7b7cb7-e43b-4eef-bbe1-2fd396e6d6b3" (UID: "4c7b7cb7-e43b-4eef-bbe1-2fd396e6d6b3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:58:12.447028 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:58:12.446998 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c7b7cb7-e43b-4eef-bbe1-2fd396e6d6b3-maas-api-xks-ca" pod "4c7b7cb7-e43b-4eef-bbe1-2fd396e6d6b3" (UID: "4c7b7cb7-e43b-4eef-bbe1-2fd396e6d6b3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:58:12.447158 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:58:12.447123 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c7b7cb7-e43b-4eef-bbe1-2fd396e6d6b3-kube-api-access-fjbqp" pod "4c7b7cb7-e43b-4eef-bbe1-2fd396e6d6b3" (UID: "4c7b7cb7-e43b-4eef-bbe1-2fd396e6d6b3"). InnerVolumeSpecName "kube-api-access-fjbqp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:58:12.546462 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:58:12.546430 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjbqp\" (UniqueName: \"kubernetes.io/projected/4c7b7cb7-e43b-4eef-bbe1-2fd396e6d6b3-kube-api-access-fjbqp\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:58:12.546462 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:58:12.546458 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c7b7cb7-e43b-4eef-bbe1-2fd396e6d6b3-maas-api-openshift-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:58:12.546644 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:58:12.546470 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c7b7cb7-e43b-4eef-bbe1-2fd396e6d6b3-maas-api-xks-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:58:13.197627 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:58:13.197595 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e6fde6-zqqg9" event={"ID":"4c7b7cb7-e43b-4eef-bbe1-2fd396e6d6b3","Type":"ContainerDied","Data":"d2c0e0732bb4bb304908e84fc9463b9261cd3b3d625e6c4c5cecf6df725d3531"} Nov 02 06:58:13.197627 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:58:13.197621 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e6fde6-zqqg9" Nov 02 06:58:13.197847 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:58:13.197640 5916 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d2c0e0732bb4bb304908e84fc9463b9261cd3b3d625e6c4c5cecf6df725d3531" Nov 02 06:58:19.893657 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:58:19.893616 5916 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p98cq" containerName="extractor" Nov 02 06:58:46.050922 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:58:46.050873 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13c46a92-dfjmg"] Nov 02 06:58:46.102304 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:58:46.102268 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13c46a92-dfjmg"] Nov 02 06:58:47.051831 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:58:47.051792 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1eab520f-9qkbc"] Nov 02 06:58:47.101004 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:58:47.100970 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1eab520f-9qkbc"] Nov 02 06:58:47.842865 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:58:47.842833 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f0419bf-81cc-4bda-a4c4-6da385635e12" path="/var/lib/kubelet/pods/5f0419bf-81cc-4bda-a4c4-6da385635e12/volumes" Nov 02 06:58:47.843323 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:58:47.843308 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f33e7b06-ffea-4c8b-8905-bd9a469213c4" path="/var/lib/kubelet/pods/f33e7b06-ffea-4c8b-8905-bd9a469213c4/volumes" Nov 02 06:58:49.894420 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:58:49.894377 5916 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p98cq" containerName="extractor" Nov 02 06:58:55.344699 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:58:55.344662 5916 scope.go:122] "RemoveContainer" containerID="c654f11aa05caef996e1c78ebd47786b3c89b9e2f7f45b392f59fbfd34a7cc47" Nov 02 06:58:55.357859 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:58:55.357832 5916 scope.go:122] "RemoveContainer" containerID="92ef91855217a5c0da50a5c0e25a551020a3d90dc6f4c3450366294c1c4ad792" Nov 02 06:58:55.370599 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:58:55.370570 5916 scope.go:122] "RemoveContainer" containerID="be7ee466641f270568ad09ba9a0eb619474b27969d1443a16e6aa6111322a0bb" Nov 02 06:58:55.379946 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:58:55.379922 5916 scope.go:122] "RemoveContainer" containerID="281e3aba2f06a1ee9a069e25cfd4da779287572122a8dc092de91392b0cb0d97" Nov 02 06:58:55.389545 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:58:55.389516 5916 scope.go:122] "RemoveContainer" containerID="dd704b5564325d7bbb957e64b2f7203ecc1e8e1cdd71162f43dee67e47c39570" Nov 02 06:59:02.050732 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:02.050635 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-adcb334f-cn4xj"] Nov 02 06:59:02.100405 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:02.100361 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-adcb334f-cn4xj"] Nov 02 06:59:02.103498 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:02.103470 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89d4bdbf-pw8vc"] Nov 02 06:59:02.106182 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:02.106155 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89d4bdbf-pw8vc"] Nov 02 06:59:03.605971 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:03.605936 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a1de0b6-rhk79"] Nov 02 06:59:03.606561 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:03.606540 5916 cpu_manager.go:414] "containerMap: removing container" podUID="4c7b7cb7-e43b-4eef-bbe1-2fd396e6d6b3" containerName="revoke-keys" Nov 02 06:59:03.606663 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:03.606564 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c7b7cb7-e43b-4eef-bbe1-2fd396e6d6b3" containerName="revoke-keys" Nov 02 06:59:03.606716 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:03.606670 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c7b7cb7-e43b-4eef-bbe1-2fd396e6d6b3" containerName="revoke-keys" Nov 02 06:59:03.611875 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:03.611851 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a1de0b6-rhk79" Nov 02 06:59:03.623486 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:03.623459 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e55fd1b3-1171-4b06-a303-34952600343f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6a1de0b6-rhk79\" (UID: \"e55fd1b3-1171-4b06-a303-34952600343f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a1de0b6-rhk79" Nov 02 06:59:03.623632 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:03.623590 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e55fd1b3-1171-4b06-a303-34952600343f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6a1de0b6-rhk79\" (UID: \"e55fd1b3-1171-4b06-a303-34952600343f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a1de0b6-rhk79" Nov 02 06:59:03.623680 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:03.623635 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mvwn\" (UniqueName: \"kubernetes.io/projected/e55fd1b3-1171-4b06-a303-34952600343f-kube-api-access-4mvwn\") pod \"maas-api-revoke-sub-6a1de0b6-rhk79\" (UID: \"e55fd1b3-1171-4b06-a303-34952600343f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a1de0b6-rhk79" Nov 02 06:59:03.666386 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:03.666346 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a1de0b6-rhk79"] Nov 02 06:59:03.711699 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:03.711656 5916 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 06:59:03.713880 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:03.713854 5916 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 06:59:03.713989 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:03.713852 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 06:59:03.713989 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:03.713964 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-68l4r\"" Nov 02 06:59:03.724733 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:03.724701 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e55fd1b3-1171-4b06-a303-34952600343f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6a1de0b6-rhk79\" (UID: \"e55fd1b3-1171-4b06-a303-34952600343f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a1de0b6-rhk79" Nov 02 06:59:03.724872 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:03.724802 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e55fd1b3-1171-4b06-a303-34952600343f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6a1de0b6-rhk79\" (UID: \"e55fd1b3-1171-4b06-a303-34952600343f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a1de0b6-rhk79" Nov 02 06:59:03.724872 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:03.724842 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mvwn\" (UniqueName: \"kubernetes.io/projected/e55fd1b3-1171-4b06-a303-34952600343f-kube-api-access-4mvwn\") pod \"maas-api-revoke-sub-6a1de0b6-rhk79\" (UID: \"e55fd1b3-1171-4b06-a303-34952600343f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a1de0b6-rhk79" Nov 02 06:59:03.725531 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:03.725510 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e55fd1b3-1171-4b06-a303-34952600343f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6a1de0b6-rhk79\" (UID: \"e55fd1b3-1171-4b06-a303-34952600343f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a1de0b6-rhk79" Nov 02 06:59:03.727417 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:03.727397 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e55fd1b3-1171-4b06-a303-34952600343f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6a1de0b6-rhk79\" (UID: \"e55fd1b3-1171-4b06-a303-34952600343f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a1de0b6-rhk79" Nov 02 06:59:03.782023 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:03.781982 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mvwn\" (UniqueName: \"kubernetes.io/projected/e55fd1b3-1171-4b06-a303-34952600343f-kube-api-access-4mvwn\") pod \"maas-api-revoke-sub-6a1de0b6-rhk79\" (UID: \"e55fd1b3-1171-4b06-a303-34952600343f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a1de0b6-rhk79" Nov 02 06:59:03.843309 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:03.843273 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48757757-4e3d-404d-8af4-93ed81f60565" path="/var/lib/kubelet/pods/48757757-4e3d-404d-8af4-93ed81f60565/volumes" Nov 02 06:59:03.843748 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:03.843725 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cab8160e-1169-4676-b764-92a3435dcc99" path="/var/lib/kubelet/pods/cab8160e-1169-4676-b764-92a3435dcc99/volumes" Nov 02 06:59:03.927348 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:03.927312 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a1de0b6-rhk79" Nov 02 06:59:04.313806 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:04.313777 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a1de0b6-rhk79"] Nov 02 06:59:04.315211 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:59:04.315186 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode55fd1b3_1171_4b06_a303_34952600343f.slice/crio-8b9ef319cb219a7e5805a71fdab45e96ad49a5fd8764f57902bb9bc6300745f3 WatchSource:0}: Error finding container 8b9ef319cb219a7e5805a71fdab45e96ad49a5fd8764f57902bb9bc6300745f3: Status 404 returned error can't find the container with id 8b9ef319cb219a7e5805a71fdab45e96ad49a5fd8764f57902bb9bc6300745f3 Nov 02 06:59:04.574020 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:04.573938 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a1de0b6-rhk79" event={"ID":"e55fd1b3-1171-4b06-a303-34952600343f","Type":"ContainerStarted","Data":"8b9ef319cb219a7e5805a71fdab45e96ad49a5fd8764f57902bb9bc6300745f3"} Nov 02 06:59:05.582037 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:05.581953 5916 generic.go:356] "Generic (PLEG): container finished" podID="e55fd1b3-1171-4b06-a303-34952600343f" containerID="84324208460eb65a7eeaf5cd8d2fcbbc25b80698997b4b845d7514b3c5828820" exitCode=0 Nov 02 06:59:05.582410 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:05.582037 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a1de0b6-rhk79" event={"ID":"e55fd1b3-1171-4b06-a303-34952600343f","Type":"ContainerDied","Data":"84324208460eb65a7eeaf5cd8d2fcbbc25b80698997b4b845d7514b3c5828820"} Nov 02 06:59:06.747452 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:06.747428 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a1de0b6-rhk79" Nov 02 06:59:06.859967 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:06.859876 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e55fd1b3-1171-4b06-a303-34952600343f-kube-api-access-4mvwn\" (UniqueName: \"kubernetes.io/projected/e55fd1b3-1171-4b06-a303-34952600343f-kube-api-access-4mvwn\") pod \"e55fd1b3-1171-4b06-a303-34952600343f\" (UID: \"e55fd1b3-1171-4b06-a303-34952600343f\") " Nov 02 06:59:06.859967 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:06.859940 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e55fd1b3-1171-4b06-a303-34952600343f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e55fd1b3-1171-4b06-a303-34952600343f-maas-api-xks-ca\") pod \"e55fd1b3-1171-4b06-a303-34952600343f\" (UID: \"e55fd1b3-1171-4b06-a303-34952600343f\") " Nov 02 06:59:06.860229 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:06.860095 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e55fd1b3-1171-4b06-a303-34952600343f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e55fd1b3-1171-4b06-a303-34952600343f-maas-api-openshift-ca\") pod \"e55fd1b3-1171-4b06-a303-34952600343f\" (UID: \"e55fd1b3-1171-4b06-a303-34952600343f\") " Nov 02 06:59:06.860568 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:06.860519 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e55fd1b3-1171-4b06-a303-34952600343f-maas-api-openshift-ca" pod "e55fd1b3-1171-4b06-a303-34952600343f" (UID: "e55fd1b3-1171-4b06-a303-34952600343f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:59:06.861165 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:06.861141 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e55fd1b3-1171-4b06-a303-34952600343f-maas-api-openshift-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:59:06.866275 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:06.866243 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e55fd1b3-1171-4b06-a303-34952600343f-kube-api-access-4mvwn" pod "e55fd1b3-1171-4b06-a303-34952600343f" (UID: "e55fd1b3-1171-4b06-a303-34952600343f"). InnerVolumeSpecName "kube-api-access-4mvwn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:59:06.869518 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:06.869478 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e55fd1b3-1171-4b06-a303-34952600343f-maas-api-xks-ca" pod "e55fd1b3-1171-4b06-a303-34952600343f" (UID: "e55fd1b3-1171-4b06-a303-34952600343f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:59:06.962975 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:06.962942 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mvwn\" (UniqueName: \"kubernetes.io/projected/e55fd1b3-1171-4b06-a303-34952600343f-kube-api-access-4mvwn\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:59:06.962975 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:06.962970 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e55fd1b3-1171-4b06-a303-34952600343f-maas-api-xks-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:59:07.600982 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:07.600935 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a1de0b6-rhk79" event={"ID":"e55fd1b3-1171-4b06-a303-34952600343f","Type":"ContainerDied","Data":"8b9ef319cb219a7e5805a71fdab45e96ad49a5fd8764f57902bb9bc6300745f3"} Nov 02 06:59:07.600982 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:07.600971 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a1de0b6-rhk79" Nov 02 06:59:07.601208 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:07.601018 5916 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8b9ef319cb219a7e5805a71fdab45e96ad49a5fd8764f57902bb9bc6300745f3" Nov 02 06:59:13.730460 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:13.730428 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb05257d-gwf8w"] Nov 02 06:59:13.730965 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:13.730950 5916 cpu_manager.go:414] "containerMap: removing container" podUID="e55fd1b3-1171-4b06-a303-34952600343f" containerName="revoke-keys" Nov 02 06:59:13.731015 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:13.730965 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e55fd1b3-1171-4b06-a303-34952600343f" containerName="revoke-keys" Nov 02 06:59:13.731060 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:13.731050 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="e55fd1b3-1171-4b06-a303-34952600343f" containerName="revoke-keys" Nov 02 06:59:13.734700 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:13.734679 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb05257d-gwf8w" Nov 02 06:59:13.790505 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:13.790472 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb05257d-gwf8w"] Nov 02 06:59:13.836783 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:13.836745 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 06:59:13.836783 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:13.836750 5916 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 06:59:13.837051 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:13.836792 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-68l4r\"" Nov 02 06:59:13.837051 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:13.836878 5916 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 06:59:13.843115 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:13.843088 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca23ad83-835d-4120-94f8-2227afef03c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bb05257d-gwf8w\" (UID: \"ca23ad83-835d-4120-94f8-2227afef03c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb05257d-gwf8w" Nov 02 06:59:13.843260 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:13.843147 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca23ad83-835d-4120-94f8-2227afef03c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bb05257d-gwf8w\" (UID: \"ca23ad83-835d-4120-94f8-2227afef03c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb05257d-gwf8w" Nov 02 06:59:13.843396 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:13.843370 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvfjm\" (UniqueName: \"kubernetes.io/projected/ca23ad83-835d-4120-94f8-2227afef03c5-kube-api-access-jvfjm\") pod \"maas-api-revoke-sub-bb05257d-gwf8w\" (UID: \"ca23ad83-835d-4120-94f8-2227afef03c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb05257d-gwf8w" Nov 02 06:59:13.944498 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:13.944466 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca23ad83-835d-4120-94f8-2227afef03c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bb05257d-gwf8w\" (UID: \"ca23ad83-835d-4120-94f8-2227afef03c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb05257d-gwf8w" Nov 02 06:59:13.944685 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:13.944645 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvfjm\" (UniqueName: \"kubernetes.io/projected/ca23ad83-835d-4120-94f8-2227afef03c5-kube-api-access-jvfjm\") pod \"maas-api-revoke-sub-bb05257d-gwf8w\" (UID: \"ca23ad83-835d-4120-94f8-2227afef03c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb05257d-gwf8w" Nov 02 06:59:13.944756 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:13.944697 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca23ad83-835d-4120-94f8-2227afef03c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bb05257d-gwf8w\" (UID: \"ca23ad83-835d-4120-94f8-2227afef03c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb05257d-gwf8w" Nov 02 06:59:13.945270 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:13.945247 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca23ad83-835d-4120-94f8-2227afef03c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bb05257d-gwf8w\" (UID: \"ca23ad83-835d-4120-94f8-2227afef03c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb05257d-gwf8w" Nov 02 06:59:13.947411 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:13.947388 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca23ad83-835d-4120-94f8-2227afef03c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bb05257d-gwf8w\" (UID: \"ca23ad83-835d-4120-94f8-2227afef03c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb05257d-gwf8w" Nov 02 06:59:14.003137 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:14.003067 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvfjm\" (UniqueName: \"kubernetes.io/projected/ca23ad83-835d-4120-94f8-2227afef03c5-kube-api-access-jvfjm\") pod \"maas-api-revoke-sub-bb05257d-gwf8w\" (UID: \"ca23ad83-835d-4120-94f8-2227afef03c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb05257d-gwf8w" Nov 02 06:59:14.048054 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:14.048020 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb05257d-gwf8w" Nov 02 06:59:14.423540 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:14.423514 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb05257d-gwf8w"] Nov 02 06:59:14.424420 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:59:14.424396 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca23ad83_835d_4120_94f8_2227afef03c5.slice/crio-16bc15fcb0fcf0c2bf4af848b42c179e0a501e1536ecf1e05ce1cda9f8965a6d WatchSource:0}: Error finding container 16bc15fcb0fcf0c2bf4af848b42c179e0a501e1536ecf1e05ce1cda9f8965a6d: Status 404 returned error can't find the container with id 16bc15fcb0fcf0c2bf4af848b42c179e0a501e1536ecf1e05ce1cda9f8965a6d Nov 02 06:59:14.655298 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:14.655268 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb05257d-gwf8w" event={"ID":"ca23ad83-835d-4120-94f8-2227afef03c5","Type":"ContainerStarted","Data":"16bc15fcb0fcf0c2bf4af848b42c179e0a501e1536ecf1e05ce1cda9f8965a6d"} Nov 02 06:59:15.667147 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:15.667107 5916 generic.go:356] "Generic (PLEG): container finished" podID="ca23ad83-835d-4120-94f8-2227afef03c5" containerID="98e01d30edee97c7d8c7855e2be111d680d5be6e9b45dcedb3a2c145d776cd25" exitCode=0 Nov 02 06:59:15.667662 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:15.667213 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb05257d-gwf8w" event={"ID":"ca23ad83-835d-4120-94f8-2227afef03c5","Type":"ContainerDied","Data":"98e01d30edee97c7d8c7855e2be111d680d5be6e9b45dcedb3a2c145d776cd25"} Nov 02 06:59:16.414668 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:16.414628 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f3f1755-4qrwx"] Nov 02 06:59:16.418593 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:16.418576 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f3f1755-4qrwx" Nov 02 06:59:16.475061 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:16.475022 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f3f1755-4qrwx"] Nov 02 06:59:16.576638 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:16.576600 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e5389b4-2147-4e2f-b5e7-f16a760ca9a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9f3f1755-4qrwx\" (UID: \"2e5389b4-2147-4e2f-b5e7-f16a760ca9a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f3f1755-4qrwx" Nov 02 06:59:16.576809 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:16.576762 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfr6b\" (UniqueName: \"kubernetes.io/projected/2e5389b4-2147-4e2f-b5e7-f16a760ca9a2-kube-api-access-rfr6b\") pod \"maas-api-revoke-sub-9f3f1755-4qrwx\" (UID: \"2e5389b4-2147-4e2f-b5e7-f16a760ca9a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f3f1755-4qrwx" Nov 02 06:59:16.576891 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:16.576825 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e5389b4-2147-4e2f-b5e7-f16a760ca9a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9f3f1755-4qrwx\" (UID: \"2e5389b4-2147-4e2f-b5e7-f16a760ca9a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f3f1755-4qrwx" Nov 02 06:59:16.677861 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:16.677832 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e5389b4-2147-4e2f-b5e7-f16a760ca9a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9f3f1755-4qrwx\" (UID: \"2e5389b4-2147-4e2f-b5e7-f16a760ca9a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f3f1755-4qrwx" Nov 02 06:59:16.678204 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:16.677924 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfr6b\" (UniqueName: \"kubernetes.io/projected/2e5389b4-2147-4e2f-b5e7-f16a760ca9a2-kube-api-access-rfr6b\") pod \"maas-api-revoke-sub-9f3f1755-4qrwx\" (UID: \"2e5389b4-2147-4e2f-b5e7-f16a760ca9a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f3f1755-4qrwx" Nov 02 06:59:16.678204 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:16.677965 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e5389b4-2147-4e2f-b5e7-f16a760ca9a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9f3f1755-4qrwx\" (UID: \"2e5389b4-2147-4e2f-b5e7-f16a760ca9a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f3f1755-4qrwx" Nov 02 06:59:16.678572 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:16.678547 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e5389b4-2147-4e2f-b5e7-f16a760ca9a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9f3f1755-4qrwx\" (UID: \"2e5389b4-2147-4e2f-b5e7-f16a760ca9a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f3f1755-4qrwx" Nov 02 06:59:16.680290 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:16.680269 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e5389b4-2147-4e2f-b5e7-f16a760ca9a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9f3f1755-4qrwx\" (UID: \"2e5389b4-2147-4e2f-b5e7-f16a760ca9a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f3f1755-4qrwx" Nov 02 06:59:16.754988 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:16.754919 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfr6b\" (UniqueName: \"kubernetes.io/projected/2e5389b4-2147-4e2f-b5e7-f16a760ca9a2-kube-api-access-rfr6b\") pod \"maas-api-revoke-sub-9f3f1755-4qrwx\" (UID: \"2e5389b4-2147-4e2f-b5e7-f16a760ca9a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f3f1755-4qrwx" Nov 02 06:59:16.823080 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:16.823053 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb05257d-gwf8w" Nov 02 06:59:16.980915 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:16.980823 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ca23ad83-835d-4120-94f8-2227afef03c5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca23ad83-835d-4120-94f8-2227afef03c5-maas-api-xks-ca\") pod \"ca23ad83-835d-4120-94f8-2227afef03c5\" (UID: \"ca23ad83-835d-4120-94f8-2227afef03c5\") " Nov 02 06:59:16.981090 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:16.980935 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca23ad83-835d-4120-94f8-2227afef03c5-kube-api-access-jvfjm\" (UniqueName: \"kubernetes.io/projected/ca23ad83-835d-4120-94f8-2227afef03c5-kube-api-access-jvfjm\") pod \"ca23ad83-835d-4120-94f8-2227afef03c5\" (UID: \"ca23ad83-835d-4120-94f8-2227afef03c5\") " Nov 02 06:59:16.981090 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:16.980974 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca23ad83-835d-4120-94f8-2227afef03c5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca23ad83-835d-4120-94f8-2227afef03c5-maas-api-openshift-ca\") pod \"ca23ad83-835d-4120-94f8-2227afef03c5\" (UID: \"ca23ad83-835d-4120-94f8-2227afef03c5\") " Nov 02 06:59:16.981474 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:16.981440 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca23ad83-835d-4120-94f8-2227afef03c5-maas-api-openshift-ca" pod "ca23ad83-835d-4120-94f8-2227afef03c5" (UID: "ca23ad83-835d-4120-94f8-2227afef03c5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:59:16.983124 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:16.983091 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca23ad83-835d-4120-94f8-2227afef03c5-maas-api-xks-ca" pod "ca23ad83-835d-4120-94f8-2227afef03c5" (UID: "ca23ad83-835d-4120-94f8-2227afef03c5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:59:16.983490 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:16.983469 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca23ad83-835d-4120-94f8-2227afef03c5-kube-api-access-jvfjm" pod "ca23ad83-835d-4120-94f8-2227afef03c5" (UID: "ca23ad83-835d-4120-94f8-2227afef03c5"). InnerVolumeSpecName "kube-api-access-jvfjm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:59:17.040257 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:17.040235 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f3f1755-4qrwx" Nov 02 06:59:17.082737 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:17.082707 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvfjm\" (UniqueName: \"kubernetes.io/projected/ca23ad83-835d-4120-94f8-2227afef03c5-kube-api-access-jvfjm\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:59:17.082737 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:17.082736 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca23ad83-835d-4120-94f8-2227afef03c5-maas-api-openshift-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:59:17.082901 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:17.082753 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca23ad83-835d-4120-94f8-2227afef03c5-maas-api-xks-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:59:17.415066 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:17.415037 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f3f1755-4qrwx"] Nov 02 06:59:17.416397 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:59:17.416371 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e5389b4_2147_4e2f_b5e7_f16a760ca9a2.slice/crio-809a985d06ebf0f54ea98b521123bd31178438ac7edf3a62728888536d83f847 WatchSource:0}: Error finding container 809a985d06ebf0f54ea98b521123bd31178438ac7edf3a62728888536d83f847: Status 404 returned error can't find the container with id 809a985d06ebf0f54ea98b521123bd31178438ac7edf3a62728888536d83f847 Nov 02 06:59:17.684676 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:17.684568 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb05257d-gwf8w" Nov 02 06:59:17.685196 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:17.684577 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb05257d-gwf8w" event={"ID":"ca23ad83-835d-4120-94f8-2227afef03c5","Type":"ContainerDied","Data":"16bc15fcb0fcf0c2bf4af848b42c179e0a501e1536ecf1e05ce1cda9f8965a6d"} Nov 02 06:59:17.685196 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:17.685016 5916 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="16bc15fcb0fcf0c2bf4af848b42c179e0a501e1536ecf1e05ce1cda9f8965a6d" Nov 02 06:59:17.686072 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:17.686045 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f3f1755-4qrwx" event={"ID":"2e5389b4-2147-4e2f-b5e7-f16a760ca9a2","Type":"ContainerStarted","Data":"809a985d06ebf0f54ea98b521123bd31178438ac7edf3a62728888536d83f847"} Nov 02 06:59:18.693720 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:18.693689 5916 generic.go:356] "Generic (PLEG): container finished" podID="2e5389b4-2147-4e2f-b5e7-f16a760ca9a2" containerID="24b51bc8e2a75779525c9f6e9aa06a8b20a5625ec055cc4c90b59226670bb3b8" exitCode=0 Nov 02 06:59:18.694155 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:18.693777 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f3f1755-4qrwx" event={"ID":"2e5389b4-2147-4e2f-b5e7-f16a760ca9a2","Type":"ContainerDied","Data":"24b51bc8e2a75779525c9f6e9aa06a8b20a5625ec055cc4c90b59226670bb3b8"} Nov 02 06:59:19.856839 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:19.856818 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f3f1755-4qrwx" Nov 02 06:59:19.892409 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:59:19.892378 5916 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p98cq" containerName="extractor" Nov 02 06:59:20.014567 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:20.014486 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e5389b4-2147-4e2f-b5e7-f16a760ca9a2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e5389b4-2147-4e2f-b5e7-f16a760ca9a2-maas-api-xks-ca\") pod \"2e5389b4-2147-4e2f-b5e7-f16a760ca9a2\" (UID: \"2e5389b4-2147-4e2f-b5e7-f16a760ca9a2\") " Nov 02 06:59:20.014781 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:20.014654 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e5389b4-2147-4e2f-b5e7-f16a760ca9a2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e5389b4-2147-4e2f-b5e7-f16a760ca9a2-maas-api-openshift-ca\") pod \"2e5389b4-2147-4e2f-b5e7-f16a760ca9a2\" (UID: \"2e5389b4-2147-4e2f-b5e7-f16a760ca9a2\") " Nov 02 06:59:20.014781 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:20.014720 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e5389b4-2147-4e2f-b5e7-f16a760ca9a2-kube-api-access-rfr6b\" (UniqueName: \"kubernetes.io/projected/2e5389b4-2147-4e2f-b5e7-f16a760ca9a2-kube-api-access-rfr6b\") pod \"2e5389b4-2147-4e2f-b5e7-f16a760ca9a2\" (UID: \"2e5389b4-2147-4e2f-b5e7-f16a760ca9a2\") " Nov 02 06:59:20.015105 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:20.015072 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e5389b4-2147-4e2f-b5e7-f16a760ca9a2-maas-api-openshift-ca" pod "2e5389b4-2147-4e2f-b5e7-f16a760ca9a2" (UID: "2e5389b4-2147-4e2f-b5e7-f16a760ca9a2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:59:20.016603 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:20.016580 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e5389b4-2147-4e2f-b5e7-f16a760ca9a2-maas-api-xks-ca" pod "2e5389b4-2147-4e2f-b5e7-f16a760ca9a2" (UID: "2e5389b4-2147-4e2f-b5e7-f16a760ca9a2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:59:20.016741 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:20.016719 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e5389b4-2147-4e2f-b5e7-f16a760ca9a2-kube-api-access-rfr6b" pod "2e5389b4-2147-4e2f-b5e7-f16a760ca9a2" (UID: "2e5389b4-2147-4e2f-b5e7-f16a760ca9a2"). InnerVolumeSpecName "kube-api-access-rfr6b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:59:20.116380 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:20.116348 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rfr6b\" (UniqueName: \"kubernetes.io/projected/2e5389b4-2147-4e2f-b5e7-f16a760ca9a2-kube-api-access-rfr6b\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:59:20.116380 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:20.116376 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e5389b4-2147-4e2f-b5e7-f16a760ca9a2-maas-api-xks-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:59:20.116380 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:20.116385 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e5389b4-2147-4e2f-b5e7-f16a760ca9a2-maas-api-openshift-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:59:20.709711 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:20.709683 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f3f1755-4qrwx" Nov 02 06:59:20.709890 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:20.709682 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f3f1755-4qrwx" event={"ID":"2e5389b4-2147-4e2f-b5e7-f16a760ca9a2","Type":"ContainerDied","Data":"809a985d06ebf0f54ea98b521123bd31178438ac7edf3a62728888536d83f847"} Nov 02 06:59:20.709890 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:20.709806 5916 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="809a985d06ebf0f54ea98b521123bd31178438ac7edf3a62728888536d83f847" Nov 02 06:59:32.483044 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:32.483008 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44df87b4-sv978"] Nov 02 06:59:32.483547 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:32.483533 5916 cpu_manager.go:414] "containerMap: removing container" podUID="2e5389b4-2147-4e2f-b5e7-f16a760ca9a2" containerName="revoke-keys" Nov 02 06:59:32.483597 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:32.483548 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e5389b4-2147-4e2f-b5e7-f16a760ca9a2" containerName="revoke-keys" Nov 02 06:59:32.483597 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:32.483568 5916 cpu_manager.go:414] "containerMap: removing container" podUID="ca23ad83-835d-4120-94f8-2227afef03c5" containerName="revoke-keys" Nov 02 06:59:32.483597 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:32.483573 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca23ad83-835d-4120-94f8-2227afef03c5" containerName="revoke-keys" Nov 02 06:59:32.483690 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:32.483661 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e5389b4-2147-4e2f-b5e7-f16a760ca9a2" containerName="revoke-keys" Nov 02 06:59:32.483690 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:32.483671 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca23ad83-835d-4120-94f8-2227afef03c5" containerName="revoke-keys" Nov 02 06:59:32.487141 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:32.487123 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44df87b4-sv978" Nov 02 06:59:32.572337 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:32.572302 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dv6r\" (UniqueName: \"kubernetes.io/projected/8519ee5f-3fde-4c74-afdf-f30eca48ad21-kube-api-access-9dv6r\") pod \"maas-api-revoke-sub-44df87b4-sv978\" (UID: \"8519ee5f-3fde-4c74-afdf-f30eca48ad21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44df87b4-sv978" Nov 02 06:59:32.572523 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:32.572373 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8519ee5f-3fde-4c74-afdf-f30eca48ad21-maas-api-xks-ca\") pod \"maas-api-revoke-sub-44df87b4-sv978\" (UID: \"8519ee5f-3fde-4c74-afdf-f30eca48ad21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44df87b4-sv978" Nov 02 06:59:32.572523 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:32.572498 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8519ee5f-3fde-4c74-afdf-f30eca48ad21-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-44df87b4-sv978\" (UID: \"8519ee5f-3fde-4c74-afdf-f30eca48ad21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44df87b4-sv978" Nov 02 06:59:32.590158 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:32.590130 5916 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 06:59:32.590320 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:32.590223 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-68l4r\"" Nov 02 06:59:32.590320 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:32.590238 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 06:59:32.590320 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:32.590257 5916 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 06:59:32.593467 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:32.593445 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44df87b4-sv978"] Nov 02 06:59:32.673685 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:32.673660 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8519ee5f-3fde-4c74-afdf-f30eca48ad21-maas-api-xks-ca\") pod \"maas-api-revoke-sub-44df87b4-sv978\" (UID: \"8519ee5f-3fde-4c74-afdf-f30eca48ad21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44df87b4-sv978" Nov 02 06:59:32.673822 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:32.673703 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8519ee5f-3fde-4c74-afdf-f30eca48ad21-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-44df87b4-sv978\" (UID: \"8519ee5f-3fde-4c74-afdf-f30eca48ad21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44df87b4-sv978" Nov 02 06:59:32.673822 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:32.673808 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dv6r\" (UniqueName: \"kubernetes.io/projected/8519ee5f-3fde-4c74-afdf-f30eca48ad21-kube-api-access-9dv6r\") pod \"maas-api-revoke-sub-44df87b4-sv978\" (UID: \"8519ee5f-3fde-4c74-afdf-f30eca48ad21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44df87b4-sv978" Nov 02 06:59:32.674458 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:32.674442 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8519ee5f-3fde-4c74-afdf-f30eca48ad21-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-44df87b4-sv978\" (UID: \"8519ee5f-3fde-4c74-afdf-f30eca48ad21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44df87b4-sv978" Nov 02 06:59:32.676081 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:32.676063 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8519ee5f-3fde-4c74-afdf-f30eca48ad21-maas-api-xks-ca\") pod \"maas-api-revoke-sub-44df87b4-sv978\" (UID: \"8519ee5f-3fde-4c74-afdf-f30eca48ad21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44df87b4-sv978" Nov 02 06:59:32.731511 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:32.731485 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dv6r\" (UniqueName: \"kubernetes.io/projected/8519ee5f-3fde-4c74-afdf-f30eca48ad21-kube-api-access-9dv6r\") pod \"maas-api-revoke-sub-44df87b4-sv978\" (UID: \"8519ee5f-3fde-4c74-afdf-f30eca48ad21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44df87b4-sv978" Nov 02 06:59:32.800590 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:32.800520 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44df87b4-sv978" Nov 02 06:59:33.194141 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:33.194105 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44df87b4-sv978"] Nov 02 06:59:33.195188 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:59:33.195159 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8519ee5f_3fde_4c74_afdf_f30eca48ad21.slice/crio-cbe30350047c73eb1cfd51b233fe459917799386219dc9b39698ae5e1061986a WatchSource:0}: Error finding container cbe30350047c73eb1cfd51b233fe459917799386219dc9b39698ae5e1061986a: Status 404 returned error can't find the container with id cbe30350047c73eb1cfd51b233fe459917799386219dc9b39698ae5e1061986a Nov 02 06:59:33.807704 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:33.807620 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44df87b4-sv978" event={"ID":"8519ee5f-3fde-4c74-afdf-f30eca48ad21","Type":"ContainerStarted","Data":"36dba79bb7b8f68b18bfbe61433ac299162d04f9d54a7d42609084c756c589a4"} Nov 02 06:59:33.807704 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:33.807686 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44df87b4-sv978" event={"ID":"8519ee5f-3fde-4c74-afdf-f30eca48ad21","Type":"ContainerStarted","Data":"cbe30350047c73eb1cfd51b233fe459917799386219dc9b39698ae5e1061986a"} Nov 02 06:59:33.990927 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:33.973494 5916 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44df87b4-sv978" podStartSLOduration=1.973473196 podStartE2EDuration="1.973473196s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:59:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:59:33.920623444 +0000 UTC m=+2562.192443839" watchObservedRunningTime="2026-11-02 06:59:33.973473196 +0000 UTC m=+2562.245293590" Nov 02 06:59:34.816404 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:34.816363 5916 generic.go:356] "Generic (PLEG): container finished" podID="8519ee5f-3fde-4c74-afdf-f30eca48ad21" containerID="36dba79bb7b8f68b18bfbe61433ac299162d04f9d54a7d42609084c756c589a4" exitCode=0 Nov 02 06:59:34.816857 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:34.816454 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44df87b4-sv978" event={"ID":"8519ee5f-3fde-4c74-afdf-f30eca48ad21","Type":"ContainerDied","Data":"36dba79bb7b8f68b18bfbe61433ac299162d04f9d54a7d42609084c756c589a4"} Nov 02 06:59:35.962272 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:35.962246 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44df87b4-sv978" Nov 02 06:59:36.042721 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:36.042688 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8519ee5f-3fde-4c74-afdf-f30eca48ad21-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8519ee5f-3fde-4c74-afdf-f30eca48ad21-maas-api-openshift-ca\") pod \"8519ee5f-3fde-4c74-afdf-f30eca48ad21\" (UID: \"8519ee5f-3fde-4c74-afdf-f30eca48ad21\") " Nov 02 06:59:36.042945 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:36.042729 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8519ee5f-3fde-4c74-afdf-f30eca48ad21-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8519ee5f-3fde-4c74-afdf-f30eca48ad21-maas-api-xks-ca\") pod \"8519ee5f-3fde-4c74-afdf-f30eca48ad21\" (UID: \"8519ee5f-3fde-4c74-afdf-f30eca48ad21\") " Nov 02 06:59:36.042945 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:36.042935 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8519ee5f-3fde-4c74-afdf-f30eca48ad21-kube-api-access-9dv6r\" (UniqueName: \"kubernetes.io/projected/8519ee5f-3fde-4c74-afdf-f30eca48ad21-kube-api-access-9dv6r\") pod \"8519ee5f-3fde-4c74-afdf-f30eca48ad21\" (UID: \"8519ee5f-3fde-4c74-afdf-f30eca48ad21\") " Nov 02 06:59:36.043085 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:36.043053 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8519ee5f-3fde-4c74-afdf-f30eca48ad21-maas-api-openshift-ca" pod "8519ee5f-3fde-4c74-afdf-f30eca48ad21" (UID: "8519ee5f-3fde-4c74-afdf-f30eca48ad21"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:59:36.043337 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:36.043319 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8519ee5f-3fde-4c74-afdf-f30eca48ad21-maas-api-openshift-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:59:36.044872 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:36.044845 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8519ee5f-3fde-4c74-afdf-f30eca48ad21-maas-api-xks-ca" pod "8519ee5f-3fde-4c74-afdf-f30eca48ad21" (UID: "8519ee5f-3fde-4c74-afdf-f30eca48ad21"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:59:36.044872 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:36.044855 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8519ee5f-3fde-4c74-afdf-f30eca48ad21-kube-api-access-9dv6r" pod "8519ee5f-3fde-4c74-afdf-f30eca48ad21" (UID: "8519ee5f-3fde-4c74-afdf-f30eca48ad21"). InnerVolumeSpecName "kube-api-access-9dv6r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:59:36.144651 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:36.144618 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8519ee5f-3fde-4c74-afdf-f30eca48ad21-maas-api-xks-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:59:36.144651 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:36.144650 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9dv6r\" (UniqueName: \"kubernetes.io/projected/8519ee5f-3fde-4c74-afdf-f30eca48ad21-kube-api-access-9dv6r\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:59:36.833441 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:36.833411 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44df87b4-sv978" Nov 02 06:59:36.833620 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:36.833439 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44df87b4-sv978" event={"ID":"8519ee5f-3fde-4c74-afdf-f30eca48ad21","Type":"ContainerDied","Data":"cbe30350047c73eb1cfd51b233fe459917799386219dc9b39698ae5e1061986a"} Nov 02 06:59:36.833620 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:36.833504 5916 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cbe30350047c73eb1cfd51b233fe459917799386219dc9b39698ae5e1061986a" Nov 02 06:59:40.053316 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:40.053279 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7318b500-w4ldh"] Nov 02 06:59:40.103272 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:40.103237 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7318b500-w4ldh"] Nov 02 06:59:41.842960 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:41.842927 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94db3b14-74f5-4309-bcc8-c0413b2beec2" path="/var/lib/kubelet/pods/94db3b14-74f5-4309-bcc8-c0413b2beec2/volumes" Nov 02 06:59:49.590657 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:49.590613 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e6fde6-zqqg9"] Nov 02 06:59:49.639973 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:49.639938 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e6fde6-zqqg9"] Nov 02 06:59:49.844887 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:49.844791 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c7b7cb7-e43b-4eef-bbe1-2fd396e6d6b3" path="/var/lib/kubelet/pods/4c7b7cb7-e43b-4eef-bbe1-2fd396e6d6b3/volumes" Nov 02 06:59:49.894239 ip-10-0-130-93 kubenswrapper[5916]: E1102 06:59:49.894199 5916 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p98cq" containerName="extractor" Nov 02 06:59:55.438963 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:55.438919 5916 scope.go:122] "RemoveContainer" containerID="c80128b3c89d29226f7b7aa8bbf582be9fb3a9ae71def60ea64a0178c4e9580a" Nov 02 06:59:55.458099 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:55.458070 5916 scope.go:122] "RemoveContainer" containerID="6531cf3632be20069dc9a5e418d81f175ba6fd354a196ea10e3802fd6d2fe2c1" Nov 02 06:59:55.467393 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:55.467368 5916 scope.go:122] "RemoveContainer" containerID="0ee1e996bb8bdc5cd5e9c5d8802822da295e1b63d86654aa38a87b9aa46016db" Nov 02 06:59:57.068832 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:57.068794 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-581608-mlz9r"] Nov 02 06:59:57.069550 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:57.069529 5916 cpu_manager.go:414] "containerMap: removing container" podUID="8519ee5f-3fde-4c74-afdf-f30eca48ad21" containerName="revoke-keys" Nov 02 06:59:57.069627 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:57.069552 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8519ee5f-3fde-4c74-afdf-f30eca48ad21" containerName="revoke-keys" Nov 02 06:59:57.069751 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:57.069738 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="8519ee5f-3fde-4c74-afdf-f30eca48ad21" containerName="revoke-keys" Nov 02 06:59:57.075394 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:57.075374 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-581608-mlz9r" Nov 02 06:59:57.129026 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:57.128993 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-581608-mlz9r"] Nov 02 06:59:57.173583 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:57.173538 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-68l4r\"" Nov 02 06:59:57.173583 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:57.173539 5916 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 06:59:57.173583 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:57.173539 5916 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 06:59:57.177156 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:57.177129 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 06:59:57.205114 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:57.205077 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rr9jd\" (UniqueName: \"kubernetes.io/projected/1837c8a1-277f-4319-8e92-6e9e27e698c5-kube-api-access-rr9jd\") pod \"maas-api-revoke-keys-e2e-worker-w2-581608-mlz9r\" (UID: \"1837c8a1-277f-4319-8e92-6e9e27e698c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-581608-mlz9r" Nov 02 06:59:57.205285 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:57.205213 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1837c8a1-277f-4319-8e92-6e9e27e698c5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-581608-mlz9r\" (UID: \"1837c8a1-277f-4319-8e92-6e9e27e698c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-581608-mlz9r" Nov 02 06:59:57.205393 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:57.205369 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1837c8a1-277f-4319-8e92-6e9e27e698c5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-581608-mlz9r\" (UID: \"1837c8a1-277f-4319-8e92-6e9e27e698c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-581608-mlz9r" Nov 02 06:59:57.307064 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:57.307027 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1837c8a1-277f-4319-8e92-6e9e27e698c5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-581608-mlz9r\" (UID: \"1837c8a1-277f-4319-8e92-6e9e27e698c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-581608-mlz9r" Nov 02 06:59:57.307236 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:57.307126 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rr9jd\" (UniqueName: \"kubernetes.io/projected/1837c8a1-277f-4319-8e92-6e9e27e698c5-kube-api-access-rr9jd\") pod \"maas-api-revoke-keys-e2e-worker-w2-581608-mlz9r\" (UID: \"1837c8a1-277f-4319-8e92-6e9e27e698c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-581608-mlz9r" Nov 02 06:59:57.307236 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:57.307164 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1837c8a1-277f-4319-8e92-6e9e27e698c5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-581608-mlz9r\" (UID: \"1837c8a1-277f-4319-8e92-6e9e27e698c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-581608-mlz9r" Nov 02 06:59:57.307864 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:57.307839 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1837c8a1-277f-4319-8e92-6e9e27e698c5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-581608-mlz9r\" (UID: \"1837c8a1-277f-4319-8e92-6e9e27e698c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-581608-mlz9r" Nov 02 06:59:57.309588 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:57.309560 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1837c8a1-277f-4319-8e92-6e9e27e698c5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-581608-mlz9r\" (UID: \"1837c8a1-277f-4319-8e92-6e9e27e698c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-581608-mlz9r" Nov 02 06:59:57.363829 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:57.363755 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rr9jd\" (UniqueName: \"kubernetes.io/projected/1837c8a1-277f-4319-8e92-6e9e27e698c5-kube-api-access-rr9jd\") pod \"maas-api-revoke-keys-e2e-worker-w2-581608-mlz9r\" (UID: \"1837c8a1-277f-4319-8e92-6e9e27e698c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-581608-mlz9r" Nov 02 06:59:57.389637 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:57.389599 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-581608-mlz9r" Nov 02 06:59:57.430235 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:57.430205 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8bc131-nhr4c"] Nov 02 06:59:57.437067 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:57.437046 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8bc131-nhr4c" Nov 02 06:59:57.510208 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:57.510120 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c031ed74-a8ca-4080-8216-9e5c55d2fc6e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-8bc131-nhr4c\" (UID: \"c031ed74-a8ca-4080-8216-9e5c55d2fc6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8bc131-nhr4c" Nov 02 06:59:57.510208 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:57.510182 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c031ed74-a8ca-4080-8216-9e5c55d2fc6e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-8bc131-nhr4c\" (UID: \"c031ed74-a8ca-4080-8216-9e5c55d2fc6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8bc131-nhr4c" Nov 02 06:59:57.510448 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:57.510265 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvxw2\" (UniqueName: \"kubernetes.io/projected/c031ed74-a8ca-4080-8216-9e5c55d2fc6e-kube-api-access-pvxw2\") pod \"maas-api-revoke-keys-e2e-inf-a-8bc131-nhr4c\" (UID: \"c031ed74-a8ca-4080-8216-9e5c55d2fc6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8bc131-nhr4c" Nov 02 06:59:57.511263 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:57.511237 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8bc131-nhr4c"] Nov 02 06:59:57.636085 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:57.632133 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c031ed74-a8ca-4080-8216-9e5c55d2fc6e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-8bc131-nhr4c\" (UID: \"c031ed74-a8ca-4080-8216-9e5c55d2fc6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8bc131-nhr4c" Nov 02 06:59:57.636085 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:57.632216 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c031ed74-a8ca-4080-8216-9e5c55d2fc6e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-8bc131-nhr4c\" (UID: \"c031ed74-a8ca-4080-8216-9e5c55d2fc6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8bc131-nhr4c" Nov 02 06:59:57.636085 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:57.632258 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvxw2\" (UniqueName: \"kubernetes.io/projected/c031ed74-a8ca-4080-8216-9e5c55d2fc6e-kube-api-access-pvxw2\") pod \"maas-api-revoke-keys-e2e-inf-a-8bc131-nhr4c\" (UID: \"c031ed74-a8ca-4080-8216-9e5c55d2fc6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8bc131-nhr4c" Nov 02 06:59:57.636085 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:57.634225 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c031ed74-a8ca-4080-8216-9e5c55d2fc6e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-8bc131-nhr4c\" (UID: \"c031ed74-a8ca-4080-8216-9e5c55d2fc6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8bc131-nhr4c" Nov 02 06:59:57.640031 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:57.638818 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c031ed74-a8ca-4080-8216-9e5c55d2fc6e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-8bc131-nhr4c\" (UID: \"c031ed74-a8ca-4080-8216-9e5c55d2fc6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8bc131-nhr4c" Nov 02 06:59:57.689834 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:57.689796 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvxw2\" (UniqueName: \"kubernetes.io/projected/c031ed74-a8ca-4080-8216-9e5c55d2fc6e-kube-api-access-pvxw2\") pod \"maas-api-revoke-keys-e2e-inf-a-8bc131-nhr4c\" (UID: \"c031ed74-a8ca-4080-8216-9e5c55d2fc6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8bc131-nhr4c" Nov 02 06:59:57.769147 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:57.769110 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8bc131-nhr4c" Nov 02 06:59:57.787255 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:59:57.787220 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1837c8a1_277f_4319_8e92_6e9e27e698c5.slice/crio-5e65fa1e4291d28143aa87333bcf24951b793204d264b32613bf8cdc2d56fa8d WatchSource:0}: Error finding container 5e65fa1e4291d28143aa87333bcf24951b793204d264b32613bf8cdc2d56fa8d: Status 404 returned error can't find the container with id 5e65fa1e4291d28143aa87333bcf24951b793204d264b32613bf8cdc2d56fa8d Nov 02 06:59:57.788042 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:57.788016 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-581608-mlz9r"] Nov 02 06:59:57.835572 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:57.835542 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a2ba11-x74pv"] Nov 02 06:59:57.859815 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:57.852591 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a2ba11-x74pv" Nov 02 06:59:57.891547 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:57.891455 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a2ba11-x74pv"] Nov 02 06:59:57.948603 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:57.948560 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e4e8418-4884-4d61-a0c3-54dcd871255e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a2ba11-x74pv\" (UID: \"1e4e8418-4884-4d61-a0c3-54dcd871255e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a2ba11-x74pv" Nov 02 06:59:57.948799 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:57.948712 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnnch\" (UniqueName: \"kubernetes.io/projected/1e4e8418-4884-4d61-a0c3-54dcd871255e-kube-api-access-dnnch\") pod \"maas-api-revoke-keys-e2e-inf-a-a2ba11-x74pv\" (UID: \"1e4e8418-4884-4d61-a0c3-54dcd871255e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a2ba11-x74pv" Nov 02 06:59:57.948885 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:57.948814 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e4e8418-4884-4d61-a0c3-54dcd871255e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a2ba11-x74pv\" (UID: \"1e4e8418-4884-4d61-a0c3-54dcd871255e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a2ba11-x74pv" Nov 02 06:59:58.003630 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:58.003593 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-581608-mlz9r" event={"ID":"1837c8a1-277f-4319-8e92-6e9e27e698c5","Type":"ContainerStarted","Data":"5e65fa1e4291d28143aa87333bcf24951b793204d264b32613bf8cdc2d56fa8d"} Nov 02 06:59:58.051644 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:58.051599 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e4e8418-4884-4d61-a0c3-54dcd871255e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a2ba11-x74pv\" (UID: \"1e4e8418-4884-4d61-a0c3-54dcd871255e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a2ba11-x74pv" Nov 02 06:59:58.052742 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:58.052716 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e4e8418-4884-4d61-a0c3-54dcd871255e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a2ba11-x74pv\" (UID: \"1e4e8418-4884-4d61-a0c3-54dcd871255e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a2ba11-x74pv" Nov 02 06:59:58.053452 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:58.053431 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnnch\" (UniqueName: \"kubernetes.io/projected/1e4e8418-4884-4d61-a0c3-54dcd871255e-kube-api-access-dnnch\") pod \"maas-api-revoke-keys-e2e-inf-a-a2ba11-x74pv\" (UID: \"1e4e8418-4884-4d61-a0c3-54dcd871255e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a2ba11-x74pv" Nov 02 06:59:58.053951 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:58.052615 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e4e8418-4884-4d61-a0c3-54dcd871255e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a2ba11-x74pv\" (UID: \"1e4e8418-4884-4d61-a0c3-54dcd871255e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a2ba11-x74pv" Nov 02 06:59:58.081922 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:58.074367 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e4e8418-4884-4d61-a0c3-54dcd871255e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a2ba11-x74pv\" (UID: \"1e4e8418-4884-4d61-a0c3-54dcd871255e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a2ba11-x74pv" Nov 02 06:59:58.121927 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:58.117717 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnnch\" (UniqueName: \"kubernetes.io/projected/1e4e8418-4884-4d61-a0c3-54dcd871255e-kube-api-access-dnnch\") pod \"maas-api-revoke-keys-e2e-inf-a-a2ba11-x74pv\" (UID: \"1e4e8418-4884-4d61-a0c3-54dcd871255e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a2ba11-x74pv" Nov 02 06:59:58.171153 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:58.169968 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a2ba11-x74pv" Nov 02 06:59:58.171153 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:58.170291 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8bc131-nhr4c"] Nov 02 06:59:58.184314 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:59:58.183006 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc031ed74_a8ca_4080_8216_9e5c55d2fc6e.slice/crio-8a74948ec4f97ae00e129116c2c2988c72d4c280a07a829d2c3a8a1d65c24fe2 WatchSource:0}: Error finding container 8a74948ec4f97ae00e129116c2c2988c72d4c280a07a829d2c3a8a1d65c24fe2: Status 404 returned error can't find the container with id 8a74948ec4f97ae00e129116c2c2988c72d4c280a07a829d2c3a8a1d65c24fe2 Nov 02 06:59:58.692545 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:58.692518 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a2ba11-x74pv"] Nov 02 06:59:58.694420 ip-10-0-130-93 kubenswrapper[5916]: W1102 06:59:58.694391 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e4e8418_4884_4d61_a0c3_54dcd871255e.slice/crio-c515159b27179733a29a7742e363f7834444a208480b88928d594c1ba5727d6f WatchSource:0}: Error finding container c515159b27179733a29a7742e363f7834444a208480b88928d594c1ba5727d6f: Status 404 returned error can't find the container with id c515159b27179733a29a7742e363f7834444a208480b88928d594c1ba5727d6f Nov 02 06:59:59.024486 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:59.024448 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8bc131-nhr4c" event={"ID":"c031ed74-a8ca-4080-8216-9e5c55d2fc6e","Type":"ContainerStarted","Data":"d12942b7208c3798c4d501220a72d41ae837aa329a81419557c944c20b983077"} Nov 02 06:59:59.024705 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:59.024506 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8bc131-nhr4c" event={"ID":"c031ed74-a8ca-4080-8216-9e5c55d2fc6e","Type":"ContainerStarted","Data":"8a74948ec4f97ae00e129116c2c2988c72d4c280a07a829d2c3a8a1d65c24fe2"} Nov 02 06:59:59.027278 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:59.027246 5916 generic.go:356] "Generic (PLEG): container finished" podID="1837c8a1-277f-4319-8e92-6e9e27e698c5" containerID="9a12ada894602c49ebede56bf7d086db1f63b4ba855ca830e48b4cb587d3ba97" exitCode=0 Nov 02 06:59:59.027456 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:59.027401 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-581608-mlz9r" event={"ID":"1837c8a1-277f-4319-8e92-6e9e27e698c5","Type":"ContainerDied","Data":"9a12ada894602c49ebede56bf7d086db1f63b4ba855ca830e48b4cb587d3ba97"} Nov 02 06:59:59.029078 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:59.029044 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a2ba11-x74pv" event={"ID":"1e4e8418-4884-4d61-a0c3-54dcd871255e","Type":"ContainerStarted","Data":"c515159b27179733a29a7742e363f7834444a208480b88928d594c1ba5727d6f"} Nov 02 06:59:59.141005 ip-10-0-130-93 kubenswrapper[5916]: I1102 06:59:59.140950 5916 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8bc131-nhr4c" podStartSLOduration=2.140929968 podStartE2EDuration="2.140929968s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:59:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:59:59.137800421 +0000 UTC m=+2587.409620819" watchObservedRunningTime="2026-11-02 06:59:59.140929968 +0000 UTC m=+2587.412750364" Nov 02 07:00:00.038079 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:00.038002 5916 generic.go:356] "Generic (PLEG): container finished" podID="c031ed74-a8ca-4080-8216-9e5c55d2fc6e" containerID="d12942b7208c3798c4d501220a72d41ae837aa329a81419557c944c20b983077" exitCode=0 Nov 02 07:00:00.038301 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:00.038093 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8bc131-nhr4c" event={"ID":"c031ed74-a8ca-4080-8216-9e5c55d2fc6e","Type":"ContainerDied","Data":"d12942b7208c3798c4d501220a72d41ae837aa329a81419557c944c20b983077"} Nov 02 07:00:00.039793 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:00.039769 5916 generic.go:356] "Generic (PLEG): container finished" podID="1e4e8418-4884-4d61-a0c3-54dcd871255e" containerID="a654f013e0dc5dcf471c4e6daf93aa7848ed84524e93c3fc5fbba0625a979cb6" exitCode=0 Nov 02 07:00:00.039923 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:00.039858 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a2ba11-x74pv" event={"ID":"1e4e8418-4884-4d61-a0c3-54dcd871255e","Type":"ContainerDied","Data":"a654f013e0dc5dcf471c4e6daf93aa7848ed84524e93c3fc5fbba0625a979cb6"} Nov 02 07:00:00.166937 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:00.166894 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2e9f81-29893380-zjnwh"] Nov 02 07:00:00.171135 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:00.171114 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2e9f81-29893380-zjnwh" Nov 02 07:00:00.171891 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:00.171865 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893380-kk78n"] Nov 02 07:00:00.176482 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:00.176463 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893380-kk78n" Nov 02 07:00:00.189860 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:00.189842 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-581608-mlz9r" Nov 02 07:00:00.205574 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:00.205550 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cdfce9-29893380-gwwf5"] Nov 02 07:00:00.206131 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:00.206111 5916 cpu_manager.go:414] "containerMap: removing container" podUID="1837c8a1-277f-4319-8e92-6e9e27e698c5" containerName="revoke-keys" Nov 02 07:00:00.206231 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:00.206132 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1837c8a1-277f-4319-8e92-6e9e27e698c5" containerName="revoke-keys" Nov 02 07:00:00.206339 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:00.206325 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="1837c8a1-277f-4319-8e92-6e9e27e698c5" containerName="revoke-keys" Nov 02 07:00:00.209575 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:00.209556 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a2ba11-29893380-f9dll"] Nov 02 07:00:00.209703 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:00.209681 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cdfce9-29893380-gwwf5" Nov 02 07:00:00.213325 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:00.213307 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a2ba11-29893380-f9dll" Nov 02 07:00:00.232630 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:00.232610 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1837c8a1-277f-4319-8e92-6e9e27e698c5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1837c8a1-277f-4319-8e92-6e9e27e698c5-maas-api-xks-ca\") pod \"1837c8a1-277f-4319-8e92-6e9e27e698c5\" (UID: \"1837c8a1-277f-4319-8e92-6e9e27e698c5\") " Nov 02 07:00:00.232721 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:00.232641 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1837c8a1-277f-4319-8e92-6e9e27e698c5-kube-api-access-rr9jd\" (UniqueName: \"kubernetes.io/projected/1837c8a1-277f-4319-8e92-6e9e27e698c5-kube-api-access-rr9jd\") pod \"1837c8a1-277f-4319-8e92-6e9e27e698c5\" (UID: \"1837c8a1-277f-4319-8e92-6e9e27e698c5\") " Nov 02 07:00:00.232763 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:00.232750 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1837c8a1-277f-4319-8e92-6e9e27e698c5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1837c8a1-277f-4319-8e92-6e9e27e698c5-maas-api-openshift-ca\") pod \"1837c8a1-277f-4319-8e92-6e9e27e698c5\" (UID: \"1837c8a1-277f-4319-8e92-6e9e27e698c5\") " Nov 02 07:00:00.233001 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:00.232978 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kpz9\" (UniqueName: \"kubernetes.io/projected/b6ee4d66-7acd-4f3f-ad2e-4650a1019c1c-kube-api-access-7kpz9\") pod \"maas-api-key-cleanup-e2e-worker-w1-cdfce9-29893380-gwwf5\" (UID: \"b6ee4d66-7acd-4f3f-ad2e-4650a1019c1c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cdfce9-29893380-gwwf5" Nov 02 07:00:00.233116 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:00.233085 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce7008b7-0096-4d6e-863e-60dce78fffe2-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893380-kk78n\" (UID: \"ce7008b7-0096-4d6e-863e-60dce78fffe2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893380-kk78n" Nov 02 07:00:00.233116 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:00.233093 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1837c8a1-277f-4319-8e92-6e9e27e698c5-maas-api-openshift-ca" pod "1837c8a1-277f-4319-8e92-6e9e27e698c5" (UID: "1837c8a1-277f-4319-8e92-6e9e27e698c5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:00:00.233251 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:00.233156 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6ee4d66-7acd-4f3f-ad2e-4650a1019c1c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-cdfce9-29893380-gwwf5\" (UID: \"b6ee4d66-7acd-4f3f-ad2e-4650a1019c1c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cdfce9-29893380-gwwf5" Nov 02 07:00:00.233251 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:00.233183 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9f76fd8-1db3-4fe8-8a4e-117813676c2d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-a2ba11-29893380-f9dll\" (UID: \"f9f76fd8-1db3-4fe8-8a4e-117813676c2d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a2ba11-29893380-f9dll" Nov 02 07:00:00.233340 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:00.233267 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsqz2\" (UniqueName: \"kubernetes.io/projected/f9f76fd8-1db3-4fe8-8a4e-117813676c2d-kube-api-access-tsqz2\") pod \"maas-api-key-cleanup-e2e-inf-a-a2ba11-29893380-f9dll\" (UID: \"f9f76fd8-1db3-4fe8-8a4e-117813676c2d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a2ba11-29893380-f9dll" Nov 02 07:00:00.233385 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:00.233344 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghnhx\" (UniqueName: \"kubernetes.io/projected/9546555f-4f4c-4ca1-80e8-3817da23ed9d-kube-api-access-ghnhx\") pod \"maas-api-key-cleanup-e2e-inf-b-2e9f81-29893380-zjnwh\" (UID: \"9546555f-4f4c-4ca1-80e8-3817da23ed9d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2e9f81-29893380-zjnwh" Nov 02 07:00:00.233440 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:00.233421 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6qnd\" (UniqueName: \"kubernetes.io/projected/ce7008b7-0096-4d6e-863e-60dce78fffe2-kube-api-access-k6qnd\") pod \"maas-api-key-cleanup-29893380-kk78n\" (UID: \"ce7008b7-0096-4d6e-863e-60dce78fffe2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893380-kk78n" Nov 02 07:00:00.233587 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:00.233569 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9546555f-4f4c-4ca1-80e8-3817da23ed9d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-2e9f81-29893380-zjnwh\" (UID: \"9546555f-4f4c-4ca1-80e8-3817da23ed9d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2e9f81-29893380-zjnwh" Nov 02 07:00:00.233691 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:00.233673 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1837c8a1-277f-4319-8e92-6e9e27e698c5-maas-api-openshift-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:00.234680 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:00.234659 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1837c8a1-277f-4319-8e92-6e9e27e698c5-maas-api-xks-ca" pod "1837c8a1-277f-4319-8e92-6e9e27e698c5" (UID: "1837c8a1-277f-4319-8e92-6e9e27e698c5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 07:00:00.234868 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:00.234845 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1837c8a1-277f-4319-8e92-6e9e27e698c5-kube-api-access-rr9jd" pod "1837c8a1-277f-4319-8e92-6e9e27e698c5" (UID: "1837c8a1-277f-4319-8e92-6e9e27e698c5"). InnerVolumeSpecName "kube-api-access-rr9jd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:00:00.266707 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:00.266669 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2e9f81-29893380-zjnwh"] Nov 02 07:00:00.336556 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:00.335854 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893380-kk78n"] Nov 02 07:00:00.336798 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:00.336766 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kpz9\" (UniqueName: \"kubernetes.io/projected/b6ee4d66-7acd-4f3f-ad2e-4650a1019c1c-kube-api-access-7kpz9\") pod \"maas-api-key-cleanup-e2e-worker-w1-cdfce9-29893380-gwwf5\" (UID: \"b6ee4d66-7acd-4f3f-ad2e-4650a1019c1c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cdfce9-29893380-gwwf5" Nov 02 07:00:00.337222 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:00.337200 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce7008b7-0096-4d6e-863e-60dce78fffe2-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893380-kk78n\" (UID: \"ce7008b7-0096-4d6e-863e-60dce78fffe2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893380-kk78n" Nov 02 07:00:00.337379 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:00.337362 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6ee4d66-7acd-4f3f-ad2e-4650a1019c1c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-cdfce9-29893380-gwwf5\" (UID: \"b6ee4d66-7acd-4f3f-ad2e-4650a1019c1c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cdfce9-29893380-gwwf5" Nov 02 07:00:00.337532 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:00.337514 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9f76fd8-1db3-4fe8-8a4e-117813676c2d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-a2ba11-29893380-f9dll\" (UID: \"f9f76fd8-1db3-4fe8-8a4e-117813676c2d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a2ba11-29893380-f9dll" Nov 02 07:00:00.337703 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:00.337685 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsqz2\" (UniqueName: \"kubernetes.io/projected/f9f76fd8-1db3-4fe8-8a4e-117813676c2d-kube-api-access-tsqz2\") pod \"maas-api-key-cleanup-e2e-inf-a-a2ba11-29893380-f9dll\" (UID: \"f9f76fd8-1db3-4fe8-8a4e-117813676c2d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a2ba11-29893380-f9dll" Nov 02 07:00:00.337891 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:00.337871 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghnhx\" (UniqueName: \"kubernetes.io/projected/9546555f-4f4c-4ca1-80e8-3817da23ed9d-kube-api-access-ghnhx\") pod \"maas-api-key-cleanup-e2e-inf-b-2e9f81-29893380-zjnwh\" (UID: \"9546555f-4f4c-4ca1-80e8-3817da23ed9d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2e9f81-29893380-zjnwh" Nov 02 07:00:00.338061 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:00.338044 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6qnd\" (UniqueName: \"kubernetes.io/projected/ce7008b7-0096-4d6e-863e-60dce78fffe2-kube-api-access-k6qnd\") pod \"maas-api-key-cleanup-29893380-kk78n\" (UID: \"ce7008b7-0096-4d6e-863e-60dce78fffe2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893380-kk78n" Nov 02 07:00:00.338273 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:00.338254 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9546555f-4f4c-4ca1-80e8-3817da23ed9d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-2e9f81-29893380-zjnwh\" (UID: \"9546555f-4f4c-4ca1-80e8-3817da23ed9d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2e9f81-29893380-zjnwh" Nov 02 07:00:00.338477 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:00.338460 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1837c8a1-277f-4319-8e92-6e9e27e698c5-maas-api-xks-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:00.338583 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:00.338570 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rr9jd\" (UniqueName: \"kubernetes.io/projected/1837c8a1-277f-4319-8e92-6e9e27e698c5-kube-api-access-rr9jd\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:00.338786 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:00.338759 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9f76fd8-1db3-4fe8-8a4e-117813676c2d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-a2ba11-29893380-f9dll\" (UID: \"f9f76fd8-1db3-4fe8-8a4e-117813676c2d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a2ba11-29893380-f9dll" Nov 02 07:00:00.339584 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:00.339557 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6ee4d66-7acd-4f3f-ad2e-4650a1019c1c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-cdfce9-29893380-gwwf5\" (UID: \"b6ee4d66-7acd-4f3f-ad2e-4650a1019c1c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cdfce9-29893380-gwwf5" Nov 02 07:00:00.339701 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:00.339569 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9546555f-4f4c-4ca1-80e8-3817da23ed9d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-2e9f81-29893380-zjnwh\" (UID: \"9546555f-4f4c-4ca1-80e8-3817da23ed9d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2e9f81-29893380-zjnwh" Nov 02 07:00:00.339852 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:00.338062 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce7008b7-0096-4d6e-863e-60dce78fffe2-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893380-kk78n\" (UID: \"ce7008b7-0096-4d6e-863e-60dce78fffe2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893380-kk78n" Nov 02 07:00:00.342127 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:00.342104 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a2ba11-29893380-f9dll"] Nov 02 07:00:00.346717 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:00.346556 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cdfce9-29893380-gwwf5"] Nov 02 07:00:00.405615 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:00.405579 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6qnd\" (UniqueName: \"kubernetes.io/projected/ce7008b7-0096-4d6e-863e-60dce78fffe2-kube-api-access-k6qnd\") pod \"maas-api-key-cleanup-29893380-kk78n\" (UID: \"ce7008b7-0096-4d6e-863e-60dce78fffe2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893380-kk78n" Nov 02 07:00:00.405828 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:00.405806 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsqz2\" (UniqueName: \"kubernetes.io/projected/f9f76fd8-1db3-4fe8-8a4e-117813676c2d-kube-api-access-tsqz2\") pod \"maas-api-key-cleanup-e2e-inf-a-a2ba11-29893380-f9dll\" (UID: \"f9f76fd8-1db3-4fe8-8a4e-117813676c2d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a2ba11-29893380-f9dll" Nov 02 07:00:00.405980 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:00.405821 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghnhx\" (UniqueName: \"kubernetes.io/projected/9546555f-4f4c-4ca1-80e8-3817da23ed9d-kube-api-access-ghnhx\") pod \"maas-api-key-cleanup-e2e-inf-b-2e9f81-29893380-zjnwh\" (UID: \"9546555f-4f4c-4ca1-80e8-3817da23ed9d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2e9f81-29893380-zjnwh" Nov 02 07:00:00.405980 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:00.405821 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kpz9\" (UniqueName: \"kubernetes.io/projected/b6ee4d66-7acd-4f3f-ad2e-4650a1019c1c-kube-api-access-7kpz9\") pod \"maas-api-key-cleanup-e2e-worker-w1-cdfce9-29893380-gwwf5\" (UID: \"b6ee4d66-7acd-4f3f-ad2e-4650a1019c1c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cdfce9-29893380-gwwf5" Nov 02 07:00:00.499738 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:00.499705 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2e9f81-29893380-zjnwh" Nov 02 07:00:00.508087 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:00.508065 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893380-kk78n" Nov 02 07:00:00.523176 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:00.523144 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cdfce9-29893380-gwwf5" Nov 02 07:00:00.530081 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:00.530046 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a2ba11-29893380-f9dll" Nov 02 07:00:00.895378 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:00.895351 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2e9f81-29893380-zjnwh"] Nov 02 07:00:00.895378 ip-10-0-130-93 kubenswrapper[5916]: W1102 07:00:00.895365 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9546555f_4f4c_4ca1_80e8_3817da23ed9d.slice/crio-b004e3c54e1ebef46cd86918e109b41badf7d2ac869ec8a6bf4bf2c1690acaaf WatchSource:0}: Error finding container b004e3c54e1ebef46cd86918e109b41badf7d2ac869ec8a6bf4bf2c1690acaaf: Status 404 returned error can't find the container with id b004e3c54e1ebef46cd86918e109b41badf7d2ac869ec8a6bf4bf2c1690acaaf Nov 02 07:00:00.946162 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:00.945977 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cdfce9-29893380-gwwf5"] Nov 02 07:00:00.954104 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:00.954074 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893380-kk78n"] Nov 02 07:00:00.972813 ip-10-0-130-93 kubenswrapper[5916]: W1102 07:00:00.972789 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9f76fd8_1db3_4fe8_8a4e_117813676c2d.slice/crio-97ae3d6409eab5723b2af96a8216c6400e1a7e5ec4de41a65af2923db1a7af6d WatchSource:0}: Error finding container 97ae3d6409eab5723b2af96a8216c6400e1a7e5ec4de41a65af2923db1a7af6d: Status 404 returned error can't find the container with id 97ae3d6409eab5723b2af96a8216c6400e1a7e5ec4de41a65af2923db1a7af6d Nov 02 07:00:00.973570 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:00.973541 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a2ba11-29893380-f9dll"] Nov 02 07:00:01.051093 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:01.051056 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a2ba11-29893380-f9dll" event={"ID":"f9f76fd8-1db3-4fe8-8a4e-117813676c2d","Type":"ContainerStarted","Data":"97ae3d6409eab5723b2af96a8216c6400e1a7e5ec4de41a65af2923db1a7af6d"} Nov 02 07:00:01.052774 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:01.052750 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-581608-mlz9r" Nov 02 07:00:01.052936 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:01.052782 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-581608-mlz9r" event={"ID":"1837c8a1-277f-4319-8e92-6e9e27e698c5","Type":"ContainerDied","Data":"5e65fa1e4291d28143aa87333bcf24951b793204d264b32613bf8cdc2d56fa8d"} Nov 02 07:00:01.052936 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:01.052851 5916 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5e65fa1e4291d28143aa87333bcf24951b793204d264b32613bf8cdc2d56fa8d" Nov 02 07:00:01.054342 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:01.054315 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893380-kk78n" event={"ID":"ce7008b7-0096-4d6e-863e-60dce78fffe2","Type":"ContainerStarted","Data":"6302be09de72a5f84ae1c69dfc1178660e20e3de8d93ea4cee12ce76a129356a"} Nov 02 07:00:01.059939 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:01.059917 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cdfce9-29893380-gwwf5" event={"ID":"b6ee4d66-7acd-4f3f-ad2e-4650a1019c1c","Type":"ContainerStarted","Data":"cb3c30e511d6ee8f64c8893a8d8728c0c9e6246d70ad58ade80ccc055cc25fc6"} Nov 02 07:00:01.061156 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:01.061134 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2e9f81-29893380-zjnwh" event={"ID":"9546555f-4f4c-4ca1-80e8-3817da23ed9d","Type":"ContainerStarted","Data":"b004e3c54e1ebef46cd86918e109b41badf7d2ac869ec8a6bf4bf2c1690acaaf"} Nov 02 07:00:01.196390 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:01.196364 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a2ba11-x74pv" Nov 02 07:00:01.249805 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:01.249775 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e4e8418-4884-4d61-a0c3-54dcd871255e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e4e8418-4884-4d61-a0c3-54dcd871255e-maas-api-openshift-ca\") pod \"1e4e8418-4884-4d61-a0c3-54dcd871255e\" (UID: \"1e4e8418-4884-4d61-a0c3-54dcd871255e\") " Nov 02 07:00:01.250052 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:01.250035 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e4e8418-4884-4d61-a0c3-54dcd871255e-kube-api-access-dnnch\" (UniqueName: \"kubernetes.io/projected/1e4e8418-4884-4d61-a0c3-54dcd871255e-kube-api-access-dnnch\") pod \"1e4e8418-4884-4d61-a0c3-54dcd871255e\" (UID: \"1e4e8418-4884-4d61-a0c3-54dcd871255e\") " Nov 02 07:00:01.250262 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:01.250247 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e4e8418-4884-4d61-a0c3-54dcd871255e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e4e8418-4884-4d61-a0c3-54dcd871255e-maas-api-xks-ca\") pod \"1e4e8418-4884-4d61-a0c3-54dcd871255e\" (UID: \"1e4e8418-4884-4d61-a0c3-54dcd871255e\") " Nov 02 07:00:01.250524 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:01.250496 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e4e8418-4884-4d61-a0c3-54dcd871255e-maas-api-openshift-ca" pod "1e4e8418-4884-4d61-a0c3-54dcd871255e" (UID: "1e4e8418-4884-4d61-a0c3-54dcd871255e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:00:01.250949 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:01.250931 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e4e8418-4884-4d61-a0c3-54dcd871255e-maas-api-openshift-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:01.253021 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:01.252994 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e4e8418-4884-4d61-a0c3-54dcd871255e-kube-api-access-dnnch" pod "1e4e8418-4884-4d61-a0c3-54dcd871255e" (UID: "1e4e8418-4884-4d61-a0c3-54dcd871255e"). InnerVolumeSpecName "kube-api-access-dnnch". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:00:01.253541 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:01.253519 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e4e8418-4884-4d61-a0c3-54dcd871255e-maas-api-xks-ca" pod "1e4e8418-4884-4d61-a0c3-54dcd871255e" (UID: "1e4e8418-4884-4d61-a0c3-54dcd871255e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 07:00:01.279051 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:01.279032 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8bc131-nhr4c" Nov 02 07:00:01.351828 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:01.351801 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c031ed74-a8ca-4080-8216-9e5c55d2fc6e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c031ed74-a8ca-4080-8216-9e5c55d2fc6e-maas-api-xks-ca\") pod \"c031ed74-a8ca-4080-8216-9e5c55d2fc6e\" (UID: \"c031ed74-a8ca-4080-8216-9e5c55d2fc6e\") " Nov 02 07:00:01.351952 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:01.351887 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c031ed74-a8ca-4080-8216-9e5c55d2fc6e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c031ed74-a8ca-4080-8216-9e5c55d2fc6e-maas-api-openshift-ca\") pod \"c031ed74-a8ca-4080-8216-9e5c55d2fc6e\" (UID: \"c031ed74-a8ca-4080-8216-9e5c55d2fc6e\") " Nov 02 07:00:01.352241 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:01.352217 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c031ed74-a8ca-4080-8216-9e5c55d2fc6e-maas-api-openshift-ca" pod "c031ed74-a8ca-4080-8216-9e5c55d2fc6e" (UID: "c031ed74-a8ca-4080-8216-9e5c55d2fc6e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:00:01.352470 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:01.352452 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c031ed74-a8ca-4080-8216-9e5c55d2fc6e-kube-api-access-pvxw2\" (UniqueName: \"kubernetes.io/projected/c031ed74-a8ca-4080-8216-9e5c55d2fc6e-kube-api-access-pvxw2\") pod \"c031ed74-a8ca-4080-8216-9e5c55d2fc6e\" (UID: \"c031ed74-a8ca-4080-8216-9e5c55d2fc6e\") " Nov 02 07:00:01.353411 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:01.353394 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c031ed74-a8ca-4080-8216-9e5c55d2fc6e-maas-api-openshift-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:01.353540 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:01.353528 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e4e8418-4884-4d61-a0c3-54dcd871255e-maas-api-xks-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:01.353627 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:01.353616 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnnch\" (UniqueName: \"kubernetes.io/projected/1e4e8418-4884-4d61-a0c3-54dcd871255e-kube-api-access-dnnch\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:01.354252 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:01.354226 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c031ed74-a8ca-4080-8216-9e5c55d2fc6e-maas-api-xks-ca" pod "c031ed74-a8ca-4080-8216-9e5c55d2fc6e" (UID: "c031ed74-a8ca-4080-8216-9e5c55d2fc6e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 07:00:01.354798 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:01.354771 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c031ed74-a8ca-4080-8216-9e5c55d2fc6e-kube-api-access-pvxw2" pod "c031ed74-a8ca-4080-8216-9e5c55d2fc6e" (UID: "c031ed74-a8ca-4080-8216-9e5c55d2fc6e"). InnerVolumeSpecName "kube-api-access-pvxw2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:00:01.456293 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:01.456265 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvxw2\" (UniqueName: \"kubernetes.io/projected/c031ed74-a8ca-4080-8216-9e5c55d2fc6e-kube-api-access-pvxw2\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:01.456408 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:01.456296 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c031ed74-a8ca-4080-8216-9e5c55d2fc6e-maas-api-xks-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:01.593328 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:01.593292 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-581608/distinct-2-w2-581608-kserve-57dd49c4b-qnt9k"] Nov 02 07:00:01.598858 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:01.598833 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-581608/distinct-w2-581608-kserve-6bb6bb7b65-v7kfj"] Nov 02 07:00:01.618409 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:01.618382 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-581608/simulator-w2-581608-kserve-68f68d8559-xkcwp"] Nov 02 07:00:02.059112 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:02.059071 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s6n9q"] Nov 02 07:00:02.059351 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:02.059324 5916 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s6n9q" podUID="ed18eacc-a61b-470d-8cd9-1d46f742fabd" containerName="discovery" containerID="cri-o://a21a0b93a292549a41fd8b3ea404a92291828fa217d3c069dc1fa840bbacf326" gracePeriod=30 Nov 02 07:00:02.076140 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:02.076096 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a2ba11-x74pv" event={"ID":"1e4e8418-4884-4d61-a0c3-54dcd871255e","Type":"ContainerDied","Data":"c515159b27179733a29a7742e363f7834444a208480b88928d594c1ba5727d6f"} Nov 02 07:00:02.076308 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:02.076162 5916 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c515159b27179733a29a7742e363f7834444a208480b88928d594c1ba5727d6f" Nov 02 07:00:02.076308 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:02.076111 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a2ba11-x74pv" Nov 02 07:00:02.078004 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:02.077975 5916 generic.go:356] "Generic (PLEG): container finished" podID="ce7008b7-0096-4d6e-863e-60dce78fffe2" containerID="f645505f036ef249ccd57ae086154740a3c0a20747886968803afa6b3ac71354" exitCode=60 Nov 02 07:00:02.078131 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:02.078083 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893380-kk78n" event={"ID":"ce7008b7-0096-4d6e-863e-60dce78fffe2","Type":"ContainerDied","Data":"f645505f036ef249ccd57ae086154740a3c0a20747886968803afa6b3ac71354"} Nov 02 07:00:02.078518 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:02.078501 5916 scope.go:122] "RemoveContainer" containerID="f645505f036ef249ccd57ae086154740a3c0a20747886968803afa6b3ac71354" Nov 02 07:00:02.082162 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:02.082138 5916 generic.go:356] "Generic (PLEG): container finished" podID="b6ee4d66-7acd-4f3f-ad2e-4650a1019c1c" containerID="269606f38423a4a926c9ea3cb227c6ca1353e9f2b8d38fca078fa53dab4cafc3" exitCode=60 Nov 02 07:00:02.082370 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:02.082278 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cdfce9-29893380-gwwf5" event={"ID":"b6ee4d66-7acd-4f3f-ad2e-4650a1019c1c","Type":"ContainerDied","Data":"269606f38423a4a926c9ea3cb227c6ca1353e9f2b8d38fca078fa53dab4cafc3"} Nov 02 07:00:02.082638 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:02.082475 5916 scope.go:122] "RemoveContainer" containerID="269606f38423a4a926c9ea3cb227c6ca1353e9f2b8d38fca078fa53dab4cafc3" Nov 02 07:00:02.088119 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:02.088056 5916 generic.go:356] "Generic (PLEG): container finished" podID="9546555f-4f4c-4ca1-80e8-3817da23ed9d" containerID="bed9d94a0683bd90694d902b1ecd366257afa3b4ddead50a6a537a201091b421" exitCode=60 Nov 02 07:00:02.088230 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:02.088184 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2e9f81-29893380-zjnwh" event={"ID":"9546555f-4f4c-4ca1-80e8-3817da23ed9d","Type":"ContainerDied","Data":"bed9d94a0683bd90694d902b1ecd366257afa3b4ddead50a6a537a201091b421"} Nov 02 07:00:02.089148 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:02.089128 5916 scope.go:122] "RemoveContainer" containerID="bed9d94a0683bd90694d902b1ecd366257afa3b4ddead50a6a537a201091b421" Nov 02 07:00:02.091001 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:02.090974 5916 generic.go:356] "Generic (PLEG): container finished" podID="f9f76fd8-1db3-4fe8-8a4e-117813676c2d" containerID="df5f7222435df0d9607bb55ad2fdf54c96894ac3278c503b88a0d73e1a0d2dfd" exitCode=60 Nov 02 07:00:02.091094 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:02.091057 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a2ba11-29893380-f9dll" event={"ID":"f9f76fd8-1db3-4fe8-8a4e-117813676c2d","Type":"ContainerDied","Data":"df5f7222435df0d9607bb55ad2fdf54c96894ac3278c503b88a0d73e1a0d2dfd"} Nov 02 07:00:02.091380 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:02.091359 5916 scope.go:122] "RemoveContainer" containerID="df5f7222435df0d9607bb55ad2fdf54c96894ac3278c503b88a0d73e1a0d2dfd" Nov 02 07:00:02.101394 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:02.093174 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8bc131-nhr4c" event={"ID":"c031ed74-a8ca-4080-8216-9e5c55d2fc6e","Type":"ContainerDied","Data":"8a74948ec4f97ae00e129116c2c2988c72d4c280a07a829d2c3a8a1d65c24fe2"} Nov 02 07:00:02.101394 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:02.093221 5916 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8a74948ec4f97ae00e129116c2c2988c72d4c280a07a829d2c3a8a1d65c24fe2" Nov 02 07:00:02.101394 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:02.093278 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8bc131-nhr4c" Nov 02 07:00:02.338343 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:02.336471 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s6n9q" Nov 02 07:00:02.371043 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:02.371009 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed18eacc-a61b-470d-8cd9-1d46f742fabd-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ed18eacc-a61b-470d-8cd9-1d46f742fabd-istio-csr-ca-configmap\") pod \"ed18eacc-a61b-470d-8cd9-1d46f742fabd\" (UID: \"ed18eacc-a61b-470d-8cd9-1d46f742fabd\") " Nov 02 07:00:02.371238 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:02.371062 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed18eacc-a61b-470d-8cd9-1d46f742fabd-istio-token\" (UniqueName: \"kubernetes.io/projected/ed18eacc-a61b-470d-8cd9-1d46f742fabd-istio-token\") pod \"ed18eacc-a61b-470d-8cd9-1d46f742fabd\" (UID: \"ed18eacc-a61b-470d-8cd9-1d46f742fabd\") " Nov 02 07:00:02.371238 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:02.371137 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed18eacc-a61b-470d-8cd9-1d46f742fabd-kube-api-access-nzjzt\" (UniqueName: \"kubernetes.io/projected/ed18eacc-a61b-470d-8cd9-1d46f742fabd-kube-api-access-nzjzt\") pod \"ed18eacc-a61b-470d-8cd9-1d46f742fabd\" (UID: \"ed18eacc-a61b-470d-8cd9-1d46f742fabd\") " Nov 02 07:00:02.371238 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:02.371204 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed18eacc-a61b-470d-8cd9-1d46f742fabd-cacerts\" (UniqueName: \"kubernetes.io/secret/ed18eacc-a61b-470d-8cd9-1d46f742fabd-cacerts\") pod \"ed18eacc-a61b-470d-8cd9-1d46f742fabd\" (UID: \"ed18eacc-a61b-470d-8cd9-1d46f742fabd\") " Nov 02 07:00:02.371415 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:02.371248 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed18eacc-a61b-470d-8cd9-1d46f742fabd-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ed18eacc-a61b-470d-8cd9-1d46f742fabd-istio-csr-dns-cert\") pod \"ed18eacc-a61b-470d-8cd9-1d46f742fabd\" (UID: \"ed18eacc-a61b-470d-8cd9-1d46f742fabd\") " Nov 02 07:00:02.371415 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:02.371284 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed18eacc-a61b-470d-8cd9-1d46f742fabd-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ed18eacc-a61b-470d-8cd9-1d46f742fabd-istio-kubeconfig\") pod \"ed18eacc-a61b-470d-8cd9-1d46f742fabd\" (UID: \"ed18eacc-a61b-470d-8cd9-1d46f742fabd\") " Nov 02 07:00:02.371415 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:02.371320 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed18eacc-a61b-470d-8cd9-1d46f742fabd-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed18eacc-a61b-470d-8cd9-1d46f742fabd-local-certs\") pod \"ed18eacc-a61b-470d-8cd9-1d46f742fabd\" (UID: \"ed18eacc-a61b-470d-8cd9-1d46f742fabd\") " Nov 02 07:00:02.371571 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:02.371403 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed18eacc-a61b-470d-8cd9-1d46f742fabd-istio-csr-ca-configmap" pod "ed18eacc-a61b-470d-8cd9-1d46f742fabd" (UID: "ed18eacc-a61b-470d-8cd9-1d46f742fabd"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:00:02.371812 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:02.371789 5916 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ed18eacc-a61b-470d-8cd9-1d46f742fabd-istio-csr-ca-configmap\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:02.426625 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:02.426584 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed18eacc-a61b-470d-8cd9-1d46f742fabd-istio-csr-dns-cert" pod "ed18eacc-a61b-470d-8cd9-1d46f742fabd" (UID: "ed18eacc-a61b-470d-8cd9-1d46f742fabd"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 07:00:02.426816 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:02.426702 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed18eacc-a61b-470d-8cd9-1d46f742fabd-istio-token" pod "ed18eacc-a61b-470d-8cd9-1d46f742fabd" (UID: "ed18eacc-a61b-470d-8cd9-1d46f742fabd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:00:02.427074 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:02.427046 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed18eacc-a61b-470d-8cd9-1d46f742fabd-cacerts" pod "ed18eacc-a61b-470d-8cd9-1d46f742fabd" (UID: "ed18eacc-a61b-470d-8cd9-1d46f742fabd"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 07:00:02.427357 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:02.427330 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed18eacc-a61b-470d-8cd9-1d46f742fabd-kube-api-access-nzjzt" pod "ed18eacc-a61b-470d-8cd9-1d46f742fabd" (UID: "ed18eacc-a61b-470d-8cd9-1d46f742fabd"). InnerVolumeSpecName "kube-api-access-nzjzt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:00:02.427528 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:02.427354 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed18eacc-a61b-470d-8cd9-1d46f742fabd-local-certs" pod "ed18eacc-a61b-470d-8cd9-1d46f742fabd" (UID: "ed18eacc-a61b-470d-8cd9-1d46f742fabd"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:00:02.427691 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:02.427669 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed18eacc-a61b-470d-8cd9-1d46f742fabd-istio-kubeconfig" pod "ed18eacc-a61b-470d-8cd9-1d46f742fabd" (UID: "ed18eacc-a61b-470d-8cd9-1d46f742fabd"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 07:00:02.473615 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:02.473579 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzjzt\" (UniqueName: \"kubernetes.io/projected/ed18eacc-a61b-470d-8cd9-1d46f742fabd-kube-api-access-nzjzt\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:02.473615 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:02.473617 5916 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ed18eacc-a61b-470d-8cd9-1d46f742fabd-cacerts\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:02.473848 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:02.473632 5916 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ed18eacc-a61b-470d-8cd9-1d46f742fabd-istio-csr-dns-cert\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:02.473848 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:02.473648 5916 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ed18eacc-a61b-470d-8cd9-1d46f742fabd-istio-kubeconfig\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:02.473848 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:02.473662 5916 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed18eacc-a61b-470d-8cd9-1d46f742fabd-local-certs\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:02.473848 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:02.473676 5916 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed18eacc-a61b-470d-8cd9-1d46f742fabd-istio-token\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:03.102863 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:03.102825 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893380-kk78n" event={"ID":"ce7008b7-0096-4d6e-863e-60dce78fffe2","Type":"ContainerStarted","Data":"0c3b52e30ea799944e98f00d0c4b99b4a6e6be248ac128be4ffd49e343104392"} Nov 02 07:00:03.104741 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:03.104717 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cdfce9-29893380-gwwf5" event={"ID":"b6ee4d66-7acd-4f3f-ad2e-4650a1019c1c","Type":"ContainerStarted","Data":"16c252f4e3f45ff9cf25357dc77cd5acdd1e7c683d5949c36d54aad20d995d2f"} Nov 02 07:00:03.106666 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:03.106646 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2e9f81-29893380-zjnwh" event={"ID":"9546555f-4f4c-4ca1-80e8-3817da23ed9d","Type":"ContainerStarted","Data":"00cd510c487984aabbe0594d9d307ab248ed06b8d6eb6f085785daab56501aeb"} Nov 02 07:00:03.108591 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:03.108571 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a2ba11-29893380-f9dll" event={"ID":"f9f76fd8-1db3-4fe8-8a4e-117813676c2d","Type":"ContainerStarted","Data":"3e3336afaa1f94318d78402e8e798a937748dbcb7c4c10dc237403dc35082fce"} Nov 02 07:00:03.110241 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:03.110214 5916 generic.go:356] "Generic (PLEG): container finished" podID="ed18eacc-a61b-470d-8cd9-1d46f742fabd" containerID="a21a0b93a292549a41fd8b3ea404a92291828fa217d3c069dc1fa840bbacf326" exitCode=0 Nov 02 07:00:03.110338 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:03.110259 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s6n9q" Nov 02 07:00:03.110338 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:03.110287 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s6n9q" event={"ID":"ed18eacc-a61b-470d-8cd9-1d46f742fabd","Type":"ContainerDied","Data":"a21a0b93a292549a41fd8b3ea404a92291828fa217d3c069dc1fa840bbacf326"} Nov 02 07:00:03.110338 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:03.110332 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s6n9q" event={"ID":"ed18eacc-a61b-470d-8cd9-1d46f742fabd","Type":"ContainerDied","Data":"5fb756db6e9c34fddaf9c0fc14bf15f109ac09f551f4eb64863c65ff49d9093a"} Nov 02 07:00:03.110450 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:03.110354 5916 scope.go:122] "RemoveContainer" containerID="a21a0b93a292549a41fd8b3ea404a92291828fa217d3c069dc1fa840bbacf326" Nov 02 07:00:03.122556 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:03.122538 5916 scope.go:122] "RemoveContainer" containerID="a21a0b93a292549a41fd8b3ea404a92291828fa217d3c069dc1fa840bbacf326" Nov 02 07:00:03.122817 ip-10-0-130-93 kubenswrapper[5916]: E1102 07:00:03.122796 5916 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a21a0b93a292549a41fd8b3ea404a92291828fa217d3c069dc1fa840bbacf326\": container with ID starting with a21a0b93a292549a41fd8b3ea404a92291828fa217d3c069dc1fa840bbacf326 not found: ID does not exist" containerID="a21a0b93a292549a41fd8b3ea404a92291828fa217d3c069dc1fa840bbacf326" Nov 02 07:00:03.122875 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:03.122829 5916 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a21a0b93a292549a41fd8b3ea404a92291828fa217d3c069dc1fa840bbacf326"} err="failed to get container status \"a21a0b93a292549a41fd8b3ea404a92291828fa217d3c069dc1fa840bbacf326\": rpc error: code = NotFound desc = could not find container \"a21a0b93a292549a41fd8b3ea404a92291828fa217d3c069dc1fa840bbacf326\": container with ID starting with a21a0b93a292549a41fd8b3ea404a92291828fa217d3c069dc1fa840bbacf326 not found: ID does not exist" Nov 02 07:00:03.227439 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:03.227382 5916 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893380-kk78n" podStartSLOduration=3.227364389 podStartE2EDuration="3.227364389s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:00:03.222796467 +0000 UTC m=+2591.494616875" watchObservedRunningTime="2026-11-02 07:00:03.227364389 +0000 UTC m=+2591.499184785" Nov 02 07:00:03.341415 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:03.341367 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s6n9q"] Nov 02 07:00:03.344899 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:03.344873 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s6n9q"] Nov 02 07:00:03.454358 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:03.454300 5916 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2e9f81-29893380-zjnwh" podStartSLOduration=3.454281924 podStartE2EDuration="3.454281924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:00:03.452385521 +0000 UTC m=+2591.724205915" watchObservedRunningTime="2026-11-02 07:00:03.454281924 +0000 UTC m=+2591.726102320" Nov 02 07:00:03.566057 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:03.566002 5916 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cdfce9-29893380-gwwf5" podStartSLOduration=3.565989589 podStartE2EDuration="3.565989589s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:00:03.564439573 +0000 UTC m=+2591.836259968" watchObservedRunningTime="2026-11-02 07:00:03.565989589 +0000 UTC m=+2591.837810011" Nov 02 07:00:03.680764 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:03.680715 5916 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a2ba11-29893380-f9dll" podStartSLOduration=3.680702917 podStartE2EDuration="3.680702917s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:00:03.679295746 +0000 UTC m=+2591.951116139" watchObservedRunningTime="2026-11-02 07:00:03.680702917 +0000 UTC m=+2591.952523310" Nov 02 07:00:03.847472 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:03.847438 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed18eacc-a61b-470d-8cd9-1d46f742fabd" path="/var/lib/kubelet/pods/ed18eacc-a61b-470d-8cd9-1d46f742fabd/volumes" Nov 02 07:00:04.121572 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:04.121485 5916 generic.go:356] "Generic (PLEG): container finished" podID="ce7008b7-0096-4d6e-863e-60dce78fffe2" containerID="0c3b52e30ea799944e98f00d0c4b99b4a6e6be248ac128be4ffd49e343104392" exitCode=60 Nov 02 07:00:04.121755 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:04.121569 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893380-kk78n" event={"ID":"ce7008b7-0096-4d6e-863e-60dce78fffe2","Type":"ContainerDied","Data":"0c3b52e30ea799944e98f00d0c4b99b4a6e6be248ac128be4ffd49e343104392"} Nov 02 07:00:04.121755 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:04.121629 5916 scope.go:122] "RemoveContainer" containerID="f645505f036ef249ccd57ae086154740a3c0a20747886968803afa6b3ac71354" Nov 02 07:00:04.121896 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:04.121814 5916 scope.go:122] "RemoveContainer" containerID="0c3b52e30ea799944e98f00d0c4b99b4a6e6be248ac128be4ffd49e343104392" Nov 02 07:00:04.122126 ip-10-0-130-93 kubenswrapper[5916]: E1102 07:00:04.122103 5916 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29893380-kk78n_odh-ai-gateway-infra(ce7008b7-0096-4d6e-863e-60dce78fffe2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893380-kk78n" podUID="ce7008b7-0096-4d6e-863e-60dce78fffe2" Nov 02 07:00:04.123749 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:04.123726 5916 generic.go:356] "Generic (PLEG): container finished" podID="b6ee4d66-7acd-4f3f-ad2e-4650a1019c1c" containerID="16c252f4e3f45ff9cf25357dc77cd5acdd1e7c683d5949c36d54aad20d995d2f" exitCode=60 Nov 02 07:00:04.123855 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:04.123804 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cdfce9-29893380-gwwf5" event={"ID":"b6ee4d66-7acd-4f3f-ad2e-4650a1019c1c","Type":"ContainerDied","Data":"16c252f4e3f45ff9cf25357dc77cd5acdd1e7c683d5949c36d54aad20d995d2f"} Nov 02 07:00:04.124220 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:04.124083 5916 scope.go:122] "RemoveContainer" containerID="16c252f4e3f45ff9cf25357dc77cd5acdd1e7c683d5949c36d54aad20d995d2f" Nov 02 07:00:04.124358 ip-10-0-130-93 kubenswrapper[5916]: E1102 07:00:04.124338 5916 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-cdfce9-29893380-gwwf5_odh-ai-gateway-infra(b6ee4d66-7acd-4f3f-ad2e-4650a1019c1c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cdfce9-29893380-gwwf5" podUID="b6ee4d66-7acd-4f3f-ad2e-4650a1019c1c" Nov 02 07:00:04.125824 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:04.125803 5916 generic.go:356] "Generic (PLEG): container finished" podID="9546555f-4f4c-4ca1-80e8-3817da23ed9d" containerID="00cd510c487984aabbe0594d9d307ab248ed06b8d6eb6f085785daab56501aeb" exitCode=60 Nov 02 07:00:04.125948 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:04.125930 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2e9f81-29893380-zjnwh" event={"ID":"9546555f-4f4c-4ca1-80e8-3817da23ed9d","Type":"ContainerDied","Data":"00cd510c487984aabbe0594d9d307ab248ed06b8d6eb6f085785daab56501aeb"} Nov 02 07:00:04.126245 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:04.126224 5916 scope.go:122] "RemoveContainer" containerID="00cd510c487984aabbe0594d9d307ab248ed06b8d6eb6f085785daab56501aeb" Nov 02 07:00:04.126610 ip-10-0-130-93 kubenswrapper[5916]: E1102 07:00:04.126584 5916 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-2e9f81-29893380-zjnwh_odh-ai-gateway-infra(9546555f-4f4c-4ca1-80e8-3817da23ed9d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2e9f81-29893380-zjnwh" podUID="9546555f-4f4c-4ca1-80e8-3817da23ed9d" Nov 02 07:00:04.128257 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:04.128235 5916 generic.go:356] "Generic (PLEG): container finished" podID="f9f76fd8-1db3-4fe8-8a4e-117813676c2d" containerID="3e3336afaa1f94318d78402e8e798a937748dbcb7c4c10dc237403dc35082fce" exitCode=60 Nov 02 07:00:04.128362 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:04.128289 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a2ba11-29893380-f9dll" event={"ID":"f9f76fd8-1db3-4fe8-8a4e-117813676c2d","Type":"ContainerDied","Data":"3e3336afaa1f94318d78402e8e798a937748dbcb7c4c10dc237403dc35082fce"} Nov 02 07:00:04.128488 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:04.128474 5916 scope.go:122] "RemoveContainer" containerID="3e3336afaa1f94318d78402e8e798a937748dbcb7c4c10dc237403dc35082fce" Nov 02 07:00:04.128668 ip-10-0-130-93 kubenswrapper[5916]: E1102 07:00:04.128650 5916 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-a2ba11-29893380-f9dll_odh-ai-gateway-infra(f9f76fd8-1db3-4fe8-8a4e-117813676c2d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a2ba11-29893380-f9dll" podUID="f9f76fd8-1db3-4fe8-8a4e-117813676c2d" Nov 02 07:00:04.137267 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:04.137242 5916 scope.go:122] "RemoveContainer" containerID="269606f38423a4a926c9ea3cb227c6ca1353e9f2b8d38fca078fa53dab4cafc3" Nov 02 07:00:04.153683 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:04.153661 5916 scope.go:122] "RemoveContainer" containerID="bed9d94a0683bd90694d902b1ecd366257afa3b4ddead50a6a537a201091b421" Nov 02 07:00:04.169115 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:04.169092 5916 scope.go:122] "RemoveContainer" containerID="df5f7222435df0d9607bb55ad2fdf54c96894ac3278c503b88a0d73e1a0d2dfd" Nov 02 07:00:05.139398 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:05.138975 5916 scope.go:122] "RemoveContainer" containerID="0c3b52e30ea799944e98f00d0c4b99b4a6e6be248ac128be4ffd49e343104392" Nov 02 07:00:05.139398 ip-10-0-130-93 kubenswrapper[5916]: E1102 07:00:05.139290 5916 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29893380-kk78n_odh-ai-gateway-infra(ce7008b7-0096-4d6e-863e-60dce78fffe2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893380-kk78n" podUID="ce7008b7-0096-4d6e-863e-60dce78fffe2" Nov 02 07:00:05.140690 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:05.140665 5916 scope.go:122] "RemoveContainer" containerID="16c252f4e3f45ff9cf25357dc77cd5acdd1e7c683d5949c36d54aad20d995d2f" Nov 02 07:00:05.140972 ip-10-0-130-93 kubenswrapper[5916]: E1102 07:00:05.140948 5916 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-cdfce9-29893380-gwwf5_odh-ai-gateway-infra(b6ee4d66-7acd-4f3f-ad2e-4650a1019c1c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cdfce9-29893380-gwwf5" podUID="b6ee4d66-7acd-4f3f-ad2e-4650a1019c1c" Nov 02 07:00:05.142513 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:05.142494 5916 scope.go:122] "RemoveContainer" containerID="00cd510c487984aabbe0594d9d307ab248ed06b8d6eb6f085785daab56501aeb" Nov 02 07:00:05.142686 ip-10-0-130-93 kubenswrapper[5916]: E1102 07:00:05.142667 5916 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-2e9f81-29893380-zjnwh_odh-ai-gateway-infra(9546555f-4f4c-4ca1-80e8-3817da23ed9d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2e9f81-29893380-zjnwh" podUID="9546555f-4f4c-4ca1-80e8-3817da23ed9d" Nov 02 07:00:05.144288 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:05.144271 5916 scope.go:122] "RemoveContainer" containerID="3e3336afaa1f94318d78402e8e798a937748dbcb7c4c10dc237403dc35082fce" Nov 02 07:00:05.144446 ip-10-0-130-93 kubenswrapper[5916]: E1102 07:00:05.144429 5916 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-a2ba11-29893380-f9dll_odh-ai-gateway-infra(f9f76fd8-1db3-4fe8-8a4e-117813676c2d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a2ba11-29893380-f9dll" podUID="f9f76fd8-1db3-4fe8-8a4e-117813676c2d" Nov 02 07:00:07.321693 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:07.321654 5916 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-s6n9q container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.138:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 07:00:07.322164 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:07.321714 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s6n9q" podUID="ed18eacc-a61b-470d-8cd9-1d46f742fabd" containerName="discovery" probeResult="failure" output="Get \"http://10.132.0.138:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 07:00:08.338403 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:08.338357 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-581608-666ff49d8c-2466x"] Nov 02 07:00:10.530900 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:10.530863 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a2ba11-58967fcd4f-qpggq"] Nov 02 07:00:10.639367 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:10.639332 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a2ba11-5c66894c8c-p472q"] Nov 02 07:00:11.052181 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:11.052154 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-73e2cf7f-zjks2"] Nov 02 07:00:11.055472 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:11.055446 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9a43fa3-7tt7w"] Nov 02 07:00:11.058112 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:11.058092 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-73e2cf7f-zjks2"] Nov 02 07:00:11.060621 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:11.060599 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9a43fa3-7tt7w"] Nov 02 07:00:11.842817 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:11.842779 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39513a10-9b43-4963-8b4b-265187974870" path="/var/lib/kubelet/pods/39513a10-9b43-4963-8b4b-265187974870/volumes" Nov 02 07:00:11.843456 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:11.843434 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b23fcf38-1c33-4668-890c-436e1b54b7a4" path="/var/lib/kubelet/pods/b23fcf38-1c33-4668-890c-436e1b54b7a4/volumes" Nov 02 07:00:13.339983 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:13.339933 5916 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-581608-666ff49d8c-2466x" podUID="63b6b2e6-166d-4571-b148-6c504875697c" containerName="payload-pre-processing" containerID="cri-o://e8049b156c9b93ea0f393748b64b6b2c992316b3b9b5488443e7c1ba0d70eb59" gracePeriod=25 Nov 02 07:00:14.835366 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:14.835324 5916 scope.go:122] "RemoveContainer" containerID="3e3336afaa1f94318d78402e8e798a937748dbcb7c4c10dc237403dc35082fce" Nov 02 07:00:14.835925 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:14.835470 5916 scope.go:122] "RemoveContainer" containerID="0c3b52e30ea799944e98f00d0c4b99b4a6e6be248ac128be4ffd49e343104392" Nov 02 07:00:14.835925 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:14.835686 5916 scope.go:122] "RemoveContainer" containerID="00cd510c487984aabbe0594d9d307ab248ed06b8d6eb6f085785daab56501aeb" Nov 02 07:00:15.244468 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:15.244437 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893380-kk78n" event={"ID":"ce7008b7-0096-4d6e-863e-60dce78fffe2","Type":"ContainerStarted","Data":"86d3646601247905878ef903df423ba09adbd6c38b30b132a827c0a53f9a19bd"} Nov 02 07:00:15.246501 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:15.246478 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2e9f81-29893380-zjnwh" event={"ID":"9546555f-4f4c-4ca1-80e8-3817da23ed9d","Type":"ContainerStarted","Data":"377f3df186c3b3b0c8594d005c652a602b73b218e382f83bc4c6d4b400027c12"} Nov 02 07:00:15.531363 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:15.531324 5916 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-a2ba11-58967fcd4f-qpggq" podUID="610f1b35-d018-4931-815c-98bfc24ebfc5" containerName="payload-processing" containerID="cri-o://55dc76fa41f5d3c108cb7aeedd7f5e4862e80c3b1daac12d38b4ccf6e8738080" gracePeriod=25 Nov 02 07:00:15.639937 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:15.639867 5916 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a2ba11-5c66894c8c-p472q" podUID="1936eac5-6027-48a9-86ff-12ad457b94d9" containerName="payload-pre-processing" containerID="cri-o://fd52bf8f16f3bd30c336e3412703766a5dcbf0e40f02ae13ddf2c1b564aec501" gracePeriod=25 Nov 02 07:00:16.258733 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:16.258694 5916 generic.go:356] "Generic (PLEG): container finished" podID="ce7008b7-0096-4d6e-863e-60dce78fffe2" containerID="86d3646601247905878ef903df423ba09adbd6c38b30b132a827c0a53f9a19bd" exitCode=60 Nov 02 07:00:16.259288 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:16.258774 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893380-kk78n" event={"ID":"ce7008b7-0096-4d6e-863e-60dce78fffe2","Type":"ContainerDied","Data":"86d3646601247905878ef903df423ba09adbd6c38b30b132a827c0a53f9a19bd"} Nov 02 07:00:16.259288 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:16.258845 5916 scope.go:122] "RemoveContainer" containerID="0c3b52e30ea799944e98f00d0c4b99b4a6e6be248ac128be4ffd49e343104392" Nov 02 07:00:16.259288 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:16.259276 5916 scope.go:122] "RemoveContainer" containerID="86d3646601247905878ef903df423ba09adbd6c38b30b132a827c0a53f9a19bd" Nov 02 07:00:16.259738 ip-10-0-130-93 kubenswrapper[5916]: E1102 07:00:16.259614 5916 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29893380-kk78n_odh-ai-gateway-infra(ce7008b7-0096-4d6e-863e-60dce78fffe2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893380-kk78n" podUID="ce7008b7-0096-4d6e-863e-60dce78fffe2" Nov 02 07:00:16.261461 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:16.261441 5916 generic.go:356] "Generic (PLEG): container finished" podID="9546555f-4f4c-4ca1-80e8-3817da23ed9d" containerID="377f3df186c3b3b0c8594d005c652a602b73b218e382f83bc4c6d4b400027c12" exitCode=60 Nov 02 07:00:16.261565 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:16.261511 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2e9f81-29893380-zjnwh" event={"ID":"9546555f-4f4c-4ca1-80e8-3817da23ed9d","Type":"ContainerDied","Data":"377f3df186c3b3b0c8594d005c652a602b73b218e382f83bc4c6d4b400027c12"} Nov 02 07:00:16.261875 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:16.261852 5916 scope.go:122] "RemoveContainer" containerID="377f3df186c3b3b0c8594d005c652a602b73b218e382f83bc4c6d4b400027c12" Nov 02 07:00:16.262185 ip-10-0-130-93 kubenswrapper[5916]: E1102 07:00:16.262166 5916 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-2e9f81-29893380-zjnwh_odh-ai-gateway-infra(9546555f-4f4c-4ca1-80e8-3817da23ed9d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2e9f81-29893380-zjnwh" podUID="9546555f-4f4c-4ca1-80e8-3817da23ed9d" Nov 02 07:00:16.263604 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:16.263583 5916 generic.go:356] "Generic (PLEG): container finished" podID="f9f76fd8-1db3-4fe8-8a4e-117813676c2d" containerID="bb617061e11493cce42d2801a59ce687d41b06d6d9cdb136edcb13217d0a0609" exitCode=60 Nov 02 07:00:16.263705 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:16.263629 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a2ba11-29893380-f9dll" event={"ID":"f9f76fd8-1db3-4fe8-8a4e-117813676c2d","Type":"ContainerDied","Data":"bb617061e11493cce42d2801a59ce687d41b06d6d9cdb136edcb13217d0a0609"} Nov 02 07:00:16.263966 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:16.263950 5916 scope.go:122] "RemoveContainer" containerID="bb617061e11493cce42d2801a59ce687d41b06d6d9cdb136edcb13217d0a0609" Nov 02 07:00:16.264189 ip-10-0-130-93 kubenswrapper[5916]: E1102 07:00:16.264172 5916 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-a2ba11-29893380-f9dll_odh-ai-gateway-infra(f9f76fd8-1db3-4fe8-8a4e-117813676c2d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a2ba11-29893380-f9dll" podUID="f9f76fd8-1db3-4fe8-8a4e-117813676c2d" Nov 02 07:00:16.273810 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:16.273793 5916 scope.go:122] "RemoveContainer" containerID="00cd510c487984aabbe0594d9d307ab248ed06b8d6eb6f085785daab56501aeb" Nov 02 07:00:16.286886 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:16.286866 5916 scope.go:122] "RemoveContainer" containerID="3e3336afaa1f94318d78402e8e798a937748dbcb7c4c10dc237403dc35082fce" Nov 02 07:00:16.372046 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:16.372013 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893380-kk78n"] Nov 02 07:00:16.485122 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:16.485084 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2e9f81-29893380-zjnwh"] Nov 02 07:00:16.614706 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:16.614594 5916 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-581608/distinct-2-w2-581608-kserve-57dd49c4b-qnt9k" podUID="837780ad-8ea3-4ed2-922f-917cffc7bb93" containerName="main" containerID="cri-o://ae464271d3cd0b7c4574877c5e265e528ac0a69b46ab0602818df9ca80517383" gracePeriod=45 Nov 02 07:00:16.621182 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:16.621124 5916 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-581608/distinct-w2-581608-kserve-6bb6bb7b65-v7kfj" podUID="dcf2f562-3c50-4de7-9c84-cb659d3bc25b" containerName="main" containerID="cri-o://7faa43095cc60bd7ff99255ae17c318ec6ccea9860454a6b281c5f57b2e65baf" gracePeriod=45 Nov 02 07:00:16.639474 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:16.639439 5916 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-581608/simulator-w2-581608-kserve-68f68d8559-xkcwp" podUID="529e25e6-4479-49d6-85c7-85dad6c2193e" containerName="main" containerID="cri-o://2b2274da40934012bf54016614c8a6ed76dd176cfb0e64478c63aade679a591c" gracePeriod=45 Nov 02 07:00:16.820643 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:16.818702 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-581608/simulator-w2-581608-kserve-68f68d8559-xkcwp" podUID="529e25e6-4479-49d6-85c7-85dad6c2193e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": dial tcp 10.132.0.68:8000: connect: connection refused" Nov 02 07:00:16.941953 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:16.925687 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-581608/distinct-2-w2-581608-kserve-57dd49c4b-qnt9k" Nov 02 07:00:16.982590 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:16.982562 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-581608/distinct-w2-581608-kserve-6bb6bb7b65-v7kfj" Nov 02 07:00:17.027149 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:17.027124 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-581608/simulator-w2-581608-kserve-68f68d8559-xkcwp" Nov 02 07:00:17.070241 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:17.070204 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/837780ad-8ea3-4ed2-922f-917cffc7bb93-kube-api-access-w4jsm\" (UniqueName: \"kubernetes.io/projected/837780ad-8ea3-4ed2-922f-917cffc7bb93-kube-api-access-w4jsm\") pod \"837780ad-8ea3-4ed2-922f-917cffc7bb93\" (UID: \"837780ad-8ea3-4ed2-922f-917cffc7bb93\") " Nov 02 07:00:17.070383 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:17.070306 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/837780ad-8ea3-4ed2-922f-917cffc7bb93-dshm\" (UniqueName: \"kubernetes.io/empty-dir/837780ad-8ea3-4ed2-922f-917cffc7bb93-dshm\") pod \"837780ad-8ea3-4ed2-922f-917cffc7bb93\" (UID: \"837780ad-8ea3-4ed2-922f-917cffc7bb93\") " Nov 02 07:00:17.070383 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:17.070342 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/837780ad-8ea3-4ed2-922f-917cffc7bb93-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/837780ad-8ea3-4ed2-922f-917cffc7bb93-tmp-dir\") pod \"837780ad-8ea3-4ed2-922f-917cffc7bb93\" (UID: \"837780ad-8ea3-4ed2-922f-917cffc7bb93\") " Nov 02 07:00:17.070468 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:17.070440 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/837780ad-8ea3-4ed2-922f-917cffc7bb93-tls-certs\" (UniqueName: \"kubernetes.io/secret/837780ad-8ea3-4ed2-922f-917cffc7bb93-tls-certs\") pod \"837780ad-8ea3-4ed2-922f-917cffc7bb93\" (UID: \"837780ad-8ea3-4ed2-922f-917cffc7bb93\") " Nov 02 07:00:17.070505 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:17.070488 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/837780ad-8ea3-4ed2-922f-917cffc7bb93-home\" (UniqueName: \"kubernetes.io/empty-dir/837780ad-8ea3-4ed2-922f-917cffc7bb93-home\") pod \"837780ad-8ea3-4ed2-922f-917cffc7bb93\" (UID: \"837780ad-8ea3-4ed2-922f-917cffc7bb93\") " Nov 02 07:00:17.070829 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:17.070807 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/837780ad-8ea3-4ed2-922f-917cffc7bb93-home" pod "837780ad-8ea3-4ed2-922f-917cffc7bb93" (UID: "837780ad-8ea3-4ed2-922f-917cffc7bb93"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:00:17.073615 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:17.073584 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/837780ad-8ea3-4ed2-922f-917cffc7bb93-kube-api-access-w4jsm" pod "837780ad-8ea3-4ed2-922f-917cffc7bb93" (UID: "837780ad-8ea3-4ed2-922f-917cffc7bb93"). InnerVolumeSpecName "kube-api-access-w4jsm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:00:17.076482 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:17.076225 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/837780ad-8ea3-4ed2-922f-917cffc7bb93-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/837780ad-8ea3-4ed2-922f-917cffc7bb93-model-cache\") pod \"837780ad-8ea3-4ed2-922f-917cffc7bb93\" (UID: \"837780ad-8ea3-4ed2-922f-917cffc7bb93\") " Nov 02 07:00:17.076482 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:17.076412 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/837780ad-8ea3-4ed2-922f-917cffc7bb93-tmp-dir" pod "837780ad-8ea3-4ed2-922f-917cffc7bb93" (UID: "837780ad-8ea3-4ed2-922f-917cffc7bb93"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:00:17.076801 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:17.076773 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4jsm\" (UniqueName: \"kubernetes.io/projected/837780ad-8ea3-4ed2-922f-917cffc7bb93-kube-api-access-w4jsm\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:17.076891 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:17.076806 5916 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/837780ad-8ea3-4ed2-922f-917cffc7bb93-tmp-dir\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:17.076891 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:17.076822 5916 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/837780ad-8ea3-4ed2-922f-917cffc7bb93-home\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:17.076891 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:17.076825 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/837780ad-8ea3-4ed2-922f-917cffc7bb93-dshm" pod "837780ad-8ea3-4ed2-922f-917cffc7bb93" (UID: "837780ad-8ea3-4ed2-922f-917cffc7bb93"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:00:17.076891 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:17.076877 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/837780ad-8ea3-4ed2-922f-917cffc7bb93-model-cache" pod "837780ad-8ea3-4ed2-922f-917cffc7bb93" (UID: "837780ad-8ea3-4ed2-922f-917cffc7bb93"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:00:17.078416 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:17.078396 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/837780ad-8ea3-4ed2-922f-917cffc7bb93-tls-certs" pod "837780ad-8ea3-4ed2-922f-917cffc7bb93" (UID: "837780ad-8ea3-4ed2-922f-917cffc7bb93"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 07:00:17.178030 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:17.177995 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/529e25e6-4479-49d6-85c7-85dad6c2193e-tls-certs\" (UniqueName: \"kubernetes.io/secret/529e25e6-4479-49d6-85c7-85dad6c2193e-tls-certs\") pod \"529e25e6-4479-49d6-85c7-85dad6c2193e\" (UID: \"529e25e6-4479-49d6-85c7-85dad6c2193e\") " Nov 02 07:00:17.178030 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:17.178038 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcf2f562-3c50-4de7-9c84-cb659d3bc25b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dcf2f562-3c50-4de7-9c84-cb659d3bc25b-tmp-dir\") pod \"dcf2f562-3c50-4de7-9c84-cb659d3bc25b\" (UID: \"dcf2f562-3c50-4de7-9c84-cb659d3bc25b\") " Nov 02 07:00:17.178300 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:17.178067 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/529e25e6-4479-49d6-85c7-85dad6c2193e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/529e25e6-4479-49d6-85c7-85dad6c2193e-model-cache\") pod \"529e25e6-4479-49d6-85c7-85dad6c2193e\" (UID: \"529e25e6-4479-49d6-85c7-85dad6c2193e\") " Nov 02 07:00:17.178300 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:17.178119 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/529e25e6-4479-49d6-85c7-85dad6c2193e-kube-api-access-8f7k7\" (UniqueName: \"kubernetes.io/projected/529e25e6-4479-49d6-85c7-85dad6c2193e-kube-api-access-8f7k7\") pod \"529e25e6-4479-49d6-85c7-85dad6c2193e\" (UID: \"529e25e6-4479-49d6-85c7-85dad6c2193e\") " Nov 02 07:00:17.178300 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:17.178150 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcf2f562-3c50-4de7-9c84-cb659d3bc25b-home\" (UniqueName: \"kubernetes.io/empty-dir/dcf2f562-3c50-4de7-9c84-cb659d3bc25b-home\") pod \"dcf2f562-3c50-4de7-9c84-cb659d3bc25b\" (UID: \"dcf2f562-3c50-4de7-9c84-cb659d3bc25b\") " Nov 02 07:00:17.178300 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:17.178181 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcf2f562-3c50-4de7-9c84-cb659d3bc25b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dcf2f562-3c50-4de7-9c84-cb659d3bc25b-model-cache\") pod \"dcf2f562-3c50-4de7-9c84-cb659d3bc25b\" (UID: \"dcf2f562-3c50-4de7-9c84-cb659d3bc25b\") " Nov 02 07:00:17.178300 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:17.178220 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/529e25e6-4479-49d6-85c7-85dad6c2193e-home\" (UniqueName: \"kubernetes.io/empty-dir/529e25e6-4479-49d6-85c7-85dad6c2193e-home\") pod \"529e25e6-4479-49d6-85c7-85dad6c2193e\" (UID: \"529e25e6-4479-49d6-85c7-85dad6c2193e\") " Nov 02 07:00:17.178300 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:17.178274 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/529e25e6-4479-49d6-85c7-85dad6c2193e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/529e25e6-4479-49d6-85c7-85dad6c2193e-dshm\") pod \"529e25e6-4479-49d6-85c7-85dad6c2193e\" (UID: \"529e25e6-4479-49d6-85c7-85dad6c2193e\") " Nov 02 07:00:17.178300 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:17.178284 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcf2f562-3c50-4de7-9c84-cb659d3bc25b-tmp-dir" pod "dcf2f562-3c50-4de7-9c84-cb659d3bc25b" (UID: "dcf2f562-3c50-4de7-9c84-cb659d3bc25b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:00:17.178712 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:17.178305 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/529e25e6-4479-49d6-85c7-85dad6c2193e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/529e25e6-4479-49d6-85c7-85dad6c2193e-tmp-dir\") pod \"529e25e6-4479-49d6-85c7-85dad6c2193e\" (UID: \"529e25e6-4479-49d6-85c7-85dad6c2193e\") " Nov 02 07:00:17.178712 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:17.178363 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/529e25e6-4479-49d6-85c7-85dad6c2193e-model-cache" pod "529e25e6-4479-49d6-85c7-85dad6c2193e" (UID: "529e25e6-4479-49d6-85c7-85dad6c2193e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:00:17.178712 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:17.178388 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcf2f562-3c50-4de7-9c84-cb659d3bc25b-kube-api-access-4j8g2\" (UniqueName: \"kubernetes.io/projected/dcf2f562-3c50-4de7-9c84-cb659d3bc25b-kube-api-access-4j8g2\") pod \"dcf2f562-3c50-4de7-9c84-cb659d3bc25b\" (UID: \"dcf2f562-3c50-4de7-9c84-cb659d3bc25b\") " Nov 02 07:00:17.178712 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:17.178417 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcf2f562-3c50-4de7-9c84-cb659d3bc25b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/dcf2f562-3c50-4de7-9c84-cb659d3bc25b-dshm\") pod \"dcf2f562-3c50-4de7-9c84-cb659d3bc25b\" (UID: \"dcf2f562-3c50-4de7-9c84-cb659d3bc25b\") " Nov 02 07:00:17.178712 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:17.178454 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcf2f562-3c50-4de7-9c84-cb659d3bc25b-model-cache" pod "dcf2f562-3c50-4de7-9c84-cb659d3bc25b" (UID: "dcf2f562-3c50-4de7-9c84-cb659d3bc25b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:00:17.178712 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:17.178537 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dcf2f562-3c50-4de7-9c84-cb659d3bc25b-tls-certs\" (UniqueName: \"kubernetes.io/secret/dcf2f562-3c50-4de7-9c84-cb659d3bc25b-tls-certs\") pod \"dcf2f562-3c50-4de7-9c84-cb659d3bc25b\" (UID: \"dcf2f562-3c50-4de7-9c84-cb659d3bc25b\") " Nov 02 07:00:17.178712 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:17.178555 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcf2f562-3c50-4de7-9c84-cb659d3bc25b-home" pod "dcf2f562-3c50-4de7-9c84-cb659d3bc25b" (UID: "dcf2f562-3c50-4de7-9c84-cb659d3bc25b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:00:17.179193 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:17.179157 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/529e25e6-4479-49d6-85c7-85dad6c2193e-home" pod "529e25e6-4479-49d6-85c7-85dad6c2193e" (UID: "529e25e6-4479-49d6-85c7-85dad6c2193e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:00:17.179279 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:17.179250 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/529e25e6-4479-49d6-85c7-85dad6c2193e-tmp-dir" pod "529e25e6-4479-49d6-85c7-85dad6c2193e" (UID: "529e25e6-4479-49d6-85c7-85dad6c2193e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:00:17.179533 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:17.179508 5916 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/837780ad-8ea3-4ed2-922f-917cffc7bb93-tls-certs\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:17.179605 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:17.179540 5916 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dcf2f562-3c50-4de7-9c84-cb659d3bc25b-home\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:17.179605 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:17.179556 5916 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dcf2f562-3c50-4de7-9c84-cb659d3bc25b-model-cache\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:17.179605 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:17.179571 5916 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/529e25e6-4479-49d6-85c7-85dad6c2193e-home\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:17.179605 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:17.179584 5916 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/529e25e6-4479-49d6-85c7-85dad6c2193e-tmp-dir\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:17.179605 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:17.179596 5916 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/837780ad-8ea3-4ed2-922f-917cffc7bb93-model-cache\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:17.179941 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:17.179607 5916 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/837780ad-8ea3-4ed2-922f-917cffc7bb93-dshm\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:17.179941 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:17.179621 5916 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dcf2f562-3c50-4de7-9c84-cb659d3bc25b-tmp-dir\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:17.179941 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:17.179633 5916 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/529e25e6-4479-49d6-85c7-85dad6c2193e-model-cache\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:17.184063 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:17.184025 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/529e25e6-4479-49d6-85c7-85dad6c2193e-tls-certs" pod "529e25e6-4479-49d6-85c7-85dad6c2193e" (UID: "529e25e6-4479-49d6-85c7-85dad6c2193e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 07:00:17.184191 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:17.184066 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/529e25e6-4479-49d6-85c7-85dad6c2193e-kube-api-access-8f7k7" pod "529e25e6-4479-49d6-85c7-85dad6c2193e" (UID: "529e25e6-4479-49d6-85c7-85dad6c2193e"). InnerVolumeSpecName "kube-api-access-8f7k7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:00:17.184191 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:17.184076 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dcf2f562-3c50-4de7-9c84-cb659d3bc25b-tls-certs" pod "dcf2f562-3c50-4de7-9c84-cb659d3bc25b" (UID: "dcf2f562-3c50-4de7-9c84-cb659d3bc25b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 07:00:17.184870 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:17.184835 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/529e25e6-4479-49d6-85c7-85dad6c2193e-dshm" pod "529e25e6-4479-49d6-85c7-85dad6c2193e" (UID: "529e25e6-4479-49d6-85c7-85dad6c2193e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:00:17.186204 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:17.186170 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcf2f562-3c50-4de7-9c84-cb659d3bc25b-dshm" pod "dcf2f562-3c50-4de7-9c84-cb659d3bc25b" (UID: "dcf2f562-3c50-4de7-9c84-cb659d3bc25b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:00:17.186304 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:17.186219 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcf2f562-3c50-4de7-9c84-cb659d3bc25b-kube-api-access-4j8g2" pod "dcf2f562-3c50-4de7-9c84-cb659d3bc25b" (UID: "dcf2f562-3c50-4de7-9c84-cb659d3bc25b"). InnerVolumeSpecName "kube-api-access-4j8g2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:00:17.275775 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:17.275720 5916 generic.go:356] "Generic (PLEG): container finished" podID="dcf2f562-3c50-4de7-9c84-cb659d3bc25b" containerID="7faa43095cc60bd7ff99255ae17c318ec6ccea9860454a6b281c5f57b2e65baf" exitCode=0 Nov 02 07:00:17.275775 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:17.275771 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-581608/distinct-w2-581608-kserve-6bb6bb7b65-v7kfj" Nov 02 07:00:17.276321 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:17.275808 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-581608/distinct-w2-581608-kserve-6bb6bb7b65-v7kfj" event={"ID":"dcf2f562-3c50-4de7-9c84-cb659d3bc25b","Type":"ContainerDied","Data":"7faa43095cc60bd7ff99255ae17c318ec6ccea9860454a6b281c5f57b2e65baf"} Nov 02 07:00:17.276321 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:17.275857 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-581608/distinct-w2-581608-kserve-6bb6bb7b65-v7kfj" event={"ID":"dcf2f562-3c50-4de7-9c84-cb659d3bc25b","Type":"ContainerDied","Data":"be49cc29f4950957803648d757499df4f7fc2b10ab2a8b8cb5a4ec13c3bf2997"} Nov 02 07:00:17.276321 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:17.275880 5916 scope.go:122] "RemoveContainer" containerID="7faa43095cc60bd7ff99255ae17c318ec6ccea9860454a6b281c5f57b2e65baf" Nov 02 07:00:17.280519 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:17.280491 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8f7k7\" (UniqueName: \"kubernetes.io/projected/529e25e6-4479-49d6-85c7-85dad6c2193e-kube-api-access-8f7k7\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:17.280519 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:17.280518 5916 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/529e25e6-4479-49d6-85c7-85dad6c2193e-dshm\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:17.280724 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:17.280532 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4j8g2\" (UniqueName: \"kubernetes.io/projected/dcf2f562-3c50-4de7-9c84-cb659d3bc25b-kube-api-access-4j8g2\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:17.280724 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:17.280544 5916 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dcf2f562-3c50-4de7-9c84-cb659d3bc25b-dshm\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:17.280724 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:17.280555 5916 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dcf2f562-3c50-4de7-9c84-cb659d3bc25b-tls-certs\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:17.280724 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:17.280566 5916 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/529e25e6-4479-49d6-85c7-85dad6c2193e-tls-certs\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:17.282421 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:17.282393 5916 generic.go:356] "Generic (PLEG): container finished" podID="529e25e6-4479-49d6-85c7-85dad6c2193e" containerID="2b2274da40934012bf54016614c8a6ed76dd176cfb0e64478c63aade679a591c" exitCode=0 Nov 02 07:00:17.282554 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:17.282436 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-581608/simulator-w2-581608-kserve-68f68d8559-xkcwp" Nov 02 07:00:17.282554 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:17.282493 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-581608/simulator-w2-581608-kserve-68f68d8559-xkcwp" event={"ID":"529e25e6-4479-49d6-85c7-85dad6c2193e","Type":"ContainerDied","Data":"2b2274da40934012bf54016614c8a6ed76dd176cfb0e64478c63aade679a591c"} Nov 02 07:00:17.282554 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:17.282528 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-581608/simulator-w2-581608-kserve-68f68d8559-xkcwp" event={"ID":"529e25e6-4479-49d6-85c7-85dad6c2193e","Type":"ContainerDied","Data":"acf644edf4c4c5afb4e1e3a3e11bf43b0f7963851c69e157638e261aad1f1465"} Nov 02 07:00:17.287949 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:17.286443 5916 generic.go:356] "Generic (PLEG): container finished" podID="837780ad-8ea3-4ed2-922f-917cffc7bb93" containerID="ae464271d3cd0b7c4574877c5e265e528ac0a69b46ab0602818df9ca80517383" exitCode=0 Nov 02 07:00:17.287949 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:17.286508 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-581608/distinct-2-w2-581608-kserve-57dd49c4b-qnt9k" event={"ID":"837780ad-8ea3-4ed2-922f-917cffc7bb93","Type":"ContainerDied","Data":"ae464271d3cd0b7c4574877c5e265e528ac0a69b46ab0602818df9ca80517383"} Nov 02 07:00:17.287949 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:17.286551 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-581608/distinct-2-w2-581608-kserve-57dd49c4b-qnt9k" event={"ID":"837780ad-8ea3-4ed2-922f-917cffc7bb93","Type":"ContainerDied","Data":"652c8f205729f878d461dd7345d43a37798fcf44b58127210f03d068b9e05346"} Nov 02 07:00:17.287949 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:17.286606 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-581608/distinct-2-w2-581608-kserve-57dd49c4b-qnt9k" Nov 02 07:00:17.294290 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:17.294266 5916 scope.go:122] "RemoveContainer" containerID="7faa43095cc60bd7ff99255ae17c318ec6ccea9860454a6b281c5f57b2e65baf" Nov 02 07:00:17.294690 ip-10-0-130-93 kubenswrapper[5916]: E1102 07:00:17.294655 5916 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7faa43095cc60bd7ff99255ae17c318ec6ccea9860454a6b281c5f57b2e65baf\": container with ID starting with 7faa43095cc60bd7ff99255ae17c318ec6ccea9860454a6b281c5f57b2e65baf not found: ID does not exist" containerID="7faa43095cc60bd7ff99255ae17c318ec6ccea9860454a6b281c5f57b2e65baf" Nov 02 07:00:17.294777 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:17.294702 5916 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7faa43095cc60bd7ff99255ae17c318ec6ccea9860454a6b281c5f57b2e65baf"} err="failed to get container status \"7faa43095cc60bd7ff99255ae17c318ec6ccea9860454a6b281c5f57b2e65baf\": rpc error: code = NotFound desc = could not find container \"7faa43095cc60bd7ff99255ae17c318ec6ccea9860454a6b281c5f57b2e65baf\": container with ID starting with 7faa43095cc60bd7ff99255ae17c318ec6ccea9860454a6b281c5f57b2e65baf not found: ID does not exist" Nov 02 07:00:17.294777 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:17.294728 5916 scope.go:122] "RemoveContainer" containerID="2b2274da40934012bf54016614c8a6ed76dd176cfb0e64478c63aade679a591c" Nov 02 07:00:17.314328 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:17.314300 5916 scope.go:122] "RemoveContainer" containerID="2b2274da40934012bf54016614c8a6ed76dd176cfb0e64478c63aade679a591c" Nov 02 07:00:17.314826 ip-10-0-130-93 kubenswrapper[5916]: E1102 07:00:17.314792 5916 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b2274da40934012bf54016614c8a6ed76dd176cfb0e64478c63aade679a591c\": container with ID starting with 2b2274da40934012bf54016614c8a6ed76dd176cfb0e64478c63aade679a591c not found: ID does not exist" containerID="2b2274da40934012bf54016614c8a6ed76dd176cfb0e64478c63aade679a591c" Nov 02 07:00:17.314917 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:17.314839 5916 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b2274da40934012bf54016614c8a6ed76dd176cfb0e64478c63aade679a591c"} err="failed to get container status \"2b2274da40934012bf54016614c8a6ed76dd176cfb0e64478c63aade679a591c\": rpc error: code = NotFound desc = could not find container \"2b2274da40934012bf54016614c8a6ed76dd176cfb0e64478c63aade679a591c\": container with ID starting with 2b2274da40934012bf54016614c8a6ed76dd176cfb0e64478c63aade679a591c not found: ID does not exist" Nov 02 07:00:17.314917 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:17.314879 5916 scope.go:122] "RemoveContainer" containerID="ae464271d3cd0b7c4574877c5e265e528ac0a69b46ab0602818df9ca80517383" Nov 02 07:00:17.328784 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:17.328759 5916 scope.go:122] "RemoveContainer" containerID="ae464271d3cd0b7c4574877c5e265e528ac0a69b46ab0602818df9ca80517383" Nov 02 07:00:17.329352 ip-10-0-130-93 kubenswrapper[5916]: E1102 07:00:17.329275 5916 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae464271d3cd0b7c4574877c5e265e528ac0a69b46ab0602818df9ca80517383\": container with ID starting with ae464271d3cd0b7c4574877c5e265e528ac0a69b46ab0602818df9ca80517383 not found: ID does not exist" containerID="ae464271d3cd0b7c4574877c5e265e528ac0a69b46ab0602818df9ca80517383" Nov 02 07:00:17.329352 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:17.329314 5916 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae464271d3cd0b7c4574877c5e265e528ac0a69b46ab0602818df9ca80517383"} err="failed to get container status \"ae464271d3cd0b7c4574877c5e265e528ac0a69b46ab0602818df9ca80517383\": rpc error: code = NotFound desc = could not find container \"ae464271d3cd0b7c4574877c5e265e528ac0a69b46ab0602818df9ca80517383\": container with ID starting with ae464271d3cd0b7c4574877c5e265e528ac0a69b46ab0602818df9ca80517383 not found: ID does not exist" Nov 02 07:00:17.487059 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:17.487032 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893380-kk78n" Nov 02 07:00:17.495404 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:17.495376 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2e9f81-29893380-zjnwh" Nov 02 07:00:17.538675 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:17.538642 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-581608/distinct-w2-581608-kserve-6bb6bb7b65-v7kfj"] Nov 02 07:00:17.544924 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:17.544870 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-581608/distinct-w2-581608-kserve-6bb6bb7b65-v7kfj"] Nov 02 07:00:17.583635 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:17.583599 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce7008b7-0096-4d6e-863e-60dce78fffe2-kube-api-access-k6qnd\" (UniqueName: \"kubernetes.io/projected/ce7008b7-0096-4d6e-863e-60dce78fffe2-kube-api-access-k6qnd\") pod \"ce7008b7-0096-4d6e-863e-60dce78fffe2\" (UID: \"ce7008b7-0096-4d6e-863e-60dce78fffe2\") " Nov 02 07:00:17.583635 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:17.583641 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9546555f-4f4c-4ca1-80e8-3817da23ed9d-kube-api-access-ghnhx\" (UniqueName: \"kubernetes.io/projected/9546555f-4f4c-4ca1-80e8-3817da23ed9d-kube-api-access-ghnhx\") pod \"9546555f-4f4c-4ca1-80e8-3817da23ed9d\" (UID: \"9546555f-4f4c-4ca1-80e8-3817da23ed9d\") " Nov 02 07:00:17.583866 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:17.583741 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9546555f-4f4c-4ca1-80e8-3817da23ed9d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9546555f-4f4c-4ca1-80e8-3817da23ed9d-maas-api-service-ca\") pod \"9546555f-4f4c-4ca1-80e8-3817da23ed9d\" (UID: \"9546555f-4f4c-4ca1-80e8-3817da23ed9d\") " Nov 02 07:00:17.583866 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:17.583796 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce7008b7-0096-4d6e-863e-60dce78fffe2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce7008b7-0096-4d6e-863e-60dce78fffe2-maas-api-service-ca\") pod \"ce7008b7-0096-4d6e-863e-60dce78fffe2\" (UID: \"ce7008b7-0096-4d6e-863e-60dce78fffe2\") " Nov 02 07:00:17.584151 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:17.584124 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9546555f-4f4c-4ca1-80e8-3817da23ed9d-maas-api-service-ca" pod "9546555f-4f4c-4ca1-80e8-3817da23ed9d" (UID: "9546555f-4f4c-4ca1-80e8-3817da23ed9d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:00:17.584219 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:17.584189 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce7008b7-0096-4d6e-863e-60dce78fffe2-maas-api-service-ca" pod "ce7008b7-0096-4d6e-863e-60dce78fffe2" (UID: "ce7008b7-0096-4d6e-863e-60dce78fffe2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:00:17.584314 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:17.584298 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9546555f-4f4c-4ca1-80e8-3817da23ed9d-maas-api-service-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:17.584380 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:17.584316 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce7008b7-0096-4d6e-863e-60dce78fffe2-maas-api-service-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:17.585848 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:17.585819 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9546555f-4f4c-4ca1-80e8-3817da23ed9d-kube-api-access-ghnhx" pod "9546555f-4f4c-4ca1-80e8-3817da23ed9d" (UID: "9546555f-4f4c-4ca1-80e8-3817da23ed9d"). InnerVolumeSpecName "kube-api-access-ghnhx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:00:17.586091 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:17.586072 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce7008b7-0096-4d6e-863e-60dce78fffe2-kube-api-access-k6qnd" pod "ce7008b7-0096-4d6e-863e-60dce78fffe2" (UID: "ce7008b7-0096-4d6e-863e-60dce78fffe2"). InnerVolumeSpecName "kube-api-access-k6qnd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:00:17.618922 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:17.618874 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a2ba11-29893380-f9dll"] Nov 02 07:00:17.685676 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:17.685632 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6qnd\" (UniqueName: \"kubernetes.io/projected/ce7008b7-0096-4d6e-863e-60dce78fffe2-kube-api-access-k6qnd\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:17.685676 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:17.685679 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghnhx\" (UniqueName: \"kubernetes.io/projected/9546555f-4f4c-4ca1-80e8-3817da23ed9d-kube-api-access-ghnhx\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:17.766392 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:17.766364 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a2ba11-29893380-f9dll" Nov 02 07:00:17.786635 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:17.786592 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9f76fd8-1db3-4fe8-8a4e-117813676c2d-kube-api-access-tsqz2\" (UniqueName: \"kubernetes.io/projected/f9f76fd8-1db3-4fe8-8a4e-117813676c2d-kube-api-access-tsqz2\") pod \"f9f76fd8-1db3-4fe8-8a4e-117813676c2d\" (UID: \"f9f76fd8-1db3-4fe8-8a4e-117813676c2d\") " Nov 02 07:00:17.786818 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:17.786750 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9f76fd8-1db3-4fe8-8a4e-117813676c2d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9f76fd8-1db3-4fe8-8a4e-117813676c2d-maas-api-service-ca\") pod \"f9f76fd8-1db3-4fe8-8a4e-117813676c2d\" (UID: \"f9f76fd8-1db3-4fe8-8a4e-117813676c2d\") " Nov 02 07:00:17.787853 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:17.787138 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9f76fd8-1db3-4fe8-8a4e-117813676c2d-maas-api-service-ca" pod "f9f76fd8-1db3-4fe8-8a4e-117813676c2d" (UID: "f9f76fd8-1db3-4fe8-8a4e-117813676c2d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:00:17.789252 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:17.789225 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9f76fd8-1db3-4fe8-8a4e-117813676c2d-kube-api-access-tsqz2" pod "f9f76fd8-1db3-4fe8-8a4e-117813676c2d" (UID: "f9f76fd8-1db3-4fe8-8a4e-117813676c2d"). InnerVolumeSpecName "kube-api-access-tsqz2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:00:17.835970 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:17.835936 5916 scope.go:122] "RemoveContainer" containerID="16c252f4e3f45ff9cf25357dc77cd5acdd1e7c683d5949c36d54aad20d995d2f" Nov 02 07:00:17.843692 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:17.843665 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dcf2f562-3c50-4de7-9c84-cb659d3bc25b" path="/var/lib/kubelet/pods/dcf2f562-3c50-4de7-9c84-cb659d3bc25b/volumes" Nov 02 07:00:17.879291 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:17.879259 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-581608/distinct-2-w2-581608-kserve-57dd49c4b-qnt9k"] Nov 02 07:00:17.884235 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:17.884207 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-581608/distinct-2-w2-581608-kserve-57dd49c4b-qnt9k"] Nov 02 07:00:17.888406 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:17.888377 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9f76fd8-1db3-4fe8-8a4e-117813676c2d-maas-api-service-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:17.888406 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:17.888401 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tsqz2\" (UniqueName: \"kubernetes.io/projected/f9f76fd8-1db3-4fe8-8a4e-117813676c2d-kube-api-access-tsqz2\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:18.161088 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:18.161015 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-581608/simulator-w2-581608-kserve-68f68d8559-xkcwp"] Nov 02 07:00:18.164269 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:18.164239 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-581608/simulator-w2-581608-kserve-68f68d8559-xkcwp"] Nov 02 07:00:18.295888 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:18.295774 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cdfce9-29893380-gwwf5" event={"ID":"b6ee4d66-7acd-4f3f-ad2e-4650a1019c1c","Type":"ContainerStarted","Data":"4bd6b8899867b00d0a108241b98646d20ebf8ea5b13d72e5bbcc8047f7acb458"} Nov 02 07:00:18.297215 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:18.297193 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2e9f81-29893380-zjnwh" Nov 02 07:00:18.297346 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:18.297228 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2e9f81-29893380-zjnwh" event={"ID":"9546555f-4f4c-4ca1-80e8-3817da23ed9d","Type":"ContainerDied","Data":"b004e3c54e1ebef46cd86918e109b41badf7d2ac869ec8a6bf4bf2c1690acaaf"} Nov 02 07:00:18.297346 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:18.297284 5916 scope.go:122] "RemoveContainer" containerID="377f3df186c3b3b0c8594d005c652a602b73b218e382f83bc4c6d4b400027c12" Nov 02 07:00:18.299931 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:18.299897 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a2ba11-29893380-f9dll" Nov 02 07:00:18.300058 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:18.299950 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a2ba11-29893380-f9dll" event={"ID":"f9f76fd8-1db3-4fe8-8a4e-117813676c2d","Type":"ContainerDied","Data":"97ae3d6409eab5723b2af96a8216c6400e1a7e5ec4de41a65af2923db1a7af6d"} Nov 02 07:00:18.303787 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:18.303766 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893380-kk78n" event={"ID":"ce7008b7-0096-4d6e-863e-60dce78fffe2","Type":"ContainerDied","Data":"6302be09de72a5f84ae1c69dfc1178660e20e3de8d93ea4cee12ce76a129356a"} Nov 02 07:00:18.303889 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:18.303773 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893380-kk78n" Nov 02 07:00:18.308595 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:18.308577 5916 scope.go:122] "RemoveContainer" containerID="bb617061e11493cce42d2801a59ce687d41b06d6d9cdb136edcb13217d0a0609" Nov 02 07:00:18.318342 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:18.318317 5916 scope.go:122] "RemoveContainer" containerID="86d3646601247905878ef903df423ba09adbd6c38b30b132a827c0a53f9a19bd" Nov 02 07:00:18.472527 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:18.472490 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893380-kk78n"] Nov 02 07:00:18.475426 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:18.475389 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893380-kk78n"] Nov 02 07:00:18.583283 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:18.583179 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2e9f81-29893380-zjnwh"] Nov 02 07:00:18.589137 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:18.587765 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2e9f81-29893380-zjnwh"] Nov 02 07:00:18.871735 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:18.871706 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a2ba11-29893380-f9dll"] Nov 02 07:00:18.875184 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:18.875157 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a2ba11-29893380-f9dll"] Nov 02 07:00:19.325710 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:19.325663 5916 generic.go:356] "Generic (PLEG): container finished" podID="b6ee4d66-7acd-4f3f-ad2e-4650a1019c1c" containerID="4bd6b8899867b00d0a108241b98646d20ebf8ea5b13d72e5bbcc8047f7acb458" exitCode=60 Nov 02 07:00:19.326259 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:19.325714 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cdfce9-29893380-gwwf5" event={"ID":"b6ee4d66-7acd-4f3f-ad2e-4650a1019c1c","Type":"ContainerDied","Data":"4bd6b8899867b00d0a108241b98646d20ebf8ea5b13d72e5bbcc8047f7acb458"} Nov 02 07:00:19.326259 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:19.325851 5916 scope.go:122] "RemoveContainer" containerID="16c252f4e3f45ff9cf25357dc77cd5acdd1e7c683d5949c36d54aad20d995d2f" Nov 02 07:00:19.326259 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:19.326219 5916 scope.go:122] "RemoveContainer" containerID="4bd6b8899867b00d0a108241b98646d20ebf8ea5b13d72e5bbcc8047f7acb458" Nov 02 07:00:19.326707 ip-10-0-130-93 kubenswrapper[5916]: E1102 07:00:19.326506 5916 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-cdfce9-29893380-gwwf5_odh-ai-gateway-infra(b6ee4d66-7acd-4f3f-ad2e-4650a1019c1c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cdfce9-29893380-gwwf5" podUID="b6ee4d66-7acd-4f3f-ad2e-4650a1019c1c" Nov 02 07:00:19.651856 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:19.651824 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a2ba11-5c66894c8c-p472q" podUID="1936eac5-6027-48a9-86ff-12ad457b94d9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 07:00:19.652048 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:19.651825 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-a2ba11-58967fcd4f-qpggq" podUID="610f1b35-d018-4931-815c-98bfc24ebfc5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 07:00:19.708682 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:19.708650 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cdfce9-29893380-gwwf5"] Nov 02 07:00:19.775110 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:19.775073 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-581608-666ff49d8c-2466x" podUID="63b6b2e6-166d-4571-b148-6c504875697c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 07:00:19.843699 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:19.843666 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="529e25e6-4479-49d6-85c7-85dad6c2193e" path="/var/lib/kubelet/pods/529e25e6-4479-49d6-85c7-85dad6c2193e/volumes" Nov 02 07:00:19.844153 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:19.844137 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="837780ad-8ea3-4ed2-922f-917cffc7bb93" path="/var/lib/kubelet/pods/837780ad-8ea3-4ed2-922f-917cffc7bb93/volumes" Nov 02 07:00:19.844519 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:19.844507 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9546555f-4f4c-4ca1-80e8-3817da23ed9d" path="/var/lib/kubelet/pods/9546555f-4f4c-4ca1-80e8-3817da23ed9d/volumes" Nov 02 07:00:19.845218 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:19.845195 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce7008b7-0096-4d6e-863e-60dce78fffe2" path="/var/lib/kubelet/pods/ce7008b7-0096-4d6e-863e-60dce78fffe2/volumes" Nov 02 07:00:19.845569 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:19.845557 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9f76fd8-1db3-4fe8-8a4e-117813676c2d" path="/var/lib/kubelet/pods/f9f76fd8-1db3-4fe8-8a4e-117813676c2d/volumes" Nov 02 07:00:19.900752 ip-10-0-130-93 kubenswrapper[5916]: E1102 07:00:19.900633 5916 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p98cq" containerName="extractor" Nov 02 07:00:20.483007 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:20.482984 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cdfce9-29893380-gwwf5" Nov 02 07:00:20.518146 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:20.518118 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6ee4d66-7acd-4f3f-ad2e-4650a1019c1c-kube-api-access-7kpz9\" (UniqueName: \"kubernetes.io/projected/b6ee4d66-7acd-4f3f-ad2e-4650a1019c1c-kube-api-access-7kpz9\") pod \"b6ee4d66-7acd-4f3f-ad2e-4650a1019c1c\" (UID: \"b6ee4d66-7acd-4f3f-ad2e-4650a1019c1c\") " Nov 02 07:00:20.518305 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:20.518191 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6ee4d66-7acd-4f3f-ad2e-4650a1019c1c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6ee4d66-7acd-4f3f-ad2e-4650a1019c1c-maas-api-service-ca\") pod \"b6ee4d66-7acd-4f3f-ad2e-4650a1019c1c\" (UID: \"b6ee4d66-7acd-4f3f-ad2e-4650a1019c1c\") " Nov 02 07:00:20.518558 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:20.518524 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6ee4d66-7acd-4f3f-ad2e-4650a1019c1c-maas-api-service-ca" pod "b6ee4d66-7acd-4f3f-ad2e-4650a1019c1c" (UID: "b6ee4d66-7acd-4f3f-ad2e-4650a1019c1c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:00:20.520296 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:20.520273 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6ee4d66-7acd-4f3f-ad2e-4650a1019c1c-kube-api-access-7kpz9" pod "b6ee4d66-7acd-4f3f-ad2e-4650a1019c1c" (UID: "b6ee4d66-7acd-4f3f-ad2e-4650a1019c1c"). InnerVolumeSpecName "kube-api-access-7kpz9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:00:20.619842 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:20.619758 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7kpz9\" (UniqueName: \"kubernetes.io/projected/b6ee4d66-7acd-4f3f-ad2e-4650a1019c1c-kube-api-access-7kpz9\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:20.619842 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:20.619797 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6ee4d66-7acd-4f3f-ad2e-4650a1019c1c-maas-api-service-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:21.347308 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:21.347272 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cdfce9-29893380-gwwf5" event={"ID":"b6ee4d66-7acd-4f3f-ad2e-4650a1019c1c","Type":"ContainerDied","Data":"cb3c30e511d6ee8f64c8893a8d8728c0c9e6246d70ad58ade80ccc055cc25fc6"} Nov 02 07:00:21.347308 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:21.347309 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cdfce9-29893380-gwwf5" Nov 02 07:00:21.347570 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:21.347345 5916 scope.go:122] "RemoveContainer" containerID="4bd6b8899867b00d0a108241b98646d20ebf8ea5b13d72e5bbcc8047f7acb458" Nov 02 07:00:21.514204 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:21.514173 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cdfce9-29893380-gwwf5"] Nov 02 07:00:21.518730 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:21.518700 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cdfce9-29893380-gwwf5"] Nov 02 07:00:21.843819 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:21.843789 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6ee4d66-7acd-4f3f-ad2e-4650a1019c1c" path="/var/lib/kubelet/pods/b6ee4d66-7acd-4f3f-ad2e-4650a1019c1c/volumes" Nov 02 07:00:29.651565 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:29.651531 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a2ba11-5c66894c8c-p472q" podUID="1936eac5-6027-48a9-86ff-12ad457b94d9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 07:00:29.652012 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:29.651578 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-a2ba11-58967fcd4f-qpggq" podUID="610f1b35-d018-4931-815c-98bfc24ebfc5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 07:00:29.662748 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:29.662712 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-44p69"] Nov 02 07:00:29.663059 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:29.663030 5916 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-44p69" podUID="fccb2c65-8f82-4808-bb0f-7d9333d8932a" containerName="istio-proxy" containerID="cri-o://2cfc1c5010d443e621858a1bb05a61b188a5c1e5e85ae7718eed002e73038e0b" gracePeriod=30 Nov 02 07:00:29.711093 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:29.711058 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-r9cmb"] Nov 02 07:00:29.711350 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:29.711301 5916 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-r9cmb" podUID="d018c073-2cde-4cbc-8209-a2d788777d2a" containerName="istio-proxy" containerID="cri-o://e708b1b42904654b3c111a2692d66305deb1c742f3a17fa3dbd74235b3bd878a" gracePeriod=30 Nov 02 07:00:29.714426 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:29.714392 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bkklk"] Nov 02 07:00:29.714606 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:29.714567 5916 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bkklk" podUID="f1173406-c493-422e-a67a-24a9580c9c0b" containerName="istio-proxy" containerID="cri-o://73b632741625a10d75b7efd1406f773f337d3b4ddad764a3f38b753c2259cd2f" gracePeriod=30 Nov 02 07:00:29.759845 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:29.759810 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-gkz2f"] Nov 02 07:00:29.760072 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:29.760048 5916 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-gkz2f" podUID="bc31d778-d117-4ca5-89e6-550d04804cd1" containerName="istio-proxy" containerID="cri-o://5ebf6a023e04a1543efb593d498ddd475ad4ae4b9e09edda81a4a03d0d1d13d8" gracePeriod=30 Nov 02 07:00:29.775703 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:29.775670 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-581608-666ff49d8c-2466x" podUID="63b6b2e6-166d-4571-b148-6c504875697c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 07:00:30.997019 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:30.996929 5916 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-r9cmb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 07:00:30.997019 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:30.996995 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-r9cmb" podUID="d018c073-2cde-4cbc-8209-a2d788777d2a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 07:00:31.917376 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:31.917333 5916 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-gkz2f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 07:00:31.917543 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:31.917388 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-gkz2f" podUID="bc31d778-d117-4ca5-89e6-550d04804cd1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 07:00:33.600517 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:33.600489 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-581608-666ff49d8c-2466x" Nov 02 07:00:33.662261 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:33.662234 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63b6b2e6-166d-4571-b148-6c504875697c-kube-api-access-m9b7j\" (UniqueName: \"kubernetes.io/projected/63b6b2e6-166d-4571-b148-6c504875697c-kube-api-access-m9b7j\") pod \"63b6b2e6-166d-4571-b148-6c504875697c\" (UID: \"63b6b2e6-166d-4571-b148-6c504875697c\") " Nov 02 07:00:33.662447 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:33.662333 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63b6b2e6-166d-4571-b148-6c504875697c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63b6b2e6-166d-4571-b148-6c504875697c-plugins-config-volume\") pod \"63b6b2e6-166d-4571-b148-6c504875697c\" (UID: \"63b6b2e6-166d-4571-b148-6c504875697c\") " Nov 02 07:00:33.662748 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:33.662717 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63b6b2e6-166d-4571-b148-6c504875697c-plugins-config-volume" pod "63b6b2e6-166d-4571-b148-6c504875697c" (UID: "63b6b2e6-166d-4571-b148-6c504875697c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:00:33.664376 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:33.664346 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63b6b2e6-166d-4571-b148-6c504875697c-kube-api-access-m9b7j" pod "63b6b2e6-166d-4571-b148-6c504875697c" (UID: "63b6b2e6-166d-4571-b148-6c504875697c"). InnerVolumeSpecName "kube-api-access-m9b7j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:00:33.763856 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:33.763821 5916 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63b6b2e6-166d-4571-b148-6c504875697c-plugins-config-volume\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:33.763856 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:33.763854 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9b7j\" (UniqueName: \"kubernetes.io/projected/63b6b2e6-166d-4571-b148-6c504875697c-kube-api-access-m9b7j\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:34.438351 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:34.438313 5916 generic.go:356] "Generic (PLEG): container finished" podID="63b6b2e6-166d-4571-b148-6c504875697c" containerID="e8049b156c9b93ea0f393748b64b6b2c992316b3b9b5488443e7c1ba0d70eb59" exitCode=0 Nov 02 07:00:34.438650 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:34.438378 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-581608-666ff49d8c-2466x" Nov 02 07:00:34.438650 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:34.438403 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-581608-666ff49d8c-2466x" event={"ID":"63b6b2e6-166d-4571-b148-6c504875697c","Type":"ContainerDied","Data":"e8049b156c9b93ea0f393748b64b6b2c992316b3b9b5488443e7c1ba0d70eb59"} Nov 02 07:00:34.438650 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:34.438498 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-581608-666ff49d8c-2466x" event={"ID":"63b6b2e6-166d-4571-b148-6c504875697c","Type":"ContainerDied","Data":"159cb26a898a0d52f3f15a745f7a82cd824c789121cb80c462a98e3d187bd449"} Nov 02 07:00:34.438650 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:34.438529 5916 scope.go:122] "RemoveContainer" containerID="e8049b156c9b93ea0f393748b64b6b2c992316b3b9b5488443e7c1ba0d70eb59" Nov 02 07:00:34.449421 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:34.449401 5916 scope.go:122] "RemoveContainer" containerID="e8049b156c9b93ea0f393748b64b6b2c992316b3b9b5488443e7c1ba0d70eb59" Nov 02 07:00:34.449721 ip-10-0-130-93 kubenswrapper[5916]: E1102 07:00:34.449701 5916 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8049b156c9b93ea0f393748b64b6b2c992316b3b9b5488443e7c1ba0d70eb59\": container with ID starting with e8049b156c9b93ea0f393748b64b6b2c992316b3b9b5488443e7c1ba0d70eb59 not found: ID does not exist" containerID="e8049b156c9b93ea0f393748b64b6b2c992316b3b9b5488443e7c1ba0d70eb59" Nov 02 07:00:34.449805 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:34.449728 5916 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8049b156c9b93ea0f393748b64b6b2c992316b3b9b5488443e7c1ba0d70eb59"} err="failed to get container status \"e8049b156c9b93ea0f393748b64b6b2c992316b3b9b5488443e7c1ba0d70eb59\": rpc error: code = NotFound desc = could not find container \"e8049b156c9b93ea0f393748b64b6b2c992316b3b9b5488443e7c1ba0d70eb59\": container with ID starting with e8049b156c9b93ea0f393748b64b6b2c992316b3b9b5488443e7c1ba0d70eb59 not found: ID does not exist" Nov 02 07:00:34.603331 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:34.603300 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-581608-666ff49d8c-2466x"] Nov 02 07:00:34.653355 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:34.653316 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-581608-666ff49d8c-2466x"] Nov 02 07:00:34.666645 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:34.666599 5916 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-bkklk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 07:00:34.666801 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:34.666673 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bkklk" podUID="f1173406-c493-422e-a67a-24a9580c9c0b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 07:00:35.014794 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.014764 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-44p69" Nov 02 07:00:35.064338 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.064250 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-r9cmb" Nov 02 07:00:35.082008 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.081977 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fccb2c65-8f82-4808-bb0f-7d9333d8932a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fccb2c65-8f82-4808-bb0f-7d9333d8932a-istio-envoy\") pod \"fccb2c65-8f82-4808-bb0f-7d9333d8932a\" (UID: \"fccb2c65-8f82-4808-bb0f-7d9333d8932a\") " Nov 02 07:00:35.082150 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.082067 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fccb2c65-8f82-4808-bb0f-7d9333d8932a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fccb2c65-8f82-4808-bb0f-7d9333d8932a-workload-certs\") pod \"fccb2c65-8f82-4808-bb0f-7d9333d8932a\" (UID: \"fccb2c65-8f82-4808-bb0f-7d9333d8932a\") " Nov 02 07:00:35.082150 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.082107 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fccb2c65-8f82-4808-bb0f-7d9333d8932a-istio-token\" (UniqueName: \"kubernetes.io/projected/fccb2c65-8f82-4808-bb0f-7d9333d8932a-istio-token\") pod \"fccb2c65-8f82-4808-bb0f-7d9333d8932a\" (UID: \"fccb2c65-8f82-4808-bb0f-7d9333d8932a\") " Nov 02 07:00:35.082150 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.082142 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fccb2c65-8f82-4808-bb0f-7d9333d8932a-kube-api-access-7mhph\" (UniqueName: \"kubernetes.io/projected/fccb2c65-8f82-4808-bb0f-7d9333d8932a-kube-api-access-7mhph\") pod \"fccb2c65-8f82-4808-bb0f-7d9333d8932a\" (UID: \"fccb2c65-8f82-4808-bb0f-7d9333d8932a\") " Nov 02 07:00:35.082417 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.082191 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fccb2c65-8f82-4808-bb0f-7d9333d8932a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fccb2c65-8f82-4808-bb0f-7d9333d8932a-istio-podinfo\") pod \"fccb2c65-8f82-4808-bb0f-7d9333d8932a\" (UID: \"fccb2c65-8f82-4808-bb0f-7d9333d8932a\") " Nov 02 07:00:35.082417 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.082225 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fccb2c65-8f82-4808-bb0f-7d9333d8932a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fccb2c65-8f82-4808-bb0f-7d9333d8932a-istio-data\") pod \"fccb2c65-8f82-4808-bb0f-7d9333d8932a\" (UID: \"fccb2c65-8f82-4808-bb0f-7d9333d8932a\") " Nov 02 07:00:35.082417 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.082272 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fccb2c65-8f82-4808-bb0f-7d9333d8932a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fccb2c65-8f82-4808-bb0f-7d9333d8932a-workload-socket\") pod \"fccb2c65-8f82-4808-bb0f-7d9333d8932a\" (UID: \"fccb2c65-8f82-4808-bb0f-7d9333d8932a\") " Nov 02 07:00:35.082417 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.082354 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fccb2c65-8f82-4808-bb0f-7d9333d8932a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fccb2c65-8f82-4808-bb0f-7d9333d8932a-istiod-ca-cert\") pod \"fccb2c65-8f82-4808-bb0f-7d9333d8932a\" (UID: \"fccb2c65-8f82-4808-bb0f-7d9333d8932a\") " Nov 02 07:00:35.083287 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.082721 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fccb2c65-8f82-4808-bb0f-7d9333d8932a-workload-certs" pod "fccb2c65-8f82-4808-bb0f-7d9333d8932a" (UID: "fccb2c65-8f82-4808-bb0f-7d9333d8932a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:00:35.083287 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.082745 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fccb2c65-8f82-4808-bb0f-7d9333d8932a-istiod-ca-cert" pod "fccb2c65-8f82-4808-bb0f-7d9333d8932a" (UID: "fccb2c65-8f82-4808-bb0f-7d9333d8932a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:00:35.083287 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.082986 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fccb2c65-8f82-4808-bb0f-7d9333d8932a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fccb2c65-8f82-4808-bb0f-7d9333d8932a-credential-socket\") pod \"fccb2c65-8f82-4808-bb0f-7d9333d8932a\" (UID: \"fccb2c65-8f82-4808-bb0f-7d9333d8932a\") " Nov 02 07:00:35.083287 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.083013 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fccb2c65-8f82-4808-bb0f-7d9333d8932a-workload-socket" pod "fccb2c65-8f82-4808-bb0f-7d9333d8932a" (UID: "fccb2c65-8f82-4808-bb0f-7d9333d8932a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:00:35.083287 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.083204 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fccb2c65-8f82-4808-bb0f-7d9333d8932a-istio-data" pod "fccb2c65-8f82-4808-bb0f-7d9333d8932a" (UID: "fccb2c65-8f82-4808-bb0f-7d9333d8932a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:00:35.083287 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.083267 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fccb2c65-8f82-4808-bb0f-7d9333d8932a-credential-socket" pod "fccb2c65-8f82-4808-bb0f-7d9333d8932a" (UID: "fccb2c65-8f82-4808-bb0f-7d9333d8932a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:00:35.083610 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.083592 5916 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fccb2c65-8f82-4808-bb0f-7d9333d8932a-istio-data\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:35.083649 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.083617 5916 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fccb2c65-8f82-4808-bb0f-7d9333d8932a-workload-socket\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:35.083649 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.083633 5916 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fccb2c65-8f82-4808-bb0f-7d9333d8932a-istiod-ca-cert\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:35.083725 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.083648 5916 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fccb2c65-8f82-4808-bb0f-7d9333d8932a-credential-socket\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:35.083725 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.083662 5916 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fccb2c65-8f82-4808-bb0f-7d9333d8932a-workload-certs\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:35.085121 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.085099 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fccb2c65-8f82-4808-bb0f-7d9333d8932a-istio-podinfo" pod "fccb2c65-8f82-4808-bb0f-7d9333d8932a" (UID: "fccb2c65-8f82-4808-bb0f-7d9333d8932a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 07:00:35.085390 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.085360 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fccb2c65-8f82-4808-bb0f-7d9333d8932a-kube-api-access-7mhph" pod "fccb2c65-8f82-4808-bb0f-7d9333d8932a" (UID: "fccb2c65-8f82-4808-bb0f-7d9333d8932a"). InnerVolumeSpecName "kube-api-access-7mhph". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:00:35.085512 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.085489 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fccb2c65-8f82-4808-bb0f-7d9333d8932a-istio-envoy" pod "fccb2c65-8f82-4808-bb0f-7d9333d8932a" (UID: "fccb2c65-8f82-4808-bb0f-7d9333d8932a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:00:35.085716 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.085688 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fccb2c65-8f82-4808-bb0f-7d9333d8932a-istio-token" pod "fccb2c65-8f82-4808-bb0f-7d9333d8932a" (UID: "fccb2c65-8f82-4808-bb0f-7d9333d8932a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:00:35.094459 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.094440 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bkklk" Nov 02 07:00:35.096720 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.096704 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-gkz2f" Nov 02 07:00:35.184751 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.184717 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d018c073-2cde-4cbc-8209-a2d788777d2a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d018c073-2cde-4cbc-8209-a2d788777d2a-credential-socket\") pod \"d018c073-2cde-4cbc-8209-a2d788777d2a\" (UID: \"d018c073-2cde-4cbc-8209-a2d788777d2a\") " Nov 02 07:00:35.184949 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.184773 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d018c073-2cde-4cbc-8209-a2d788777d2a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d018c073-2cde-4cbc-8209-a2d788777d2a-istiod-ca-cert\") pod \"d018c073-2cde-4cbc-8209-a2d788777d2a\" (UID: \"d018c073-2cde-4cbc-8209-a2d788777d2a\") " Nov 02 07:00:35.184949 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.184796 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bc31d778-d117-4ca5-89e6-550d04804cd1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc31d778-d117-4ca5-89e6-550d04804cd1-istio-podinfo\") pod \"bc31d778-d117-4ca5-89e6-550d04804cd1\" (UID: \"bc31d778-d117-4ca5-89e6-550d04804cd1\") " Nov 02 07:00:35.184949 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.184822 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1173406-c493-422e-a67a-24a9580c9c0b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1173406-c493-422e-a67a-24a9580c9c0b-istiod-ca-cert\") pod \"f1173406-c493-422e-a67a-24a9580c9c0b\" (UID: \"f1173406-c493-422e-a67a-24a9580c9c0b\") " Nov 02 07:00:35.184949 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.184854 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d018c073-2cde-4cbc-8209-a2d788777d2a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d018c073-2cde-4cbc-8209-a2d788777d2a-istio-envoy\") pod \"d018c073-2cde-4cbc-8209-a2d788777d2a\" (UID: \"d018c073-2cde-4cbc-8209-a2d788777d2a\") " Nov 02 07:00:35.184949 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.184871 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1173406-c493-422e-a67a-24a9580c9c0b-istio-token\" (UniqueName: \"kubernetes.io/projected/f1173406-c493-422e-a67a-24a9580c9c0b-istio-token\") pod \"f1173406-c493-422e-a67a-24a9580c9c0b\" (UID: \"f1173406-c493-422e-a67a-24a9580c9c0b\") " Nov 02 07:00:35.184949 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.184893 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f1173406-c493-422e-a67a-24a9580c9c0b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1173406-c493-422e-a67a-24a9580c9c0b-istio-podinfo\") pod \"f1173406-c493-422e-a67a-24a9580c9c0b\" (UID: \"f1173406-c493-422e-a67a-24a9580c9c0b\") " Nov 02 07:00:35.184949 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.184939 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d018c073-2cde-4cbc-8209-a2d788777d2a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d018c073-2cde-4cbc-8209-a2d788777d2a-istio-data\") pod \"d018c073-2cde-4cbc-8209-a2d788777d2a\" (UID: \"d018c073-2cde-4cbc-8209-a2d788777d2a\") " Nov 02 07:00:35.185349 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.184972 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d018c073-2cde-4cbc-8209-a2d788777d2a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d018c073-2cde-4cbc-8209-a2d788777d2a-workload-certs\") pod \"d018c073-2cde-4cbc-8209-a2d788777d2a\" (UID: \"d018c073-2cde-4cbc-8209-a2d788777d2a\") " Nov 02 07:00:35.185349 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.185012 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d018c073-2cde-4cbc-8209-a2d788777d2a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d018c073-2cde-4cbc-8209-a2d788777d2a-workload-socket\") pod \"d018c073-2cde-4cbc-8209-a2d788777d2a\" (UID: \"d018c073-2cde-4cbc-8209-a2d788777d2a\") " Nov 02 07:00:35.185349 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.185042 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc31d778-d117-4ca5-89e6-550d04804cd1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc31d778-d117-4ca5-89e6-550d04804cd1-workload-certs\") pod \"bc31d778-d117-4ca5-89e6-550d04804cd1\" (UID: \"bc31d778-d117-4ca5-89e6-550d04804cd1\") " Nov 02 07:00:35.185349 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.185084 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d018c073-2cde-4cbc-8209-a2d788777d2a-istio-token\" (UniqueName: \"kubernetes.io/projected/d018c073-2cde-4cbc-8209-a2d788777d2a-istio-token\") pod \"d018c073-2cde-4cbc-8209-a2d788777d2a\" (UID: \"d018c073-2cde-4cbc-8209-a2d788777d2a\") " Nov 02 07:00:35.185349 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.185088 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d018c073-2cde-4cbc-8209-a2d788777d2a-credential-socket" pod "d018c073-2cde-4cbc-8209-a2d788777d2a" (UID: "d018c073-2cde-4cbc-8209-a2d788777d2a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:00:35.185349 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.185109 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc31d778-d117-4ca5-89e6-550d04804cd1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc31d778-d117-4ca5-89e6-550d04804cd1-istio-data\") pod \"bc31d778-d117-4ca5-89e6-550d04804cd1\" (UID: \"bc31d778-d117-4ca5-89e6-550d04804cd1\") " Nov 02 07:00:35.185349 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.185195 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1173406-c493-422e-a67a-24a9580c9c0b-kube-api-access-4skqf\" (UniqueName: \"kubernetes.io/projected/f1173406-c493-422e-a67a-24a9580c9c0b-kube-api-access-4skqf\") pod \"f1173406-c493-422e-a67a-24a9580c9c0b\" (UID: \"f1173406-c493-422e-a67a-24a9580c9c0b\") " Nov 02 07:00:35.185720 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.185245 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1173406-c493-422e-a67a-24a9580c9c0b-istiod-ca-cert" pod "f1173406-c493-422e-a67a-24a9580c9c0b" (UID: "f1173406-c493-422e-a67a-24a9580c9c0b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:00:35.185720 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.185362 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d018c073-2cde-4cbc-8209-a2d788777d2a-istiod-ca-cert" pod "d018c073-2cde-4cbc-8209-a2d788777d2a" (UID: "d018c073-2cde-4cbc-8209-a2d788777d2a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:00:35.185720 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.185334 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc31d778-d117-4ca5-89e6-550d04804cd1-istio-data" pod "bc31d778-d117-4ca5-89e6-550d04804cd1" (UID: "bc31d778-d117-4ca5-89e6-550d04804cd1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:00:35.185720 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.185234 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc31d778-d117-4ca5-89e6-550d04804cd1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc31d778-d117-4ca5-89e6-550d04804cd1-istio-envoy\") pod \"bc31d778-d117-4ca5-89e6-550d04804cd1\" (UID: \"bc31d778-d117-4ca5-89e6-550d04804cd1\") " Nov 02 07:00:35.185720 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.185525 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1173406-c493-422e-a67a-24a9580c9c0b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1173406-c493-422e-a67a-24a9580c9c0b-istio-data\") pod \"f1173406-c493-422e-a67a-24a9580c9c0b\" (UID: \"f1173406-c493-422e-a67a-24a9580c9c0b\") " Nov 02 07:00:35.185720 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.185573 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1173406-c493-422e-a67a-24a9580c9c0b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1173406-c493-422e-a67a-24a9580c9c0b-workload-certs\") pod \"f1173406-c493-422e-a67a-24a9580c9c0b\" (UID: \"f1173406-c493-422e-a67a-24a9580c9c0b\") " Nov 02 07:00:35.185720 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.185610 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc31d778-d117-4ca5-89e6-550d04804cd1-istio-token\" (UniqueName: \"kubernetes.io/projected/bc31d778-d117-4ca5-89e6-550d04804cd1-istio-token\") pod \"bc31d778-d117-4ca5-89e6-550d04804cd1\" (UID: \"bc31d778-d117-4ca5-89e6-550d04804cd1\") " Nov 02 07:00:35.185720 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.185665 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1173406-c493-422e-a67a-24a9580c9c0b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1173406-c493-422e-a67a-24a9580c9c0b-credential-socket\") pod \"f1173406-c493-422e-a67a-24a9580c9c0b\" (UID: \"f1173406-c493-422e-a67a-24a9580c9c0b\") " Nov 02 07:00:35.185720 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.185693 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1173406-c493-422e-a67a-24a9580c9c0b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1173406-c493-422e-a67a-24a9580c9c0b-workload-socket\") pod \"f1173406-c493-422e-a67a-24a9580c9c0b\" (UID: \"f1173406-c493-422e-a67a-24a9580c9c0b\") " Nov 02 07:00:35.186592 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.185730 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc31d778-d117-4ca5-89e6-550d04804cd1-kube-api-access-lcw5f\" (UniqueName: \"kubernetes.io/projected/bc31d778-d117-4ca5-89e6-550d04804cd1-kube-api-access-lcw5f\") pod \"bc31d778-d117-4ca5-89e6-550d04804cd1\" (UID: \"bc31d778-d117-4ca5-89e6-550d04804cd1\") " Nov 02 07:00:35.186592 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.185774 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc31d778-d117-4ca5-89e6-550d04804cd1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc31d778-d117-4ca5-89e6-550d04804cd1-credential-socket\") pod \"bc31d778-d117-4ca5-89e6-550d04804cd1\" (UID: \"bc31d778-d117-4ca5-89e6-550d04804cd1\") " Nov 02 07:00:35.186592 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.185810 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc31d778-d117-4ca5-89e6-550d04804cd1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc31d778-d117-4ca5-89e6-550d04804cd1-istiod-ca-cert\") pod \"bc31d778-d117-4ca5-89e6-550d04804cd1\" (UID: \"bc31d778-d117-4ca5-89e6-550d04804cd1\") " Nov 02 07:00:35.186592 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.185844 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1173406-c493-422e-a67a-24a9580c9c0b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1173406-c493-422e-a67a-24a9580c9c0b-istio-envoy\") pod \"f1173406-c493-422e-a67a-24a9580c9c0b\" (UID: \"f1173406-c493-422e-a67a-24a9580c9c0b\") " Nov 02 07:00:35.186592 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.185876 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc31d778-d117-4ca5-89e6-550d04804cd1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc31d778-d117-4ca5-89e6-550d04804cd1-workload-socket\") pod \"bc31d778-d117-4ca5-89e6-550d04804cd1\" (UID: \"bc31d778-d117-4ca5-89e6-550d04804cd1\") " Nov 02 07:00:35.186592 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.185992 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d018c073-2cde-4cbc-8209-a2d788777d2a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d018c073-2cde-4cbc-8209-a2d788777d2a-istio-podinfo\") pod \"d018c073-2cde-4cbc-8209-a2d788777d2a\" (UID: \"d018c073-2cde-4cbc-8209-a2d788777d2a\") " Nov 02 07:00:35.186592 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.186034 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d018c073-2cde-4cbc-8209-a2d788777d2a-kube-api-access-spr4h\" (UniqueName: \"kubernetes.io/projected/d018c073-2cde-4cbc-8209-a2d788777d2a-kube-api-access-spr4h\") pod \"d018c073-2cde-4cbc-8209-a2d788777d2a\" (UID: \"d018c073-2cde-4cbc-8209-a2d788777d2a\") " Nov 02 07:00:35.186592 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.186488 5916 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fccb2c65-8f82-4808-bb0f-7d9333d8932a-istio-token\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:35.186592 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.186513 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mhph\" (UniqueName: \"kubernetes.io/projected/fccb2c65-8f82-4808-bb0f-7d9333d8932a-kube-api-access-7mhph\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:35.186592 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.186529 5916 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fccb2c65-8f82-4808-bb0f-7d9333d8932a-istio-podinfo\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:35.186592 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.186543 5916 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d018c073-2cde-4cbc-8209-a2d788777d2a-credential-socket\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:35.186592 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.186560 5916 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d018c073-2cde-4cbc-8209-a2d788777d2a-istiod-ca-cert\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:35.186592 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.186574 5916 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1173406-c493-422e-a67a-24a9580c9c0b-istiod-ca-cert\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:35.186592 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.186588 5916 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc31d778-d117-4ca5-89e6-550d04804cd1-istio-data\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:35.186592 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.186600 5916 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fccb2c65-8f82-4808-bb0f-7d9333d8932a-istio-envoy\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:35.189634 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.189448 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d018c073-2cde-4cbc-8209-a2d788777d2a-istio-envoy" pod "d018c073-2cde-4cbc-8209-a2d788777d2a" (UID: "d018c073-2cde-4cbc-8209-a2d788777d2a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:00:35.189634 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.189569 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc31d778-d117-4ca5-89e6-550d04804cd1-kube-api-access-lcw5f" pod "bc31d778-d117-4ca5-89e6-550d04804cd1" (UID: "bc31d778-d117-4ca5-89e6-550d04804cd1"). InnerVolumeSpecName "kube-api-access-lcw5f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:00:35.189821 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.189606 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bc31d778-d117-4ca5-89e6-550d04804cd1-istio-podinfo" pod "bc31d778-d117-4ca5-89e6-550d04804cd1" (UID: "bc31d778-d117-4ca5-89e6-550d04804cd1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 07:00:35.189821 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.189625 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1173406-c493-422e-a67a-24a9580c9c0b-kube-api-access-4skqf" pod "f1173406-c493-422e-a67a-24a9580c9c0b" (UID: "f1173406-c493-422e-a67a-24a9580c9c0b"). InnerVolumeSpecName "kube-api-access-4skqf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:00:35.189821 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.189643 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1173406-c493-422e-a67a-24a9580c9c0b-istio-token" pod "f1173406-c493-422e-a67a-24a9580c9c0b" (UID: "f1173406-c493-422e-a67a-24a9580c9c0b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:00:35.189821 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.189665 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1173406-c493-422e-a67a-24a9580c9c0b-istio-envoy" pod "f1173406-c493-422e-a67a-24a9580c9c0b" (UID: "f1173406-c493-422e-a67a-24a9580c9c0b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:00:35.190098 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.189846 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d018c073-2cde-4cbc-8209-a2d788777d2a-workload-socket" pod "d018c073-2cde-4cbc-8209-a2d788777d2a" (UID: "d018c073-2cde-4cbc-8209-a2d788777d2a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:00:35.190098 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.189934 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d018c073-2cde-4cbc-8209-a2d788777d2a-kube-api-access-spr4h" pod "d018c073-2cde-4cbc-8209-a2d788777d2a" (UID: "d018c073-2cde-4cbc-8209-a2d788777d2a"). InnerVolumeSpecName "kube-api-access-spr4h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:00:35.190098 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.189966 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1173406-c493-422e-a67a-24a9580c9c0b-istio-data" pod "f1173406-c493-422e-a67a-24a9580c9c0b" (UID: "f1173406-c493-422e-a67a-24a9580c9c0b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:00:35.190098 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.189977 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc31d778-d117-4ca5-89e6-550d04804cd1-credential-socket" pod "bc31d778-d117-4ca5-89e6-550d04804cd1" (UID: "bc31d778-d117-4ca5-89e6-550d04804cd1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:00:35.190353 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.190122 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc31d778-d117-4ca5-89e6-550d04804cd1-workload-certs" pod "bc31d778-d117-4ca5-89e6-550d04804cd1" (UID: "bc31d778-d117-4ca5-89e6-550d04804cd1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:00:35.190353 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.190158 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc31d778-d117-4ca5-89e6-550d04804cd1-istiod-ca-cert" pod "bc31d778-d117-4ca5-89e6-550d04804cd1" (UID: "bc31d778-d117-4ca5-89e6-550d04804cd1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:00:35.190353 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.190164 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc31d778-d117-4ca5-89e6-550d04804cd1-workload-socket" pod "bc31d778-d117-4ca5-89e6-550d04804cd1" (UID: "bc31d778-d117-4ca5-89e6-550d04804cd1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:00:35.190353 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.190192 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1173406-c493-422e-a67a-24a9580c9c0b-workload-certs" pod "f1173406-c493-422e-a67a-24a9580c9c0b" (UID: "f1173406-c493-422e-a67a-24a9580c9c0b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:00:35.190353 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.190208 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d018c073-2cde-4cbc-8209-a2d788777d2a-istio-data" pod "d018c073-2cde-4cbc-8209-a2d788777d2a" (UID: "d018c073-2cde-4cbc-8209-a2d788777d2a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:00:35.190671 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.190405 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1173406-c493-422e-a67a-24a9580c9c0b-credential-socket" pod "f1173406-c493-422e-a67a-24a9580c9c0b" (UID: "f1173406-c493-422e-a67a-24a9580c9c0b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:00:35.190671 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.190416 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1173406-c493-422e-a67a-24a9580c9c0b-workload-socket" pod "f1173406-c493-422e-a67a-24a9580c9c0b" (UID: "f1173406-c493-422e-a67a-24a9580c9c0b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:00:35.190671 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.190475 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d018c073-2cde-4cbc-8209-a2d788777d2a-workload-certs" pod "d018c073-2cde-4cbc-8209-a2d788777d2a" (UID: "d018c073-2cde-4cbc-8209-a2d788777d2a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:00:35.191131 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.190791 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc31d778-d117-4ca5-89e6-550d04804cd1-istio-envoy" pod "bc31d778-d117-4ca5-89e6-550d04804cd1" (UID: "bc31d778-d117-4ca5-89e6-550d04804cd1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:00:35.192068 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.192043 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc31d778-d117-4ca5-89e6-550d04804cd1-istio-token" pod "bc31d778-d117-4ca5-89e6-550d04804cd1" (UID: "bc31d778-d117-4ca5-89e6-550d04804cd1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:00:35.192339 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.192324 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d018c073-2cde-4cbc-8209-a2d788777d2a-istio-token" pod "d018c073-2cde-4cbc-8209-a2d788777d2a" (UID: "d018c073-2cde-4cbc-8209-a2d788777d2a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:00:35.192445 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.192428 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f1173406-c493-422e-a67a-24a9580c9c0b-istio-podinfo" pod "f1173406-c493-422e-a67a-24a9580c9c0b" (UID: "f1173406-c493-422e-a67a-24a9580c9c0b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 07:00:35.192493 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.192432 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d018c073-2cde-4cbc-8209-a2d788777d2a-istio-podinfo" pod "d018c073-2cde-4cbc-8209-a2d788777d2a" (UID: "d018c073-2cde-4cbc-8209-a2d788777d2a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 07:00:35.288114 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.288038 5916 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d018c073-2cde-4cbc-8209-a2d788777d2a-istio-token\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:35.288114 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.288076 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4skqf\" (UniqueName: \"kubernetes.io/projected/f1173406-c493-422e-a67a-24a9580c9c0b-kube-api-access-4skqf\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:35.288114 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.288093 5916 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc31d778-d117-4ca5-89e6-550d04804cd1-istio-envoy\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:35.288114 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.288106 5916 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1173406-c493-422e-a67a-24a9580c9c0b-istio-data\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:35.288406 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.288119 5916 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1173406-c493-422e-a67a-24a9580c9c0b-workload-certs\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:35.288406 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.288132 5916 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc31d778-d117-4ca5-89e6-550d04804cd1-istio-token\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:35.288406 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.288146 5916 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1173406-c493-422e-a67a-24a9580c9c0b-credential-socket\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:35.288406 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.288159 5916 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1173406-c493-422e-a67a-24a9580c9c0b-workload-socket\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:35.288406 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.288173 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lcw5f\" (UniqueName: \"kubernetes.io/projected/bc31d778-d117-4ca5-89e6-550d04804cd1-kube-api-access-lcw5f\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:35.288406 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.288187 5916 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc31d778-d117-4ca5-89e6-550d04804cd1-credential-socket\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:35.288406 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.288200 5916 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc31d778-d117-4ca5-89e6-550d04804cd1-istiod-ca-cert\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:35.288406 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.288215 5916 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1173406-c493-422e-a67a-24a9580c9c0b-istio-envoy\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:35.288406 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.288227 5916 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc31d778-d117-4ca5-89e6-550d04804cd1-workload-socket\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:35.288406 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.288241 5916 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d018c073-2cde-4cbc-8209-a2d788777d2a-istio-podinfo\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:35.288406 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.288254 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-spr4h\" (UniqueName: \"kubernetes.io/projected/d018c073-2cde-4cbc-8209-a2d788777d2a-kube-api-access-spr4h\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:35.288406 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.288268 5916 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc31d778-d117-4ca5-89e6-550d04804cd1-istio-podinfo\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:35.288406 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.288279 5916 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d018c073-2cde-4cbc-8209-a2d788777d2a-istio-envoy\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:35.288406 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.288293 5916 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1173406-c493-422e-a67a-24a9580c9c0b-istio-token\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:35.288406 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.288306 5916 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1173406-c493-422e-a67a-24a9580c9c0b-istio-podinfo\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:35.288406 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.288318 5916 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d018c073-2cde-4cbc-8209-a2d788777d2a-istio-data\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:35.288406 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.288333 5916 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d018c073-2cde-4cbc-8209-a2d788777d2a-workload-certs\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:35.288406 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.288346 5916 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d018c073-2cde-4cbc-8209-a2d788777d2a-workload-socket\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:35.288406 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.288359 5916 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc31d778-d117-4ca5-89e6-550d04804cd1-workload-certs\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:35.447037 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.446998 5916 generic.go:356] "Generic (PLEG): container finished" podID="f1173406-c493-422e-a67a-24a9580c9c0b" containerID="73b632741625a10d75b7efd1406f773f337d3b4ddad764a3f38b753c2259cd2f" exitCode=0 Nov 02 07:00:35.447037 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.447043 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bkklk" Nov 02 07:00:35.447310 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.447080 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bkklk" event={"ID":"f1173406-c493-422e-a67a-24a9580c9c0b","Type":"ContainerDied","Data":"73b632741625a10d75b7efd1406f773f337d3b4ddad764a3f38b753c2259cd2f"} Nov 02 07:00:35.447310 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.447136 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bkklk" event={"ID":"f1173406-c493-422e-a67a-24a9580c9c0b","Type":"ContainerDied","Data":"6b50126c3eea3c0655146190fff2135b23cd5f2dd8d0fc6410f3d0d1a40b8b52"} Nov 02 07:00:35.447310 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.447165 5916 scope.go:122] "RemoveContainer" containerID="73b632741625a10d75b7efd1406f773f337d3b4ddad764a3f38b753c2259cd2f" Nov 02 07:00:35.449477 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.449438 5916 generic.go:356] "Generic (PLEG): container finished" podID="fccb2c65-8f82-4808-bb0f-7d9333d8932a" containerID="2cfc1c5010d443e621858a1bb05a61b188a5c1e5e85ae7718eed002e73038e0b" exitCode=0 Nov 02 07:00:35.449611 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.449491 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-44p69" Nov 02 07:00:35.449611 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.449558 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-44p69" event={"ID":"fccb2c65-8f82-4808-bb0f-7d9333d8932a","Type":"ContainerDied","Data":"2cfc1c5010d443e621858a1bb05a61b188a5c1e5e85ae7718eed002e73038e0b"} Nov 02 07:00:35.449611 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.449596 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-44p69" event={"ID":"fccb2c65-8f82-4808-bb0f-7d9333d8932a","Type":"ContainerDied","Data":"d9677834eac5820f26e5c4eb45c0704a8c06232ea5fdbaf6d3da206207cb5169"} Nov 02 07:00:35.451291 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.451267 5916 generic.go:356] "Generic (PLEG): container finished" podID="d018c073-2cde-4cbc-8209-a2d788777d2a" containerID="e708b1b42904654b3c111a2692d66305deb1c742f3a17fa3dbd74235b3bd878a" exitCode=0 Nov 02 07:00:35.451417 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.451321 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-r9cmb" Nov 02 07:00:35.451495 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.451415 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-r9cmb" event={"ID":"d018c073-2cde-4cbc-8209-a2d788777d2a","Type":"ContainerDied","Data":"e708b1b42904654b3c111a2692d66305deb1c742f3a17fa3dbd74235b3bd878a"} Nov 02 07:00:35.451495 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.451451 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-r9cmb" event={"ID":"d018c073-2cde-4cbc-8209-a2d788777d2a","Type":"ContainerDied","Data":"c24f9ff71ffb135ad9d598ba5e15d7e08ced816daefef7f7df59adedfa680e43"} Nov 02 07:00:35.456949 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.456417 5916 generic.go:356] "Generic (PLEG): container finished" podID="bc31d778-d117-4ca5-89e6-550d04804cd1" containerID="5ebf6a023e04a1543efb593d498ddd475ad4ae4b9e09edda81a4a03d0d1d13d8" exitCode=0 Nov 02 07:00:35.456949 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.456466 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-gkz2f" event={"ID":"bc31d778-d117-4ca5-89e6-550d04804cd1","Type":"ContainerDied","Data":"5ebf6a023e04a1543efb593d498ddd475ad4ae4b9e09edda81a4a03d0d1d13d8"} Nov 02 07:00:35.456949 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.456499 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-gkz2f" event={"ID":"bc31d778-d117-4ca5-89e6-550d04804cd1","Type":"ContainerDied","Data":"e4f8dc887ad0953f328e3747fbfbce791301321ed53a4ca5d1fd7afe39763ee8"} Nov 02 07:00:35.456949 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.456557 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-gkz2f" Nov 02 07:00:35.461251 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.461212 5916 scope.go:122] "RemoveContainer" containerID="73b632741625a10d75b7efd1406f773f337d3b4ddad764a3f38b753c2259cd2f" Nov 02 07:00:35.461499 ip-10-0-130-93 kubenswrapper[5916]: E1102 07:00:35.461473 5916 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73b632741625a10d75b7efd1406f773f337d3b4ddad764a3f38b753c2259cd2f\": container with ID starting with 73b632741625a10d75b7efd1406f773f337d3b4ddad764a3f38b753c2259cd2f not found: ID does not exist" containerID="73b632741625a10d75b7efd1406f773f337d3b4ddad764a3f38b753c2259cd2f" Nov 02 07:00:35.461559 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.461508 5916 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73b632741625a10d75b7efd1406f773f337d3b4ddad764a3f38b753c2259cd2f"} err="failed to get container status \"73b632741625a10d75b7efd1406f773f337d3b4ddad764a3f38b753c2259cd2f\": rpc error: code = NotFound desc = could not find container \"73b632741625a10d75b7efd1406f773f337d3b4ddad764a3f38b753c2259cd2f\": container with ID starting with 73b632741625a10d75b7efd1406f773f337d3b4ddad764a3f38b753c2259cd2f not found: ID does not exist" Nov 02 07:00:35.461559 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.461528 5916 scope.go:122] "RemoveContainer" containerID="2cfc1c5010d443e621858a1bb05a61b188a5c1e5e85ae7718eed002e73038e0b" Nov 02 07:00:35.473842 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.473825 5916 scope.go:122] "RemoveContainer" containerID="0433bc04f4a45475241fe4622439501b58a25a1db75351c76111fbbbbe571ef6" Nov 02 07:00:35.484059 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.484037 5916 scope.go:122] "RemoveContainer" containerID="2cfc1c5010d443e621858a1bb05a61b188a5c1e5e85ae7718eed002e73038e0b" Nov 02 07:00:35.484501 ip-10-0-130-93 kubenswrapper[5916]: E1102 07:00:35.484480 5916 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2cfc1c5010d443e621858a1bb05a61b188a5c1e5e85ae7718eed002e73038e0b\": container with ID starting with 2cfc1c5010d443e621858a1bb05a61b188a5c1e5e85ae7718eed002e73038e0b not found: ID does not exist" containerID="2cfc1c5010d443e621858a1bb05a61b188a5c1e5e85ae7718eed002e73038e0b" Nov 02 07:00:35.484600 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.484511 5916 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2cfc1c5010d443e621858a1bb05a61b188a5c1e5e85ae7718eed002e73038e0b"} err="failed to get container status \"2cfc1c5010d443e621858a1bb05a61b188a5c1e5e85ae7718eed002e73038e0b\": rpc error: code = NotFound desc = could not find container \"2cfc1c5010d443e621858a1bb05a61b188a5c1e5e85ae7718eed002e73038e0b\": container with ID starting with 2cfc1c5010d443e621858a1bb05a61b188a5c1e5e85ae7718eed002e73038e0b not found: ID does not exist" Nov 02 07:00:35.484600 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.484533 5916 scope.go:122] "RemoveContainer" containerID="0433bc04f4a45475241fe4622439501b58a25a1db75351c76111fbbbbe571ef6" Nov 02 07:00:35.485314 ip-10-0-130-93 kubenswrapper[5916]: E1102 07:00:35.485006 5916 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0433bc04f4a45475241fe4622439501b58a25a1db75351c76111fbbbbe571ef6\": container with ID starting with 0433bc04f4a45475241fe4622439501b58a25a1db75351c76111fbbbbe571ef6 not found: ID does not exist" containerID="0433bc04f4a45475241fe4622439501b58a25a1db75351c76111fbbbbe571ef6" Nov 02 07:00:35.485314 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.485053 5916 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0433bc04f4a45475241fe4622439501b58a25a1db75351c76111fbbbbe571ef6"} err="failed to get container status \"0433bc04f4a45475241fe4622439501b58a25a1db75351c76111fbbbbe571ef6\": rpc error: code = NotFound desc = could not find container \"0433bc04f4a45475241fe4622439501b58a25a1db75351c76111fbbbbe571ef6\": container with ID starting with 0433bc04f4a45475241fe4622439501b58a25a1db75351c76111fbbbbe571ef6 not found: ID does not exist" Nov 02 07:00:35.485314 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.485074 5916 scope.go:122] "RemoveContainer" containerID="e708b1b42904654b3c111a2692d66305deb1c742f3a17fa3dbd74235b3bd878a" Nov 02 07:00:35.494333 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.494315 5916 scope.go:122] "RemoveContainer" containerID="e708b1b42904654b3c111a2692d66305deb1c742f3a17fa3dbd74235b3bd878a" Nov 02 07:00:35.494640 ip-10-0-130-93 kubenswrapper[5916]: E1102 07:00:35.494614 5916 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e708b1b42904654b3c111a2692d66305deb1c742f3a17fa3dbd74235b3bd878a\": container with ID starting with e708b1b42904654b3c111a2692d66305deb1c742f3a17fa3dbd74235b3bd878a not found: ID does not exist" containerID="e708b1b42904654b3c111a2692d66305deb1c742f3a17fa3dbd74235b3bd878a" Nov 02 07:00:35.494882 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.494644 5916 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e708b1b42904654b3c111a2692d66305deb1c742f3a17fa3dbd74235b3bd878a"} err="failed to get container status \"e708b1b42904654b3c111a2692d66305deb1c742f3a17fa3dbd74235b3bd878a\": rpc error: code = NotFound desc = could not find container \"e708b1b42904654b3c111a2692d66305deb1c742f3a17fa3dbd74235b3bd878a\": container with ID starting with e708b1b42904654b3c111a2692d66305deb1c742f3a17fa3dbd74235b3bd878a not found: ID does not exist" Nov 02 07:00:35.494882 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.494661 5916 scope.go:122] "RemoveContainer" containerID="5ebf6a023e04a1543efb593d498ddd475ad4ae4b9e09edda81a4a03d0d1d13d8" Nov 02 07:00:35.510549 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.510527 5916 scope.go:122] "RemoveContainer" containerID="5ebf6a023e04a1543efb593d498ddd475ad4ae4b9e09edda81a4a03d0d1d13d8" Nov 02 07:00:35.511319 ip-10-0-130-93 kubenswrapper[5916]: E1102 07:00:35.511283 5916 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ebf6a023e04a1543efb593d498ddd475ad4ae4b9e09edda81a4a03d0d1d13d8\": container with ID starting with 5ebf6a023e04a1543efb593d498ddd475ad4ae4b9e09edda81a4a03d0d1d13d8 not found: ID does not exist" containerID="5ebf6a023e04a1543efb593d498ddd475ad4ae4b9e09edda81a4a03d0d1d13d8" Nov 02 07:00:35.511406 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.511321 5916 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ebf6a023e04a1543efb593d498ddd475ad4ae4b9e09edda81a4a03d0d1d13d8"} err="failed to get container status \"5ebf6a023e04a1543efb593d498ddd475ad4ae4b9e09edda81a4a03d0d1d13d8\": rpc error: code = NotFound desc = could not find container \"5ebf6a023e04a1543efb593d498ddd475ad4ae4b9e09edda81a4a03d0d1d13d8\": container with ID starting with 5ebf6a023e04a1543efb593d498ddd475ad4ae4b9e09edda81a4a03d0d1d13d8 not found: ID does not exist" Nov 02 07:00:35.613674 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.613642 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bkklk"] Nov 02 07:00:35.618505 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.618481 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bkklk"] Nov 02 07:00:35.734061 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.733432 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-r9cmb"] Nov 02 07:00:35.738205 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.738175 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-r9cmb"] Nov 02 07:00:35.804572 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.804491 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a2ba11-58967fcd4f-qpggq" Nov 02 07:00:35.849946 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.847433 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63b6b2e6-166d-4571-b148-6c504875697c" path="/var/lib/kubelet/pods/63b6b2e6-166d-4571-b148-6c504875697c/volumes" Nov 02 07:00:35.849946 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.848130 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d018c073-2cde-4cbc-8209-a2d788777d2a" path="/var/lib/kubelet/pods/d018c073-2cde-4cbc-8209-a2d788777d2a/volumes" Nov 02 07:00:35.849946 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.848723 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1173406-c493-422e-a67a-24a9580c9c0b" path="/var/lib/kubelet/pods/f1173406-c493-422e-a67a-24a9580c9c0b/volumes" Nov 02 07:00:35.869721 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.869659 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-44p69"] Nov 02 07:00:35.873929 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.873887 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-44p69"] Nov 02 07:00:35.894742 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.894722 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/610f1b35-d018-4931-815c-98bfc24ebfc5-kube-api-access-g9l8m\" (UniqueName: \"kubernetes.io/projected/610f1b35-d018-4931-815c-98bfc24ebfc5-kube-api-access-g9l8m\") pod \"610f1b35-d018-4931-815c-98bfc24ebfc5\" (UID: \"610f1b35-d018-4931-815c-98bfc24ebfc5\") " Nov 02 07:00:35.894851 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.894797 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/610f1b35-d018-4931-815c-98bfc24ebfc5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/610f1b35-d018-4931-815c-98bfc24ebfc5-plugins-config-volume\") pod \"610f1b35-d018-4931-815c-98bfc24ebfc5\" (UID: \"610f1b35-d018-4931-815c-98bfc24ebfc5\") " Nov 02 07:00:35.895174 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.895154 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/610f1b35-d018-4931-815c-98bfc24ebfc5-plugins-config-volume" pod "610f1b35-d018-4931-815c-98bfc24ebfc5" (UID: "610f1b35-d018-4931-815c-98bfc24ebfc5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:00:35.896766 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.896742 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/610f1b35-d018-4931-815c-98bfc24ebfc5-kube-api-access-g9l8m" pod "610f1b35-d018-4931-815c-98bfc24ebfc5" (UID: "610f1b35-d018-4931-815c-98bfc24ebfc5"). InnerVolumeSpecName "kube-api-access-g9l8m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:00:35.904469 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.904451 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a2ba11-5c66894c8c-p472q" Nov 02 07:00:35.954201 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.954172 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-gkz2f"] Nov 02 07:00:35.959530 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.959503 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-gkz2f"] Nov 02 07:00:35.995429 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.995406 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1936eac5-6027-48a9-86ff-12ad457b94d9-kube-api-access-r4czx\" (UniqueName: \"kubernetes.io/projected/1936eac5-6027-48a9-86ff-12ad457b94d9-kube-api-access-r4czx\") pod \"1936eac5-6027-48a9-86ff-12ad457b94d9\" (UID: \"1936eac5-6027-48a9-86ff-12ad457b94d9\") " Nov 02 07:00:35.995535 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.995518 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1936eac5-6027-48a9-86ff-12ad457b94d9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1936eac5-6027-48a9-86ff-12ad457b94d9-plugins-config-volume\") pod \"1936eac5-6027-48a9-86ff-12ad457b94d9\" (UID: \"1936eac5-6027-48a9-86ff-12ad457b94d9\") " Nov 02 07:00:35.995711 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.995700 5916 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/610f1b35-d018-4931-815c-98bfc24ebfc5-plugins-config-volume\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:35.995756 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.995714 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9l8m\" (UniqueName: \"kubernetes.io/projected/610f1b35-d018-4931-815c-98bfc24ebfc5-kube-api-access-g9l8m\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:35.995879 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.995861 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1936eac5-6027-48a9-86ff-12ad457b94d9-plugins-config-volume" pod "1936eac5-6027-48a9-86ff-12ad457b94d9" (UID: "1936eac5-6027-48a9-86ff-12ad457b94d9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:00:35.997373 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:35.997355 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1936eac5-6027-48a9-86ff-12ad457b94d9-kube-api-access-r4czx" pod "1936eac5-6027-48a9-86ff-12ad457b94d9" (UID: "1936eac5-6027-48a9-86ff-12ad457b94d9"). InnerVolumeSpecName "kube-api-access-r4czx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:00:36.096634 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:36.096557 5916 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1936eac5-6027-48a9-86ff-12ad457b94d9-plugins-config-volume\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:36.096634 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:36.096591 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4czx\" (UniqueName: \"kubernetes.io/projected/1936eac5-6027-48a9-86ff-12ad457b94d9-kube-api-access-r4czx\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:36.464407 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:36.464371 5916 generic.go:356] "Generic (PLEG): container finished" podID="610f1b35-d018-4931-815c-98bfc24ebfc5" containerID="55dc76fa41f5d3c108cb7aeedd7f5e4862e80c3b1daac12d38b4ccf6e8738080" exitCode=0 Nov 02 07:00:36.464629 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:36.464419 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a2ba11-58967fcd4f-qpggq" Nov 02 07:00:36.464629 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:36.464466 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a2ba11-58967fcd4f-qpggq" event={"ID":"610f1b35-d018-4931-815c-98bfc24ebfc5","Type":"ContainerDied","Data":"55dc76fa41f5d3c108cb7aeedd7f5e4862e80c3b1daac12d38b4ccf6e8738080"} Nov 02 07:00:36.464629 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:36.464517 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a2ba11-58967fcd4f-qpggq" event={"ID":"610f1b35-d018-4931-815c-98bfc24ebfc5","Type":"ContainerDied","Data":"ba9f00a31b5bf9ecbddc49ec78ddd8994a281b85795ee2bcf0a369f057fab194"} Nov 02 07:00:36.464629 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:36.464540 5916 scope.go:122] "RemoveContainer" containerID="55dc76fa41f5d3c108cb7aeedd7f5e4862e80c3b1daac12d38b4ccf6e8738080" Nov 02 07:00:36.470926 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:36.470874 5916 generic.go:356] "Generic (PLEG): container finished" podID="1936eac5-6027-48a9-86ff-12ad457b94d9" containerID="fd52bf8f16f3bd30c336e3412703766a5dcbf0e40f02ae13ddf2c1b564aec501" exitCode=0 Nov 02 07:00:36.471073 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:36.471064 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a2ba11-5c66894c8c-p472q" Nov 02 07:00:36.471363 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:36.471224 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a2ba11-5c66894c8c-p472q" event={"ID":"1936eac5-6027-48a9-86ff-12ad457b94d9","Type":"ContainerDied","Data":"fd52bf8f16f3bd30c336e3412703766a5dcbf0e40f02ae13ddf2c1b564aec501"} Nov 02 07:00:36.471363 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:36.471276 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a2ba11-5c66894c8c-p472q" event={"ID":"1936eac5-6027-48a9-86ff-12ad457b94d9","Type":"ContainerDied","Data":"160ef2ef732422f6c10567ec93ae13def8a623c4b67a016a9fbaf55ac31f2b0b"} Nov 02 07:00:36.476807 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:36.476791 5916 scope.go:122] "RemoveContainer" containerID="55dc76fa41f5d3c108cb7aeedd7f5e4862e80c3b1daac12d38b4ccf6e8738080" Nov 02 07:00:36.477094 ip-10-0-130-93 kubenswrapper[5916]: E1102 07:00:36.477067 5916 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55dc76fa41f5d3c108cb7aeedd7f5e4862e80c3b1daac12d38b4ccf6e8738080\": container with ID starting with 55dc76fa41f5d3c108cb7aeedd7f5e4862e80c3b1daac12d38b4ccf6e8738080 not found: ID does not exist" containerID="55dc76fa41f5d3c108cb7aeedd7f5e4862e80c3b1daac12d38b4ccf6e8738080" Nov 02 07:00:36.477167 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:36.477104 5916 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55dc76fa41f5d3c108cb7aeedd7f5e4862e80c3b1daac12d38b4ccf6e8738080"} err="failed to get container status \"55dc76fa41f5d3c108cb7aeedd7f5e4862e80c3b1daac12d38b4ccf6e8738080\": rpc error: code = NotFound desc = could not find container \"55dc76fa41f5d3c108cb7aeedd7f5e4862e80c3b1daac12d38b4ccf6e8738080\": container with ID starting with 55dc76fa41f5d3c108cb7aeedd7f5e4862e80c3b1daac12d38b4ccf6e8738080 not found: ID does not exist" Nov 02 07:00:36.477167 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:36.477128 5916 scope.go:122] "RemoveContainer" containerID="fd52bf8f16f3bd30c336e3412703766a5dcbf0e40f02ae13ddf2c1b564aec501" Nov 02 07:00:36.486548 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:36.486527 5916 scope.go:122] "RemoveContainer" containerID="fd52bf8f16f3bd30c336e3412703766a5dcbf0e40f02ae13ddf2c1b564aec501" Nov 02 07:00:36.486792 ip-10-0-130-93 kubenswrapper[5916]: E1102 07:00:36.486773 5916 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd52bf8f16f3bd30c336e3412703766a5dcbf0e40f02ae13ddf2c1b564aec501\": container with ID starting with fd52bf8f16f3bd30c336e3412703766a5dcbf0e40f02ae13ddf2c1b564aec501 not found: ID does not exist" containerID="fd52bf8f16f3bd30c336e3412703766a5dcbf0e40f02ae13ddf2c1b564aec501" Nov 02 07:00:36.486847 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:36.486797 5916 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd52bf8f16f3bd30c336e3412703766a5dcbf0e40f02ae13ddf2c1b564aec501"} err="failed to get container status \"fd52bf8f16f3bd30c336e3412703766a5dcbf0e40f02ae13ddf2c1b564aec501\": rpc error: code = NotFound desc = could not find container \"fd52bf8f16f3bd30c336e3412703766a5dcbf0e40f02ae13ddf2c1b564aec501\": container with ID starting with fd52bf8f16f3bd30c336e3412703766a5dcbf0e40f02ae13ddf2c1b564aec501 not found: ID does not exist" Nov 02 07:00:36.631805 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:36.631776 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a2ba11-5c66894c8c-p472q"] Nov 02 07:00:36.637154 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:36.637129 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a2ba11-5c66894c8c-p472q"] Nov 02 07:00:36.747856 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:36.747784 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a2ba11-58967fcd4f-qpggq"] Nov 02 07:00:36.751420 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:36.751401 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a2ba11-58967fcd4f-qpggq"] Nov 02 07:00:37.841774 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:37.841744 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1936eac5-6027-48a9-86ff-12ad457b94d9" path="/var/lib/kubelet/pods/1936eac5-6027-48a9-86ff-12ad457b94d9/volumes" Nov 02 07:00:37.842240 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:37.842135 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="610f1b35-d018-4931-815c-98bfc24ebfc5" path="/var/lib/kubelet/pods/610f1b35-d018-4931-815c-98bfc24ebfc5/volumes" Nov 02 07:00:37.842526 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:37.842512 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc31d778-d117-4ca5-89e6-550d04804cd1" path="/var/lib/kubelet/pods/bc31d778-d117-4ca5-89e6-550d04804cd1/volumes" Nov 02 07:00:37.843418 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:37.843396 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fccb2c65-8f82-4808-bb0f-7d9333d8932a" path="/var/lib/kubelet/pods/fccb2c65-8f82-4808-bb0f-7d9333d8932a/volumes" Nov 02 07:00:41.114721 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:41.114685 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2212d36a-mjpqb"] Nov 02 07:00:41.115189 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:41.115130 5916 cpu_manager.go:414] "containerMap: removing container" podUID="ce7008b7-0096-4d6e-863e-60dce78fffe2" containerName="cleanup" Nov 02 07:00:41.115189 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:41.115142 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce7008b7-0096-4d6e-863e-60dce78fffe2" containerName="cleanup" Nov 02 07:00:41.115189 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:41.115151 5916 cpu_manager.go:414] "containerMap: removing container" podUID="f9f76fd8-1db3-4fe8-8a4e-117813676c2d" containerName="cleanup" Nov 02 07:00:41.115189 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:41.115156 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9f76fd8-1db3-4fe8-8a4e-117813676c2d" containerName="cleanup" Nov 02 07:00:41.115189 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:41.115164 5916 cpu_manager.go:414] "containerMap: removing container" podUID="f1173406-c493-422e-a67a-24a9580c9c0b" containerName="istio-proxy" Nov 02 07:00:41.115189 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:41.115169 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1173406-c493-422e-a67a-24a9580c9c0b" containerName="istio-proxy" Nov 02 07:00:41.115189 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:41.115175 5916 cpu_manager.go:414] "containerMap: removing container" podUID="f9f76fd8-1db3-4fe8-8a4e-117813676c2d" containerName="cleanup" Nov 02 07:00:41.115189 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:41.115179 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9f76fd8-1db3-4fe8-8a4e-117813676c2d" containerName="cleanup" Nov 02 07:00:41.115189 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:41.115190 5916 cpu_manager.go:414] "containerMap: removing container" podUID="dcf2f562-3c50-4de7-9c84-cb659d3bc25b" containerName="main" Nov 02 07:00:41.115189 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:41.115194 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcf2f562-3c50-4de7-9c84-cb659d3bc25b" containerName="main" Nov 02 07:00:41.115510 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:41.115202 5916 cpu_manager.go:414] "containerMap: removing container" podUID="b6ee4d66-7acd-4f3f-ad2e-4650a1019c1c" containerName="cleanup" Nov 02 07:00:41.115510 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:41.115206 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6ee4d66-7acd-4f3f-ad2e-4650a1019c1c" containerName="cleanup" Nov 02 07:00:41.115510 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:41.115215 5916 cpu_manager.go:414] "containerMap: removing container" podUID="837780ad-8ea3-4ed2-922f-917cffc7bb93" containerName="main" Nov 02 07:00:41.115510 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:41.115218 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="837780ad-8ea3-4ed2-922f-917cffc7bb93" containerName="main" Nov 02 07:00:41.115510 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:41.115226 5916 cpu_manager.go:414] "containerMap: removing container" podUID="9546555f-4f4c-4ca1-80e8-3817da23ed9d" containerName="cleanup" Nov 02 07:00:41.115510 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:41.115231 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9546555f-4f4c-4ca1-80e8-3817da23ed9d" containerName="cleanup" Nov 02 07:00:41.115510 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:41.115236 5916 cpu_manager.go:414] "containerMap: removing container" podUID="b6ee4d66-7acd-4f3f-ad2e-4650a1019c1c" containerName="cleanup" Nov 02 07:00:41.115510 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:41.115240 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6ee4d66-7acd-4f3f-ad2e-4650a1019c1c" containerName="cleanup" Nov 02 07:00:41.115510 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:41.115248 5916 cpu_manager.go:414] "containerMap: removing container" podUID="9546555f-4f4c-4ca1-80e8-3817da23ed9d" containerName="cleanup" Nov 02 07:00:41.115510 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:41.115252 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9546555f-4f4c-4ca1-80e8-3817da23ed9d" containerName="cleanup" Nov 02 07:00:41.115510 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:41.115258 5916 cpu_manager.go:414] "containerMap: removing container" podUID="b6ee4d66-7acd-4f3f-ad2e-4650a1019c1c" containerName="cleanup" Nov 02 07:00:41.115510 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:41.115262 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6ee4d66-7acd-4f3f-ad2e-4650a1019c1c" containerName="cleanup" Nov 02 07:00:41.115510 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:41.115269 5916 cpu_manager.go:414] "containerMap: removing container" podUID="610f1b35-d018-4931-815c-98bfc24ebfc5" containerName="payload-processing" Nov 02 07:00:41.115510 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:41.115273 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="610f1b35-d018-4931-815c-98bfc24ebfc5" containerName="payload-processing" Nov 02 07:00:41.115510 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:41.115284 5916 cpu_manager.go:414] "containerMap: removing container" podUID="529e25e6-4479-49d6-85c7-85dad6c2193e" containerName="main" Nov 02 07:00:41.115510 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:41.115288 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="529e25e6-4479-49d6-85c7-85dad6c2193e" containerName="main" Nov 02 07:00:41.115510 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:41.115296 5916 cpu_manager.go:414] "containerMap: removing container" podUID="ce7008b7-0096-4d6e-863e-60dce78fffe2" containerName="cleanup" Nov 02 07:00:41.115510 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:41.115300 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce7008b7-0096-4d6e-863e-60dce78fffe2" containerName="cleanup" Nov 02 07:00:41.115510 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:41.115311 5916 cpu_manager.go:414] "containerMap: removing container" podUID="d018c073-2cde-4cbc-8209-a2d788777d2a" containerName="istio-proxy" Nov 02 07:00:41.115510 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:41.115316 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d018c073-2cde-4cbc-8209-a2d788777d2a" containerName="istio-proxy" Nov 02 07:00:41.115510 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:41.115323 5916 cpu_manager.go:414] "containerMap: removing container" podUID="fccb2c65-8f82-4808-bb0f-7d9333d8932a" containerName="istio-proxy" Nov 02 07:00:41.115510 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:41.115327 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fccb2c65-8f82-4808-bb0f-7d9333d8932a" containerName="istio-proxy" Nov 02 07:00:41.115510 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:41.115334 5916 cpu_manager.go:414] "containerMap: removing container" podUID="fccb2c65-8f82-4808-bb0f-7d9333d8932a" containerName="istio-proxy" Nov 02 07:00:41.115510 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:41.115338 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fccb2c65-8f82-4808-bb0f-7d9333d8932a" containerName="istio-proxy" Nov 02 07:00:41.115510 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:41.115343 5916 cpu_manager.go:414] "containerMap: removing container" podUID="ed18eacc-a61b-470d-8cd9-1d46f742fabd" containerName="discovery" Nov 02 07:00:41.115510 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:41.115348 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed18eacc-a61b-470d-8cd9-1d46f742fabd" containerName="discovery" Nov 02 07:00:41.115510 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:41.115354 5916 cpu_manager.go:414] "containerMap: removing container" podUID="bc31d778-d117-4ca5-89e6-550d04804cd1" containerName="istio-proxy" Nov 02 07:00:41.115510 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:41.115359 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc31d778-d117-4ca5-89e6-550d04804cd1" containerName="istio-proxy" Nov 02 07:00:41.115510 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:41.115365 5916 cpu_manager.go:414] "containerMap: removing container" podUID="c031ed74-a8ca-4080-8216-9e5c55d2fc6e" containerName="revoke-keys" Nov 02 07:00:41.115510 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:41.115370 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c031ed74-a8ca-4080-8216-9e5c55d2fc6e" containerName="revoke-keys" Nov 02 07:00:41.115510 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:41.115380 5916 cpu_manager.go:414] "containerMap: removing container" podUID="63b6b2e6-166d-4571-b148-6c504875697c" containerName="payload-pre-processing" Nov 02 07:00:41.115510 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:41.115385 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63b6b2e6-166d-4571-b148-6c504875697c" containerName="payload-pre-processing" Nov 02 07:00:41.115510 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:41.115391 5916 cpu_manager.go:414] "containerMap: removing container" podUID="1e4e8418-4884-4d61-a0c3-54dcd871255e" containerName="revoke-keys" Nov 02 07:00:41.115510 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:41.115395 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e4e8418-4884-4d61-a0c3-54dcd871255e" containerName="revoke-keys" Nov 02 07:00:41.115510 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:41.115401 5916 cpu_manager.go:414] "containerMap: removing container" podUID="1936eac5-6027-48a9-86ff-12ad457b94d9" containerName="payload-pre-processing" Nov 02 07:00:41.115510 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:41.115405 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1936eac5-6027-48a9-86ff-12ad457b94d9" containerName="payload-pre-processing" Nov 02 07:00:41.115510 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:41.115463 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="837780ad-8ea3-4ed2-922f-917cffc7bb93" containerName="main" Nov 02 07:00:41.115510 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:41.115470 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="9546555f-4f4c-4ca1-80e8-3817da23ed9d" containerName="cleanup" Nov 02 07:00:41.115510 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:41.115476 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6ee4d66-7acd-4f3f-ad2e-4650a1019c1c" containerName="cleanup" Nov 02 07:00:41.115510 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:41.115483 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="9546555f-4f4c-4ca1-80e8-3817da23ed9d" containerName="cleanup" Nov 02 07:00:41.115510 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:41.115488 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6ee4d66-7acd-4f3f-ad2e-4650a1019c1c" containerName="cleanup" Nov 02 07:00:41.115510 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:41.115493 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="610f1b35-d018-4931-815c-98bfc24ebfc5" containerName="payload-processing" Nov 02 07:00:41.115510 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:41.115500 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="529e25e6-4479-49d6-85c7-85dad6c2193e" containerName="main" Nov 02 07:00:41.115510 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:41.115507 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce7008b7-0096-4d6e-863e-60dce78fffe2" containerName="cleanup" Nov 02 07:00:41.115510 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:41.115513 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="d018c073-2cde-4cbc-8209-a2d788777d2a" containerName="istio-proxy" Nov 02 07:00:41.115510 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:41.115518 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9f76fd8-1db3-4fe8-8a4e-117813676c2d" containerName="cleanup" Nov 02 07:00:41.115510 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:41.115524 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="fccb2c65-8f82-4808-bb0f-7d9333d8932a" containerName="istio-proxy" Nov 02 07:00:41.115510 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:41.115529 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="fccb2c65-8f82-4808-bb0f-7d9333d8932a" containerName="istio-proxy" Nov 02 07:00:41.115510 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:41.115533 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed18eacc-a61b-470d-8cd9-1d46f742fabd" containerName="discovery" Nov 02 07:00:41.116981 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:41.115539 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc31d778-d117-4ca5-89e6-550d04804cd1" containerName="istio-proxy" Nov 02 07:00:41.116981 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:41.115543 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="c031ed74-a8ca-4080-8216-9e5c55d2fc6e" containerName="revoke-keys" Nov 02 07:00:41.116981 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:41.115549 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="63b6b2e6-166d-4571-b148-6c504875697c" containerName="payload-pre-processing" Nov 02 07:00:41.116981 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:41.115554 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e4e8418-4884-4d61-a0c3-54dcd871255e" containerName="revoke-keys" Nov 02 07:00:41.116981 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:41.115558 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="1936eac5-6027-48a9-86ff-12ad457b94d9" containerName="payload-pre-processing" Nov 02 07:00:41.116981 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:41.115564 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce7008b7-0096-4d6e-863e-60dce78fffe2" containerName="cleanup" Nov 02 07:00:41.116981 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:41.115568 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9f76fd8-1db3-4fe8-8a4e-117813676c2d" containerName="cleanup" Nov 02 07:00:41.116981 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:41.115573 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1173406-c493-422e-a67a-24a9580c9c0b" containerName="istio-proxy" Nov 02 07:00:41.116981 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:41.115578 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9f76fd8-1db3-4fe8-8a4e-117813676c2d" containerName="cleanup" Nov 02 07:00:41.116981 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:41.115583 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="dcf2f562-3c50-4de7-9c84-cb659d3bc25b" containerName="main" Nov 02 07:00:41.120385 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:41.120364 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2212d36a-mjpqb" Nov 02 07:00:41.175301 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:41.175267 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2212d36a-mjpqb"] Nov 02 07:00:41.222502 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:41.222471 5916 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 07:00:41.222722 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:41.222476 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 07:00:41.222875 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:41.222506 5916 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 07:00:41.225335 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:41.225310 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b87d28f4-pbmdq"] Nov 02 07:00:41.225847 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:41.225829 5916 cpu_manager.go:414] "containerMap: removing container" podUID="ce7008b7-0096-4d6e-863e-60dce78fffe2" containerName="cleanup" Nov 02 07:00:41.225956 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:41.225848 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce7008b7-0096-4d6e-863e-60dce78fffe2" containerName="cleanup" Nov 02 07:00:41.225956 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:41.225866 5916 cpu_manager.go:414] "containerMap: removing container" podUID="f9f76fd8-1db3-4fe8-8a4e-117813676c2d" containerName="cleanup" Nov 02 07:00:41.225956 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:41.225875 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9f76fd8-1db3-4fe8-8a4e-117813676c2d" containerName="cleanup" Nov 02 07:00:41.225956 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:41.225895 5916 cpu_manager.go:414] "containerMap: removing container" podUID="9546555f-4f4c-4ca1-80e8-3817da23ed9d" containerName="cleanup" Nov 02 07:00:41.225956 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:41.225928 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9546555f-4f4c-4ca1-80e8-3817da23ed9d" containerName="cleanup" Nov 02 07:00:41.226205 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:41.226049 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="9546555f-4f4c-4ca1-80e8-3817da23ed9d" containerName="cleanup" Nov 02 07:00:41.226205 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:41.226061 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6ee4d66-7acd-4f3f-ad2e-4650a1019c1c" containerName="cleanup" Nov 02 07:00:41.226205 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:41.226075 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce7008b7-0096-4d6e-863e-60dce78fffe2" containerName="cleanup" Nov 02 07:00:41.228408 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:41.228392 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-68l4r\"" Nov 02 07:00:41.229741 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:41.229723 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b87d28f4-pbmdq" Nov 02 07:00:41.243391 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:41.243366 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17bfa708-9d69-4ca7-ac46-d987e81292a7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2212d36a-mjpqb\" (UID: \"17bfa708-9d69-4ca7-ac46-d987e81292a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2212d36a-mjpqb" Nov 02 07:00:41.243487 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:41.243446 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17bfa708-9d69-4ca7-ac46-d987e81292a7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2212d36a-mjpqb\" (UID: \"17bfa708-9d69-4ca7-ac46-d987e81292a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2212d36a-mjpqb" Nov 02 07:00:41.243540 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:41.243521 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pm22f\" (UniqueName: \"kubernetes.io/projected/17bfa708-9d69-4ca7-ac46-d987e81292a7-kube-api-access-pm22f\") pod \"maas-api-revoke-sub-2212d36a-mjpqb\" (UID: \"17bfa708-9d69-4ca7-ac46-d987e81292a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2212d36a-mjpqb" Nov 02 07:00:41.272595 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:41.272521 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b87d28f4-pbmdq"] Nov 02 07:00:41.345068 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:41.345032 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bdb5f812-0bc7-4d0a-a5aa-fe18615da989-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b87d28f4-pbmdq\" (UID: \"bdb5f812-0bc7-4d0a-a5aa-fe18615da989\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b87d28f4-pbmdq" Nov 02 07:00:41.345068 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:41.345072 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bdb5f812-0bc7-4d0a-a5aa-fe18615da989-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b87d28f4-pbmdq\" (UID: \"bdb5f812-0bc7-4d0a-a5aa-fe18615da989\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b87d28f4-pbmdq" Nov 02 07:00:41.345272 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:41.345243 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17bfa708-9d69-4ca7-ac46-d987e81292a7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2212d36a-mjpqb\" (UID: \"17bfa708-9d69-4ca7-ac46-d987e81292a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2212d36a-mjpqb" Nov 02 07:00:41.345343 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:41.345316 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9tmv\" (UniqueName: \"kubernetes.io/projected/bdb5f812-0bc7-4d0a-a5aa-fe18615da989-kube-api-access-x9tmv\") pod \"maas-api-revoke-sub-b87d28f4-pbmdq\" (UID: \"bdb5f812-0bc7-4d0a-a5aa-fe18615da989\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b87d28f4-pbmdq" Nov 02 07:00:41.345519 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:41.345358 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17bfa708-9d69-4ca7-ac46-d987e81292a7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2212d36a-mjpqb\" (UID: \"17bfa708-9d69-4ca7-ac46-d987e81292a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2212d36a-mjpqb" Nov 02 07:00:41.345519 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:41.345388 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pm22f\" (UniqueName: \"kubernetes.io/projected/17bfa708-9d69-4ca7-ac46-d987e81292a7-kube-api-access-pm22f\") pod \"maas-api-revoke-sub-2212d36a-mjpqb\" (UID: \"17bfa708-9d69-4ca7-ac46-d987e81292a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2212d36a-mjpqb" Nov 02 07:00:41.345993 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:41.345973 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17bfa708-9d69-4ca7-ac46-d987e81292a7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2212d36a-mjpqb\" (UID: \"17bfa708-9d69-4ca7-ac46-d987e81292a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2212d36a-mjpqb" Nov 02 07:00:41.348011 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:41.347991 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17bfa708-9d69-4ca7-ac46-d987e81292a7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2212d36a-mjpqb\" (UID: \"17bfa708-9d69-4ca7-ac46-d987e81292a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2212d36a-mjpqb" Nov 02 07:00:41.403017 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:41.402988 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pm22f\" (UniqueName: \"kubernetes.io/projected/17bfa708-9d69-4ca7-ac46-d987e81292a7-kube-api-access-pm22f\") pod \"maas-api-revoke-sub-2212d36a-mjpqb\" (UID: \"17bfa708-9d69-4ca7-ac46-d987e81292a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2212d36a-mjpqb" Nov 02 07:00:41.434000 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:41.433966 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2212d36a-mjpqb" Nov 02 07:00:41.446675 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:41.446640 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9tmv\" (UniqueName: \"kubernetes.io/projected/bdb5f812-0bc7-4d0a-a5aa-fe18615da989-kube-api-access-x9tmv\") pod \"maas-api-revoke-sub-b87d28f4-pbmdq\" (UID: \"bdb5f812-0bc7-4d0a-a5aa-fe18615da989\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b87d28f4-pbmdq" Nov 02 07:00:41.446805 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:41.446687 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bdb5f812-0bc7-4d0a-a5aa-fe18615da989-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b87d28f4-pbmdq\" (UID: \"bdb5f812-0bc7-4d0a-a5aa-fe18615da989\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b87d28f4-pbmdq" Nov 02 07:00:41.446805 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:41.446704 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bdb5f812-0bc7-4d0a-a5aa-fe18615da989-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b87d28f4-pbmdq\" (UID: \"bdb5f812-0bc7-4d0a-a5aa-fe18615da989\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b87d28f4-pbmdq" Nov 02 07:00:41.447674 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:41.447641 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bdb5f812-0bc7-4d0a-a5aa-fe18615da989-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b87d28f4-pbmdq\" (UID: \"bdb5f812-0bc7-4d0a-a5aa-fe18615da989\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b87d28f4-pbmdq" Nov 02 07:00:41.449333 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:41.449305 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bdb5f812-0bc7-4d0a-a5aa-fe18615da989-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b87d28f4-pbmdq\" (UID: \"bdb5f812-0bc7-4d0a-a5aa-fe18615da989\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b87d28f4-pbmdq" Nov 02 07:00:41.511588 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:41.511561 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9tmv\" (UniqueName: \"kubernetes.io/projected/bdb5f812-0bc7-4d0a-a5aa-fe18615da989-kube-api-access-x9tmv\") pod \"maas-api-revoke-sub-b87d28f4-pbmdq\" (UID: \"bdb5f812-0bc7-4d0a-a5aa-fe18615da989\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b87d28f4-pbmdq" Nov 02 07:00:41.546354 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:41.546085 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b87d28f4-pbmdq" Nov 02 07:00:41.823208 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:41.823175 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2212d36a-mjpqb"] Nov 02 07:00:41.824243 ip-10-0-130-93 kubenswrapper[5916]: W1102 07:00:41.824216 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17bfa708_9d69_4ca7_ac46_d987e81292a7.slice/crio-e46a2cb1cc2d9340bdf81d8372a52a94c664ab4c498b8dea99d591d7252a0329 WatchSource:0}: Error finding container e46a2cb1cc2d9340bdf81d8372a52a94c664ab4c498b8dea99d591d7252a0329: Status 404 returned error can't find the container with id e46a2cb1cc2d9340bdf81d8372a52a94c664ab4c498b8dea99d591d7252a0329 Nov 02 07:00:41.939378 ip-10-0-130-93 kubenswrapper[5916]: W1102 07:00:41.939349 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdb5f812_0bc7_4d0a_a5aa_fe18615da989.slice/crio-693fcbdb38a64d5028f7b42e33d2cbc1c7c4a490fcb68cea187fba0b9c00f9c9 WatchSource:0}: Error finding container 693fcbdb38a64d5028f7b42e33d2cbc1c7c4a490fcb68cea187fba0b9c00f9c9: Status 404 returned error can't find the container with id 693fcbdb38a64d5028f7b42e33d2cbc1c7c4a490fcb68cea187fba0b9c00f9c9 Nov 02 07:00:41.939580 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:41.939553 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b87d28f4-pbmdq"] Nov 02 07:00:42.542349 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:42.542259 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2212d36a-mjpqb" event={"ID":"17bfa708-9d69-4ca7-ac46-d987e81292a7","Type":"ContainerStarted","Data":"cf9ca3a606f9776f13687bc4e6d4d4b8111e358676eb0f33a090471dc24db776"} Nov 02 07:00:42.542349 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:42.542328 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2212d36a-mjpqb" event={"ID":"17bfa708-9d69-4ca7-ac46-d987e81292a7","Type":"ContainerStarted","Data":"e46a2cb1cc2d9340bdf81d8372a52a94c664ab4c498b8dea99d591d7252a0329"} Nov 02 07:00:42.545051 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:42.545022 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b87d28f4-pbmdq" event={"ID":"bdb5f812-0bc7-4d0a-a5aa-fe18615da989","Type":"ContainerStarted","Data":"0cf12270d11cde0e9f57f6b402c8fdf574c7a7e0a2e5db6e9a9c3b91390e6b27"} Nov 02 07:00:42.545198 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:42.545071 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b87d28f4-pbmdq" event={"ID":"bdb5f812-0bc7-4d0a-a5aa-fe18615da989","Type":"ContainerStarted","Data":"693fcbdb38a64d5028f7b42e33d2cbc1c7c4a490fcb68cea187fba0b9c00f9c9"} Nov 02 07:00:42.657409 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:42.657354 5916 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2212d36a-mjpqb" podStartSLOduration=1.657341183 podStartE2EDuration="1.657341183s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:00:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:00:42.656348439 +0000 UTC m=+2630.928168831" watchObservedRunningTime="2026-11-02 07:00:42.657341183 +0000 UTC m=+2630.929161577" Nov 02 07:00:42.775397 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:42.775340 5916 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b87d28f4-pbmdq" podStartSLOduration=1.775326646 podStartE2EDuration="1.775326646s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:00:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:00:42.772532679 +0000 UTC m=+2631.044353075" watchObservedRunningTime="2026-11-02 07:00:42.775326646 +0000 UTC m=+2631.047147039" Nov 02 07:00:43.247585 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:43.247545 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a2ba11-x74pv"] Nov 02 07:00:43.249974 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:43.249946 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a2ba11-x74pv"] Nov 02 07:00:43.554396 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:43.554315 5916 generic.go:356] "Generic (PLEG): container finished" podID="17bfa708-9d69-4ca7-ac46-d987e81292a7" containerID="cf9ca3a606f9776f13687bc4e6d4d4b8111e358676eb0f33a090471dc24db776" exitCode=0 Nov 02 07:00:43.554847 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:43.554426 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2212d36a-mjpqb" event={"ID":"17bfa708-9d69-4ca7-ac46-d987e81292a7","Type":"ContainerDied","Data":"cf9ca3a606f9776f13687bc4e6d4d4b8111e358676eb0f33a090471dc24db776"} Nov 02 07:00:43.556176 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:43.556155 5916 generic.go:356] "Generic (PLEG): container finished" podID="bdb5f812-0bc7-4d0a-a5aa-fe18615da989" containerID="0cf12270d11cde0e9f57f6b402c8fdf574c7a7e0a2e5db6e9a9c3b91390e6b27" exitCode=0 Nov 02 07:00:43.556310 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:43.556279 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b87d28f4-pbmdq" event={"ID":"bdb5f812-0bc7-4d0a-a5aa-fe18615da989","Type":"ContainerDied","Data":"0cf12270d11cde0e9f57f6b402c8fdf574c7a7e0a2e5db6e9a9c3b91390e6b27"} Nov 02 07:00:43.844482 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:43.844401 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e4e8418-4884-4d61-a0c3-54dcd871255e" path="/var/lib/kubelet/pods/1e4e8418-4884-4d61-a0c3-54dcd871255e/volumes" Nov 02 07:00:44.557858 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:44.557823 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8bc131-nhr4c"] Nov 02 07:00:44.561089 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:44.561060 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8bc131-nhr4c"] Nov 02 07:00:44.706650 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:44.706620 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2212d36a-mjpqb" Nov 02 07:00:44.740151 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:44.740129 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b87d28f4-pbmdq" Nov 02 07:00:44.883869 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:44.883843 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17bfa708-9d69-4ca7-ac46-d987e81292a7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17bfa708-9d69-4ca7-ac46-d987e81292a7-maas-api-xks-ca\") pod \"17bfa708-9d69-4ca7-ac46-d987e81292a7\" (UID: \"17bfa708-9d69-4ca7-ac46-d987e81292a7\") " Nov 02 07:00:44.884153 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:44.883916 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17bfa708-9d69-4ca7-ac46-d987e81292a7-kube-api-access-pm22f\" (UniqueName: \"kubernetes.io/projected/17bfa708-9d69-4ca7-ac46-d987e81292a7-kube-api-access-pm22f\") pod \"17bfa708-9d69-4ca7-ac46-d987e81292a7\" (UID: \"17bfa708-9d69-4ca7-ac46-d987e81292a7\") " Nov 02 07:00:44.884153 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:44.883936 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bdb5f812-0bc7-4d0a-a5aa-fe18615da989-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bdb5f812-0bc7-4d0a-a5aa-fe18615da989-maas-api-xks-ca\") pod \"bdb5f812-0bc7-4d0a-a5aa-fe18615da989\" (UID: \"bdb5f812-0bc7-4d0a-a5aa-fe18615da989\") " Nov 02 07:00:44.884153 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:44.883967 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bdb5f812-0bc7-4d0a-a5aa-fe18615da989-kube-api-access-x9tmv\" (UniqueName: \"kubernetes.io/projected/bdb5f812-0bc7-4d0a-a5aa-fe18615da989-kube-api-access-x9tmv\") pod \"bdb5f812-0bc7-4d0a-a5aa-fe18615da989\" (UID: \"bdb5f812-0bc7-4d0a-a5aa-fe18615da989\") " Nov 02 07:00:44.884153 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:44.883995 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17bfa708-9d69-4ca7-ac46-d987e81292a7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17bfa708-9d69-4ca7-ac46-d987e81292a7-maas-api-openshift-ca\") pod \"17bfa708-9d69-4ca7-ac46-d987e81292a7\" (UID: \"17bfa708-9d69-4ca7-ac46-d987e81292a7\") " Nov 02 07:00:44.884153 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:44.884044 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bdb5f812-0bc7-4d0a-a5aa-fe18615da989-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bdb5f812-0bc7-4d0a-a5aa-fe18615da989-maas-api-openshift-ca\") pod \"bdb5f812-0bc7-4d0a-a5aa-fe18615da989\" (UID: \"bdb5f812-0bc7-4d0a-a5aa-fe18615da989\") " Nov 02 07:00:44.884472 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:44.884369 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17bfa708-9d69-4ca7-ac46-d987e81292a7-maas-api-openshift-ca" pod "17bfa708-9d69-4ca7-ac46-d987e81292a7" (UID: "17bfa708-9d69-4ca7-ac46-d987e81292a7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:00:44.884472 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:44.884397 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdb5f812-0bc7-4d0a-a5aa-fe18615da989-maas-api-openshift-ca" pod "bdb5f812-0bc7-4d0a-a5aa-fe18615da989" (UID: "bdb5f812-0bc7-4d0a-a5aa-fe18615da989"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:00:44.886160 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:44.886134 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bdb5f812-0bc7-4d0a-a5aa-fe18615da989-maas-api-xks-ca" pod "bdb5f812-0bc7-4d0a-a5aa-fe18615da989" (UID: "bdb5f812-0bc7-4d0a-a5aa-fe18615da989"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 07:00:44.886275 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:44.886210 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17bfa708-9d69-4ca7-ac46-d987e81292a7-kube-api-access-pm22f" pod "17bfa708-9d69-4ca7-ac46-d987e81292a7" (UID: "17bfa708-9d69-4ca7-ac46-d987e81292a7"). InnerVolumeSpecName "kube-api-access-pm22f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:00:44.886275 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:44.886212 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17bfa708-9d69-4ca7-ac46-d987e81292a7-maas-api-xks-ca" pod "17bfa708-9d69-4ca7-ac46-d987e81292a7" (UID: "17bfa708-9d69-4ca7-ac46-d987e81292a7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 07:00:44.886275 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:44.886234 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdb5f812-0bc7-4d0a-a5aa-fe18615da989-kube-api-access-x9tmv" pod "bdb5f812-0bc7-4d0a-a5aa-fe18615da989" (UID: "bdb5f812-0bc7-4d0a-a5aa-fe18615da989"). InnerVolumeSpecName "kube-api-access-x9tmv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:00:44.985721 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:44.985689 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17bfa708-9d69-4ca7-ac46-d987e81292a7-maas-api-xks-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:44.985721 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:44.985715 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pm22f\" (UniqueName: \"kubernetes.io/projected/17bfa708-9d69-4ca7-ac46-d987e81292a7-kube-api-access-pm22f\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:44.985721 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:44.985727 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bdb5f812-0bc7-4d0a-a5aa-fe18615da989-maas-api-xks-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:44.985995 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:44.985736 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9tmv\" (UniqueName: \"kubernetes.io/projected/bdb5f812-0bc7-4d0a-a5aa-fe18615da989-kube-api-access-x9tmv\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:44.985995 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:44.985744 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17bfa708-9d69-4ca7-ac46-d987e81292a7-maas-api-openshift-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:44.985995 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:44.985755 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bdb5f812-0bc7-4d0a-a5aa-fe18615da989-maas-api-openshift-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:45.573598 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:45.573570 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2212d36a-mjpqb" Nov 02 07:00:45.574075 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:45.573608 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2212d36a-mjpqb" event={"ID":"17bfa708-9d69-4ca7-ac46-d987e81292a7","Type":"ContainerDied","Data":"e46a2cb1cc2d9340bdf81d8372a52a94c664ab4c498b8dea99d591d7252a0329"} Nov 02 07:00:45.574075 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:45.573657 5916 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e46a2cb1cc2d9340bdf81d8372a52a94c664ab4c498b8dea99d591d7252a0329" Nov 02 07:00:45.575104 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:45.575080 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b87d28f4-pbmdq" Nov 02 07:00:45.575227 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:45.575120 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b87d28f4-pbmdq" event={"ID":"bdb5f812-0bc7-4d0a-a5aa-fe18615da989","Type":"ContainerDied","Data":"693fcbdb38a64d5028f7b42e33d2cbc1c7c4a490fcb68cea187fba0b9c00f9c9"} Nov 02 07:00:45.575227 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:45.575165 5916 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="693fcbdb38a64d5028f7b42e33d2cbc1c7c4a490fcb68cea187fba0b9c00f9c9" Nov 02 07:00:45.848095 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:45.848000 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c031ed74-a8ca-4080-8216-9e5c55d2fc6e" path="/var/lib/kubelet/pods/c031ed74-a8ca-4080-8216-9e5c55d2fc6e/volumes" Nov 02 07:00:49.898216 ip-10-0-130-93 kubenswrapper[5916]: E1102 07:00:49.898149 5916 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p98cq" containerName="extractor" Nov 02 07:00:50.676455 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:50.676420 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5a85b0-dxrlv"] Nov 02 07:00:50.676955 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:50.676937 5916 cpu_manager.go:414] "containerMap: removing container" podUID="17bfa708-9d69-4ca7-ac46-d987e81292a7" containerName="revoke-keys" Nov 02 07:00:50.677126 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:50.676956 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17bfa708-9d69-4ca7-ac46-d987e81292a7" containerName="revoke-keys" Nov 02 07:00:50.677126 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:50.676975 5916 cpu_manager.go:414] "containerMap: removing container" podUID="bdb5f812-0bc7-4d0a-a5aa-fe18615da989" containerName="revoke-keys" Nov 02 07:00:50.677126 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:50.676982 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bdb5f812-0bc7-4d0a-a5aa-fe18615da989" containerName="revoke-keys" Nov 02 07:00:50.677298 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:50.677130 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="17bfa708-9d69-4ca7-ac46-d987e81292a7" containerName="revoke-keys" Nov 02 07:00:50.677298 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:50.677144 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="bdb5f812-0bc7-4d0a-a5aa-fe18615da989" containerName="revoke-keys" Nov 02 07:00:50.680573 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:50.680554 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5a85b0-dxrlv" Nov 02 07:00:50.732983 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:50.732956 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5a85b0-dxrlv"] Nov 02 07:00:50.737140 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:50.737116 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34d10ad6-09c2-4135-8981-fa140ec26bb6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-5a85b0-dxrlv\" (UID: \"34d10ad6-09c2-4135-8981-fa140ec26bb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5a85b0-dxrlv" Nov 02 07:00:50.737258 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:50.737171 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34d10ad6-09c2-4135-8981-fa140ec26bb6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-5a85b0-dxrlv\" (UID: \"34d10ad6-09c2-4135-8981-fa140ec26bb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5a85b0-dxrlv" Nov 02 07:00:50.737258 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:50.737239 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrkvn\" (UniqueName: \"kubernetes.io/projected/34d10ad6-09c2-4135-8981-fa140ec26bb6-kube-api-access-xrkvn\") pod \"maas-api-revoke-keys-e2e-inf-b-5a85b0-dxrlv\" (UID: \"34d10ad6-09c2-4135-8981-fa140ec26bb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5a85b0-dxrlv" Nov 02 07:00:50.779927 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:50.779870 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 07:00:50.780071 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:50.779953 5916 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 07:00:50.780071 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:50.779987 5916 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 07:00:50.782385 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:50.782369 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-68l4r\"" Nov 02 07:00:50.837827 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:50.837799 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34d10ad6-09c2-4135-8981-fa140ec26bb6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-5a85b0-dxrlv\" (UID: \"34d10ad6-09c2-4135-8981-fa140ec26bb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5a85b0-dxrlv" Nov 02 07:00:50.838018 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:50.837867 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34d10ad6-09c2-4135-8981-fa140ec26bb6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-5a85b0-dxrlv\" (UID: \"34d10ad6-09c2-4135-8981-fa140ec26bb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5a85b0-dxrlv" Nov 02 07:00:50.838018 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:50.837973 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrkvn\" (UniqueName: \"kubernetes.io/projected/34d10ad6-09c2-4135-8981-fa140ec26bb6-kube-api-access-xrkvn\") pod \"maas-api-revoke-keys-e2e-inf-b-5a85b0-dxrlv\" (UID: \"34d10ad6-09c2-4135-8981-fa140ec26bb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5a85b0-dxrlv" Nov 02 07:00:50.838598 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:50.838577 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34d10ad6-09c2-4135-8981-fa140ec26bb6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-5a85b0-dxrlv\" (UID: \"34d10ad6-09c2-4135-8981-fa140ec26bb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5a85b0-dxrlv" Nov 02 07:00:50.840542 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:50.840521 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34d10ad6-09c2-4135-8981-fa140ec26bb6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-5a85b0-dxrlv\" (UID: \"34d10ad6-09c2-4135-8981-fa140ec26bb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5a85b0-dxrlv" Nov 02 07:00:50.895242 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:50.895208 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrkvn\" (UniqueName: \"kubernetes.io/projected/34d10ad6-09c2-4135-8981-fa140ec26bb6-kube-api-access-xrkvn\") pod \"maas-api-revoke-keys-e2e-inf-b-5a85b0-dxrlv\" (UID: \"34d10ad6-09c2-4135-8981-fa140ec26bb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5a85b0-dxrlv" Nov 02 07:00:50.993288 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:50.993215 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5a85b0-dxrlv" Nov 02 07:00:51.368821 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:51.368797 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5a85b0-dxrlv"] Nov 02 07:00:51.370672 ip-10-0-130-93 kubenswrapper[5916]: W1102 07:00:51.370644 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34d10ad6_09c2_4135_8981_fa140ec26bb6.slice/crio-5ddf5e41617661db437e359f9e8d6bd2c385d83f39d48559d14d96d414e33ef6 WatchSource:0}: Error finding container 5ddf5e41617661db437e359f9e8d6bd2c385d83f39d48559d14d96d414e33ef6: Status 404 returned error can't find the container with id 5ddf5e41617661db437e359f9e8d6bd2c385d83f39d48559d14d96d414e33ef6 Nov 02 07:00:51.647201 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:51.647166 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5a85b0-dxrlv" event={"ID":"34d10ad6-09c2-4135-8981-fa140ec26bb6","Type":"ContainerStarted","Data":"d9d1973197793ed88b68766b1e282f0eaf612a4d2194a36d74738c82a2d9b833"} Nov 02 07:00:51.647336 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:51.647261 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5a85b0-dxrlv" event={"ID":"34d10ad6-09c2-4135-8981-fa140ec26bb6","Type":"ContainerStarted","Data":"5ddf5e41617661db437e359f9e8d6bd2c385d83f39d48559d14d96d414e33ef6"} Nov 02 07:00:51.763491 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:51.763443 5916 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5a85b0-dxrlv" podStartSLOduration=1.763429454 podStartE2EDuration="1.763429454s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:00:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:00:51.760305361 +0000 UTC m=+2640.032125754" watchObservedRunningTime="2026-11-02 07:00:51.763429454 +0000 UTC m=+2640.035249871" Nov 02 07:00:52.662176 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:52.662136 5916 generic.go:356] "Generic (PLEG): container finished" podID="34d10ad6-09c2-4135-8981-fa140ec26bb6" containerID="d9d1973197793ed88b68766b1e282f0eaf612a4d2194a36d74738c82a2d9b833" exitCode=0 Nov 02 07:00:52.662690 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:52.662230 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5a85b0-dxrlv" event={"ID":"34d10ad6-09c2-4135-8981-fa140ec26bb6","Type":"ContainerDied","Data":"d9d1973197793ed88b68766b1e282f0eaf612a4d2194a36d74738c82a2d9b833"} Nov 02 07:00:53.308916 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:53.308862 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2e9f81-rn89z"] Nov 02 07:00:53.312700 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:53.312676 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2e9f81-rn89z" Nov 02 07:00:53.359168 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:53.359139 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68ls9\" (UniqueName: \"kubernetes.io/projected/223f9a36-39dc-4472-8928-92608418e54d-kube-api-access-68ls9\") pod \"maas-api-revoke-keys-e2e-inf-b-2e9f81-rn89z\" (UID: \"223f9a36-39dc-4472-8928-92608418e54d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2e9f81-rn89z" Nov 02 07:00:53.359298 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:53.359189 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/223f9a36-39dc-4472-8928-92608418e54d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-2e9f81-rn89z\" (UID: \"223f9a36-39dc-4472-8928-92608418e54d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2e9f81-rn89z" Nov 02 07:00:53.359390 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:53.359365 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/223f9a36-39dc-4472-8928-92608418e54d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-2e9f81-rn89z\" (UID: \"223f9a36-39dc-4472-8928-92608418e54d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2e9f81-rn89z" Nov 02 07:00:53.365736 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:53.365712 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2e9f81-rn89z"] Nov 02 07:00:53.460658 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:53.460630 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68ls9\" (UniqueName: \"kubernetes.io/projected/223f9a36-39dc-4472-8928-92608418e54d-kube-api-access-68ls9\") pod \"maas-api-revoke-keys-e2e-inf-b-2e9f81-rn89z\" (UID: \"223f9a36-39dc-4472-8928-92608418e54d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2e9f81-rn89z" Nov 02 07:00:53.460806 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:53.460667 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/223f9a36-39dc-4472-8928-92608418e54d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-2e9f81-rn89z\" (UID: \"223f9a36-39dc-4472-8928-92608418e54d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2e9f81-rn89z" Nov 02 07:00:53.460806 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:53.460713 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/223f9a36-39dc-4472-8928-92608418e54d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-2e9f81-rn89z\" (UID: \"223f9a36-39dc-4472-8928-92608418e54d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2e9f81-rn89z" Nov 02 07:00:53.461360 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:53.461336 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/223f9a36-39dc-4472-8928-92608418e54d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-2e9f81-rn89z\" (UID: \"223f9a36-39dc-4472-8928-92608418e54d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2e9f81-rn89z" Nov 02 07:00:53.462958 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:53.462932 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/223f9a36-39dc-4472-8928-92608418e54d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-2e9f81-rn89z\" (UID: \"223f9a36-39dc-4472-8928-92608418e54d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2e9f81-rn89z" Nov 02 07:00:53.517956 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:53.517927 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68ls9\" (UniqueName: \"kubernetes.io/projected/223f9a36-39dc-4472-8928-92608418e54d-kube-api-access-68ls9\") pod \"maas-api-revoke-keys-e2e-inf-b-2e9f81-rn89z\" (UID: \"223f9a36-39dc-4472-8928-92608418e54d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2e9f81-rn89z" Nov 02 07:00:53.626030 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:53.625896 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2e9f81-rn89z" Nov 02 07:00:53.829353 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:53.829326 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5a85b0-dxrlv" Nov 02 07:00:53.864737 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:53.864712 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34d10ad6-09c2-4135-8981-fa140ec26bb6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34d10ad6-09c2-4135-8981-fa140ec26bb6-maas-api-xks-ca\") pod \"34d10ad6-09c2-4135-8981-fa140ec26bb6\" (UID: \"34d10ad6-09c2-4135-8981-fa140ec26bb6\") " Nov 02 07:00:53.864876 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:53.864823 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34d10ad6-09c2-4135-8981-fa140ec26bb6-kube-api-access-xrkvn\" (UniqueName: \"kubernetes.io/projected/34d10ad6-09c2-4135-8981-fa140ec26bb6-kube-api-access-xrkvn\") pod \"34d10ad6-09c2-4135-8981-fa140ec26bb6\" (UID: \"34d10ad6-09c2-4135-8981-fa140ec26bb6\") " Nov 02 07:00:53.864876 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:53.864865 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34d10ad6-09c2-4135-8981-fa140ec26bb6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34d10ad6-09c2-4135-8981-fa140ec26bb6-maas-api-openshift-ca\") pod \"34d10ad6-09c2-4135-8981-fa140ec26bb6\" (UID: \"34d10ad6-09c2-4135-8981-fa140ec26bb6\") " Nov 02 07:00:53.865282 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:53.865258 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34d10ad6-09c2-4135-8981-fa140ec26bb6-maas-api-openshift-ca" pod "34d10ad6-09c2-4135-8981-fa140ec26bb6" (UID: "34d10ad6-09c2-4135-8981-fa140ec26bb6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:00:53.866820 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:53.866794 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34d10ad6-09c2-4135-8981-fa140ec26bb6-maas-api-xks-ca" pod "34d10ad6-09c2-4135-8981-fa140ec26bb6" (UID: "34d10ad6-09c2-4135-8981-fa140ec26bb6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 07:00:53.866820 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:53.866809 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34d10ad6-09c2-4135-8981-fa140ec26bb6-kube-api-access-xrkvn" pod "34d10ad6-09c2-4135-8981-fa140ec26bb6" (UID: "34d10ad6-09c2-4135-8981-fa140ec26bb6"). InnerVolumeSpecName "kube-api-access-xrkvn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:00:53.965869 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:53.965837 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrkvn\" (UniqueName: \"kubernetes.io/projected/34d10ad6-09c2-4135-8981-fa140ec26bb6-kube-api-access-xrkvn\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:53.965869 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:53.965862 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34d10ad6-09c2-4135-8981-fa140ec26bb6-maas-api-openshift-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:53.965869 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:53.965872 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34d10ad6-09c2-4135-8981-fa140ec26bb6-maas-api-xks-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:54.056386 ip-10-0-130-93 kubenswrapper[5916]: W1102 07:00:54.056355 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod223f9a36_39dc_4472_8928_92608418e54d.slice/crio-c5c8834604a441a496d50481ef8fbe10c8b15030076113e2e5f4f9e854239242 WatchSource:0}: Error finding container c5c8834604a441a496d50481ef8fbe10c8b15030076113e2e5f4f9e854239242: Status 404 returned error can't find the container with id c5c8834604a441a496d50481ef8fbe10c8b15030076113e2e5f4f9e854239242 Nov 02 07:00:54.056819 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:54.056793 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2e9f81-rn89z"] Nov 02 07:00:54.678024 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:54.677993 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5a85b0-dxrlv" Nov 02 07:00:54.678202 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:54.678022 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5a85b0-dxrlv" event={"ID":"34d10ad6-09c2-4135-8981-fa140ec26bb6","Type":"ContainerDied","Data":"5ddf5e41617661db437e359f9e8d6bd2c385d83f39d48559d14d96d414e33ef6"} Nov 02 07:00:54.678202 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:54.678093 5916 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5ddf5e41617661db437e359f9e8d6bd2c385d83f39d48559d14d96d414e33ef6" Nov 02 07:00:54.680131 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:54.680088 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2e9f81-rn89z" event={"ID":"223f9a36-39dc-4472-8928-92608418e54d","Type":"ContainerStarted","Data":"281d2cbfa8b190b7d462b680171eef0bc3227fa9f2e5fa4e46bef9413c5a9ef3"} Nov 02 07:00:54.680271 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:54.680148 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2e9f81-rn89z" event={"ID":"223f9a36-39dc-4472-8928-92608418e54d","Type":"ContainerStarted","Data":"c5c8834604a441a496d50481ef8fbe10c8b15030076113e2e5f4f9e854239242"} Nov 02 07:00:54.796726 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:54.796683 5916 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2e9f81-rn89z" podStartSLOduration=1.796670524 podStartE2EDuration="1.796670524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:00:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:00:54.795737602 +0000 UTC m=+2643.067557995" watchObservedRunningTime="2026-11-02 07:00:54.796670524 +0000 UTC m=+2643.068490918" Nov 02 07:00:55.503595 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:55.503561 5916 scope.go:122] "RemoveContainer" containerID="b66ea00bf9c33cf0075b1d2ce34c30a67aff3aa516da411c1ed4ceaecc2f5b1e" Nov 02 07:00:55.513667 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:55.513645 5916 scope.go:122] "RemoveContainer" containerID="430f3dd7caf08ceac50097a8c8899b1ded2b83ec11bea0a5d3f427d6f4f8b1ad" Nov 02 07:00:55.527663 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:55.527643 5916 scope.go:122] "RemoveContainer" containerID="802cab162f948cd147fde88be449216111acb7b026d34eae16a5a25685fe1b04" Nov 02 07:00:55.537724 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:55.537704 5916 scope.go:122] "RemoveContainer" containerID="5fc6ee3bab744309d6334dd10dcda78a68cc1317e25722f67d0f92d3d1aa96cb" Nov 02 07:00:55.688055 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:55.688026 5916 generic.go:356] "Generic (PLEG): container finished" podID="223f9a36-39dc-4472-8928-92608418e54d" containerID="281d2cbfa8b190b7d462b680171eef0bc3227fa9f2e5fa4e46bef9413c5a9ef3" exitCode=0 Nov 02 07:00:55.688184 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:55.688070 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2e9f81-rn89z" event={"ID":"223f9a36-39dc-4472-8928-92608418e54d","Type":"ContainerDied","Data":"281d2cbfa8b190b7d462b680171eef0bc3227fa9f2e5fa4e46bef9413c5a9ef3"} Nov 02 07:00:56.836818 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:56.836788 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2e9f81-rn89z" Nov 02 07:00:56.896004 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:56.895976 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/223f9a36-39dc-4472-8928-92608418e54d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/223f9a36-39dc-4472-8928-92608418e54d-maas-api-xks-ca\") pod \"223f9a36-39dc-4472-8928-92608418e54d\" (UID: \"223f9a36-39dc-4472-8928-92608418e54d\") " Nov 02 07:00:56.896130 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:56.896012 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/223f9a36-39dc-4472-8928-92608418e54d-kube-api-access-68ls9\" (UniqueName: \"kubernetes.io/projected/223f9a36-39dc-4472-8928-92608418e54d-kube-api-access-68ls9\") pod \"223f9a36-39dc-4472-8928-92608418e54d\" (UID: \"223f9a36-39dc-4472-8928-92608418e54d\") " Nov 02 07:00:56.896174 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:56.896132 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/223f9a36-39dc-4472-8928-92608418e54d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/223f9a36-39dc-4472-8928-92608418e54d-maas-api-openshift-ca\") pod \"223f9a36-39dc-4472-8928-92608418e54d\" (UID: \"223f9a36-39dc-4472-8928-92608418e54d\") " Nov 02 07:00:56.896505 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:56.896475 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/223f9a36-39dc-4472-8928-92608418e54d-maas-api-openshift-ca" pod "223f9a36-39dc-4472-8928-92608418e54d" (UID: "223f9a36-39dc-4472-8928-92608418e54d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:00:56.897929 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:56.897896 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/223f9a36-39dc-4472-8928-92608418e54d-maas-api-xks-ca" pod "223f9a36-39dc-4472-8928-92608418e54d" (UID: "223f9a36-39dc-4472-8928-92608418e54d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 07:00:56.898168 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:56.898153 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/223f9a36-39dc-4472-8928-92608418e54d-kube-api-access-68ls9" pod "223f9a36-39dc-4472-8928-92608418e54d" (UID: "223f9a36-39dc-4472-8928-92608418e54d"). InnerVolumeSpecName "kube-api-access-68ls9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:00:56.997284 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:56.997219 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/223f9a36-39dc-4472-8928-92608418e54d-maas-api-xks-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:56.997284 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:56.997245 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68ls9\" (UniqueName: \"kubernetes.io/projected/223f9a36-39dc-4472-8928-92608418e54d-kube-api-access-68ls9\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:56.997284 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:56.997256 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/223f9a36-39dc-4472-8928-92608418e54d-maas-api-openshift-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:57.706496 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:57.706467 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2e9f81-rn89z" Nov 02 07:00:57.706713 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:57.706502 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2e9f81-rn89z" event={"ID":"223f9a36-39dc-4472-8928-92608418e54d","Type":"ContainerDied","Data":"c5c8834604a441a496d50481ef8fbe10c8b15030076113e2e5f4f9e854239242"} Nov 02 07:00:57.706713 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:00:57.706549 5916 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c5c8834604a441a496d50481ef8fbe10c8b15030076113e2e5f4f9e854239242" Nov 02 07:01:19.893029 ip-10-0-130-93 kubenswrapper[5916]: E1102 07:01:19.892995 5916 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p98cq" containerName="extractor" Nov 02 07:01:20.341033 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:20.340949 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-581608-mlz9r"] Nov 02 07:01:20.390418 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:20.390384 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-581608-mlz9r"] Nov 02 07:01:21.535522 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:21.535494 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5a85b0-dxrlv"] Nov 02 07:01:21.584231 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:21.584202 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5a85b0-dxrlv"] Nov 02 07:01:21.843317 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:21.843241 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1837c8a1-277f-4319-8e92-6e9e27e698c5" path="/var/lib/kubelet/pods/1837c8a1-277f-4319-8e92-6e9e27e698c5/volumes" Nov 02 07:01:21.843817 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:21.843799 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34d10ad6-09c2-4135-8981-fa140ec26bb6" path="/var/lib/kubelet/pods/34d10ad6-09c2-4135-8981-fa140ec26bb6/volumes" Nov 02 07:01:22.706024 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:22.705989 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2e9f81-rn89z"] Nov 02 07:01:22.756045 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:22.756013 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2e9f81-rn89z"] Nov 02 07:01:23.841979 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:23.841950 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="223f9a36-39dc-4472-8928-92608418e54d" path="/var/lib/kubelet/pods/223f9a36-39dc-4472-8928-92608418e54d/volumes" Nov 02 07:01:27.033484 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:27.033443 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-9mj97"] Nov 02 07:01:27.033947 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:27.033692 5916 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-9mj97" podUID="9ee541ef-78e0-4075-9780-eb53b607ddcc" containerName="istio-proxy" containerID="cri-o://f1fee98fec7a45b64284962d0f5ea124ca899e15fc4bb4e65fb4ab30aca40d76" gracePeriod=30 Nov 02 07:01:29.541153 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:29.541115 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-cdfce9-5fwjj"] Nov 02 07:01:29.541749 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:29.541728 5916 cpu_manager.go:414] "containerMap: removing container" podUID="223f9a36-39dc-4472-8928-92608418e54d" containerName="revoke-keys" Nov 02 07:01:29.541846 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:29.541752 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="223f9a36-39dc-4472-8928-92608418e54d" containerName="revoke-keys" Nov 02 07:01:29.541846 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:29.541772 5916 cpu_manager.go:414] "containerMap: removing container" podUID="34d10ad6-09c2-4135-8981-fa140ec26bb6" containerName="revoke-keys" Nov 02 07:01:29.541846 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:29.541780 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34d10ad6-09c2-4135-8981-fa140ec26bb6" containerName="revoke-keys" Nov 02 07:01:29.542022 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:29.541929 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="223f9a36-39dc-4472-8928-92608418e54d" containerName="revoke-keys" Nov 02 07:01:29.542022 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:29.541949 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="34d10ad6-09c2-4135-8981-fa140ec26bb6" containerName="revoke-keys" Nov 02 07:01:29.546146 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:29.546127 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-cdfce9-5fwjj" Nov 02 07:01:29.600821 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:29.600796 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-cdfce9-5fwjj"] Nov 02 07:01:29.602001 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:29.601980 5916 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 07:01:29.647618 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:29.647592 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 07:01:29.647618 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:29.647630 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-68l4r\"" Nov 02 07:01:29.647825 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:29.647651 5916 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 07:01:29.719834 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:29.719804 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e2fdb61-3a2c-46df-92ff-27547f130b0f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-cdfce9-5fwjj\" (UID: \"2e2fdb61-3a2c-46df-92ff-27547f130b0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-cdfce9-5fwjj" Nov 02 07:01:29.720024 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:29.719885 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e2fdb61-3a2c-46df-92ff-27547f130b0f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-cdfce9-5fwjj\" (UID: \"2e2fdb61-3a2c-46df-92ff-27547f130b0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-cdfce9-5fwjj" Nov 02 07:01:29.720024 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:29.720008 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hp89v\" (UniqueName: \"kubernetes.io/projected/2e2fdb61-3a2c-46df-92ff-27547f130b0f-kube-api-access-hp89v\") pod \"maas-api-revoke-keys-e2e-worker-w1-cdfce9-5fwjj\" (UID: \"2e2fdb61-3a2c-46df-92ff-27547f130b0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-cdfce9-5fwjj" Nov 02 07:01:29.821563 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:29.821488 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hp89v\" (UniqueName: \"kubernetes.io/projected/2e2fdb61-3a2c-46df-92ff-27547f130b0f-kube-api-access-hp89v\") pod \"maas-api-revoke-keys-e2e-worker-w1-cdfce9-5fwjj\" (UID: \"2e2fdb61-3a2c-46df-92ff-27547f130b0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-cdfce9-5fwjj" Nov 02 07:01:29.821563 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:29.821527 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e2fdb61-3a2c-46df-92ff-27547f130b0f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-cdfce9-5fwjj\" (UID: \"2e2fdb61-3a2c-46df-92ff-27547f130b0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-cdfce9-5fwjj" Nov 02 07:01:29.821764 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:29.821577 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e2fdb61-3a2c-46df-92ff-27547f130b0f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-cdfce9-5fwjj\" (UID: \"2e2fdb61-3a2c-46df-92ff-27547f130b0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-cdfce9-5fwjj" Nov 02 07:01:29.822216 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:29.822193 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e2fdb61-3a2c-46df-92ff-27547f130b0f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-cdfce9-5fwjj\" (UID: \"2e2fdb61-3a2c-46df-92ff-27547f130b0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-cdfce9-5fwjj" Nov 02 07:01:29.823939 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:29.823894 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e2fdb61-3a2c-46df-92ff-27547f130b0f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-cdfce9-5fwjj\" (UID: \"2e2fdb61-3a2c-46df-92ff-27547f130b0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-cdfce9-5fwjj" Nov 02 07:01:29.890152 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:29.890116 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hp89v\" (UniqueName: \"kubernetes.io/projected/2e2fdb61-3a2c-46df-92ff-27547f130b0f-kube-api-access-hp89v\") pod \"maas-api-revoke-keys-e2e-worker-w1-cdfce9-5fwjj\" (UID: \"2e2fdb61-3a2c-46df-92ff-27547f130b0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-cdfce9-5fwjj" Nov 02 07:01:30.158676 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:30.158637 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-cdfce9-5fwjj" Nov 02 07:01:30.551072 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:30.551047 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-cdfce9-5fwjj"] Nov 02 07:01:30.554482 ip-10-0-130-93 kubenswrapper[5916]: W1102 07:01:30.554449 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e2fdb61_3a2c_46df_92ff_27547f130b0f.slice/crio-55388ab792da142f7553bfcc93e22cca795b1865677a164a7cf7ba222d14ed57 WatchSource:0}: Error finding container 55388ab792da142f7553bfcc93e22cca795b1865677a164a7cf7ba222d14ed57: Status 404 returned error can't find the container with id 55388ab792da142f7553bfcc93e22cca795b1865677a164a7cf7ba222d14ed57 Nov 02 07:01:30.931959 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:30.931921 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-cdfce9-5fwjj" event={"ID":"2e2fdb61-3a2c-46df-92ff-27547f130b0f","Type":"ContainerStarted","Data":"c97181451da2384040f654c5060d6441f23ebf964c6fd8ad8a0be1849b0d59b7"} Nov 02 07:01:30.932159 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:30.931986 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-cdfce9-5fwjj" event={"ID":"2e2fdb61-3a2c-46df-92ff-27547f130b0f","Type":"ContainerStarted","Data":"55388ab792da142f7553bfcc93e22cca795b1865677a164a7cf7ba222d14ed57"} Nov 02 07:01:30.936928 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:30.936880 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-3e96be-skgc9"] Nov 02 07:01:30.941375 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:30.941356 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-3e96be-skgc9" Nov 02 07:01:30.992294 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:30.992250 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-3e96be-skgc9"] Nov 02 07:01:31.031473 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:31.031436 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74e1873c-18db-4d6e-ad41-44ca49a732bd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-3e96be-skgc9\" (UID: \"74e1873c-18db-4d6e-ad41-44ca49a732bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-3e96be-skgc9" Nov 02 07:01:31.031647 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:31.031509 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74e1873c-18db-4d6e-ad41-44ca49a732bd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-3e96be-skgc9\" (UID: \"74e1873c-18db-4d6e-ad41-44ca49a732bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-3e96be-skgc9" Nov 02 07:01:31.031647 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:31.031622 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xz7r4\" (UniqueName: \"kubernetes.io/projected/74e1873c-18db-4d6e-ad41-44ca49a732bd-kube-api-access-xz7r4\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-3e96be-skgc9\" (UID: \"74e1873c-18db-4d6e-ad41-44ca49a732bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-3e96be-skgc9" Nov 02 07:01:31.054689 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:31.054643 5916 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-cdfce9-5fwjj" podStartSLOduration=2.054631446 podStartE2EDuration="2.054631446s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:01:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:01:31.052288268 +0000 UTC m=+2679.324108663" watchObservedRunningTime="2026-11-02 07:01:31.054631446 +0000 UTC m=+2679.326451831" Nov 02 07:01:31.132490 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:31.132410 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74e1873c-18db-4d6e-ad41-44ca49a732bd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-3e96be-skgc9\" (UID: \"74e1873c-18db-4d6e-ad41-44ca49a732bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-3e96be-skgc9" Nov 02 07:01:31.132490 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:31.132443 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74e1873c-18db-4d6e-ad41-44ca49a732bd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-3e96be-skgc9\" (UID: \"74e1873c-18db-4d6e-ad41-44ca49a732bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-3e96be-skgc9" Nov 02 07:01:31.132697 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:31.132505 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xz7r4\" (UniqueName: \"kubernetes.io/projected/74e1873c-18db-4d6e-ad41-44ca49a732bd-kube-api-access-xz7r4\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-3e96be-skgc9\" (UID: \"74e1873c-18db-4d6e-ad41-44ca49a732bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-3e96be-skgc9" Nov 02 07:01:31.133231 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:31.133209 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74e1873c-18db-4d6e-ad41-44ca49a732bd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-3e96be-skgc9\" (UID: \"74e1873c-18db-4d6e-ad41-44ca49a732bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-3e96be-skgc9" Nov 02 07:01:31.134902 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:31.134876 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74e1873c-18db-4d6e-ad41-44ca49a732bd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-3e96be-skgc9\" (UID: \"74e1873c-18db-4d6e-ad41-44ca49a732bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-3e96be-skgc9" Nov 02 07:01:31.190340 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:31.190316 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xz7r4\" (UniqueName: \"kubernetes.io/projected/74e1873c-18db-4d6e-ad41-44ca49a732bd-kube-api-access-xz7r4\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-3e96be-skgc9\" (UID: \"74e1873c-18db-4d6e-ad41-44ca49a732bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-3e96be-skgc9" Nov 02 07:01:31.256629 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:31.256598 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-3e96be-skgc9" Nov 02 07:01:31.633192 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:31.633162 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-3e96be-skgc9"] Nov 02 07:01:31.634843 ip-10-0-130-93 kubenswrapper[5916]: W1102 07:01:31.634815 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74e1873c_18db_4d6e_ad41_44ca49a732bd.slice/crio-014241a3640cd7d03df14113724a6d49c66efeb4420ce45241dec6ea43d22307 WatchSource:0}: Error finding container 014241a3640cd7d03df14113724a6d49c66efeb4420ce45241dec6ea43d22307: Status 404 returned error can't find the container with id 014241a3640cd7d03df14113724a6d49c66efeb4420ce45241dec6ea43d22307 Nov 02 07:01:31.942921 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:31.942877 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-3e96be-skgc9" event={"ID":"74e1873c-18db-4d6e-ad41-44ca49a732bd","Type":"ContainerStarted","Data":"014241a3640cd7d03df14113724a6d49c66efeb4420ce45241dec6ea43d22307"} Nov 02 07:01:31.944482 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:31.944459 5916 generic.go:356] "Generic (PLEG): container finished" podID="2e2fdb61-3a2c-46df-92ff-27547f130b0f" containerID="c97181451da2384040f654c5060d6441f23ebf964c6fd8ad8a0be1849b0d59b7" exitCode=0 Nov 02 07:01:31.944571 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:31.944533 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-cdfce9-5fwjj" event={"ID":"2e2fdb61-3a2c-46df-92ff-27547f130b0f","Type":"ContainerDied","Data":"c97181451da2384040f654c5060d6441f23ebf964c6fd8ad8a0be1849b0d59b7"} Nov 02 07:01:32.289222 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:32.289199 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-9mj97" Nov 02 07:01:32.342332 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:32.342307 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ee541ef-78e0-4075-9780-eb53b607ddcc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ee541ef-78e0-4075-9780-eb53b607ddcc-workload-socket\") pod \"9ee541ef-78e0-4075-9780-eb53b607ddcc\" (UID: \"9ee541ef-78e0-4075-9780-eb53b607ddcc\") " Nov 02 07:01:32.342480 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:32.342372 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ee541ef-78e0-4075-9780-eb53b607ddcc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ee541ef-78e0-4075-9780-eb53b607ddcc-istio-data\") pod \"9ee541ef-78e0-4075-9780-eb53b607ddcc\" (UID: \"9ee541ef-78e0-4075-9780-eb53b607ddcc\") " Nov 02 07:01:32.342480 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:32.342433 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9ee541ef-78e0-4075-9780-eb53b607ddcc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ee541ef-78e0-4075-9780-eb53b607ddcc-istio-podinfo\") pod \"9ee541ef-78e0-4075-9780-eb53b607ddcc\" (UID: \"9ee541ef-78e0-4075-9780-eb53b607ddcc\") " Nov 02 07:01:32.342480 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:32.342461 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ee541ef-78e0-4075-9780-eb53b607ddcc-kube-api-access-zmdzl\" (UniqueName: \"kubernetes.io/projected/9ee541ef-78e0-4075-9780-eb53b607ddcc-kube-api-access-zmdzl\") pod \"9ee541ef-78e0-4075-9780-eb53b607ddcc\" (UID: \"9ee541ef-78e0-4075-9780-eb53b607ddcc\") " Nov 02 07:01:32.342659 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:32.342497 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ee541ef-78e0-4075-9780-eb53b607ddcc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ee541ef-78e0-4075-9780-eb53b607ddcc-istio-envoy\") pod \"9ee541ef-78e0-4075-9780-eb53b607ddcc\" (UID: \"9ee541ef-78e0-4075-9780-eb53b607ddcc\") " Nov 02 07:01:32.342659 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:32.342544 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ee541ef-78e0-4075-9780-eb53b607ddcc-workload-socket" pod "9ee541ef-78e0-4075-9780-eb53b607ddcc" (UID: "9ee541ef-78e0-4075-9780-eb53b607ddcc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:01:32.342659 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:32.342628 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ee541ef-78e0-4075-9780-eb53b607ddcc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ee541ef-78e0-4075-9780-eb53b607ddcc-istiod-ca-cert\") pod \"9ee541ef-78e0-4075-9780-eb53b607ddcc\" (UID: \"9ee541ef-78e0-4075-9780-eb53b607ddcc\") " Nov 02 07:01:32.342821 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:32.342635 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ee541ef-78e0-4075-9780-eb53b607ddcc-istio-data" pod "9ee541ef-78e0-4075-9780-eb53b607ddcc" (UID: "9ee541ef-78e0-4075-9780-eb53b607ddcc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:01:32.342821 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:32.342720 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ee541ef-78e0-4075-9780-eb53b607ddcc-istio-token\" (UniqueName: \"kubernetes.io/projected/9ee541ef-78e0-4075-9780-eb53b607ddcc-istio-token\") pod \"9ee541ef-78e0-4075-9780-eb53b607ddcc\" (UID: \"9ee541ef-78e0-4075-9780-eb53b607ddcc\") " Nov 02 07:01:32.342821 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:32.342766 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ee541ef-78e0-4075-9780-eb53b607ddcc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ee541ef-78e0-4075-9780-eb53b607ddcc-workload-certs\") pod \"9ee541ef-78e0-4075-9780-eb53b607ddcc\" (UID: \"9ee541ef-78e0-4075-9780-eb53b607ddcc\") " Nov 02 07:01:32.342821 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:32.342810 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ee541ef-78e0-4075-9780-eb53b607ddcc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ee541ef-78e0-4075-9780-eb53b607ddcc-credential-socket\") pod \"9ee541ef-78e0-4075-9780-eb53b607ddcc\" (UID: \"9ee541ef-78e0-4075-9780-eb53b607ddcc\") " Nov 02 07:01:32.343057 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:32.342920 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ee541ef-78e0-4075-9780-eb53b607ddcc-istiod-ca-cert" pod "9ee541ef-78e0-4075-9780-eb53b607ddcc" (UID: "9ee541ef-78e0-4075-9780-eb53b607ddcc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:01:32.343191 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:32.343171 5916 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ee541ef-78e0-4075-9780-eb53b607ddcc-istio-data\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:01:32.343191 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:32.343180 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ee541ef-78e0-4075-9780-eb53b607ddcc-workload-certs" pod "9ee541ef-78e0-4075-9780-eb53b607ddcc" (UID: "9ee541ef-78e0-4075-9780-eb53b607ddcc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:01:32.343319 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:32.343195 5916 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ee541ef-78e0-4075-9780-eb53b607ddcc-istiod-ca-cert\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:01:32.343319 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:32.343210 5916 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ee541ef-78e0-4075-9780-eb53b607ddcc-workload-socket\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:01:32.343435 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:32.343421 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ee541ef-78e0-4075-9780-eb53b607ddcc-credential-socket" pod "9ee541ef-78e0-4075-9780-eb53b607ddcc" (UID: "9ee541ef-78e0-4075-9780-eb53b607ddcc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:01:32.344744 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:32.344722 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9ee541ef-78e0-4075-9780-eb53b607ddcc-istio-podinfo" pod "9ee541ef-78e0-4075-9780-eb53b607ddcc" (UID: "9ee541ef-78e0-4075-9780-eb53b607ddcc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 07:01:32.344859 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:32.344788 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ee541ef-78e0-4075-9780-eb53b607ddcc-istio-envoy" pod "9ee541ef-78e0-4075-9780-eb53b607ddcc" (UID: "9ee541ef-78e0-4075-9780-eb53b607ddcc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:01:32.344859 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:32.344800 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ee541ef-78e0-4075-9780-eb53b607ddcc-istio-token" pod "9ee541ef-78e0-4075-9780-eb53b607ddcc" (UID: "9ee541ef-78e0-4075-9780-eb53b607ddcc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:01:32.344859 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:32.344847 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ee541ef-78e0-4075-9780-eb53b607ddcc-kube-api-access-zmdzl" pod "9ee541ef-78e0-4075-9780-eb53b607ddcc" (UID: "9ee541ef-78e0-4075-9780-eb53b607ddcc"). InnerVolumeSpecName "kube-api-access-zmdzl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:01:32.443939 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:32.443886 5916 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ee541ef-78e0-4075-9780-eb53b607ddcc-istio-podinfo\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:01:32.443939 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:32.443936 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmdzl\" (UniqueName: \"kubernetes.io/projected/9ee541ef-78e0-4075-9780-eb53b607ddcc-kube-api-access-zmdzl\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:01:32.444134 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:32.443954 5916 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ee541ef-78e0-4075-9780-eb53b607ddcc-istio-envoy\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:01:32.444134 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:32.443968 5916 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ee541ef-78e0-4075-9780-eb53b607ddcc-istio-token\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:01:32.444134 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:32.443982 5916 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ee541ef-78e0-4075-9780-eb53b607ddcc-workload-certs\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:01:32.444134 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:32.443996 5916 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ee541ef-78e0-4075-9780-eb53b607ddcc-credential-socket\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:01:32.952385 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:32.952353 5916 generic.go:356] "Generic (PLEG): container finished" podID="9ee541ef-78e0-4075-9780-eb53b607ddcc" containerID="f1fee98fec7a45b64284962d0f5ea124ca899e15fc4bb4e65fb4ab30aca40d76" exitCode=0 Nov 02 07:01:32.952842 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:32.952404 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-9mj97" Nov 02 07:01:32.952842 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:32.952443 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-9mj97" event={"ID":"9ee541ef-78e0-4075-9780-eb53b607ddcc","Type":"ContainerDied","Data":"f1fee98fec7a45b64284962d0f5ea124ca899e15fc4bb4e65fb4ab30aca40d76"} Nov 02 07:01:32.952842 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:32.952497 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-9mj97" event={"ID":"9ee541ef-78e0-4075-9780-eb53b607ddcc","Type":"ContainerDied","Data":"7f9381e29b4e774b1697710c4268a04db70e6a6bdf68145e24f5e2a3026ef571"} Nov 02 07:01:32.952842 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:32.952521 5916 scope.go:122] "RemoveContainer" containerID="f1fee98fec7a45b64284962d0f5ea124ca899e15fc4bb4e65fb4ab30aca40d76" Nov 02 07:01:32.954323 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:32.954298 5916 generic.go:356] "Generic (PLEG): container finished" podID="74e1873c-18db-4d6e-ad41-44ca49a732bd" containerID="a77e27297af4c2bfad519fa2596405e0714eb0dcd44511086a8db15d184a74f8" exitCode=0 Nov 02 07:01:32.954434 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:32.954384 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-3e96be-skgc9" event={"ID":"74e1873c-18db-4d6e-ad41-44ca49a732bd","Type":"ContainerDied","Data":"a77e27297af4c2bfad519fa2596405e0714eb0dcd44511086a8db15d184a74f8"} Nov 02 07:01:32.963729 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:32.963711 5916 scope.go:122] "RemoveContainer" containerID="f1fee98fec7a45b64284962d0f5ea124ca899e15fc4bb4e65fb4ab30aca40d76" Nov 02 07:01:32.964028 ip-10-0-130-93 kubenswrapper[5916]: E1102 07:01:32.964008 5916 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1fee98fec7a45b64284962d0f5ea124ca899e15fc4bb4e65fb4ab30aca40d76\": container with ID starting with f1fee98fec7a45b64284962d0f5ea124ca899e15fc4bb4e65fb4ab30aca40d76 not found: ID does not exist" containerID="f1fee98fec7a45b64284962d0f5ea124ca899e15fc4bb4e65fb4ab30aca40d76" Nov 02 07:01:32.964080 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:32.964037 5916 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1fee98fec7a45b64284962d0f5ea124ca899e15fc4bb4e65fb4ab30aca40d76"} err="failed to get container status \"f1fee98fec7a45b64284962d0f5ea124ca899e15fc4bb4e65fb4ab30aca40d76\": rpc error: code = NotFound desc = could not find container \"f1fee98fec7a45b64284962d0f5ea124ca899e15fc4bb4e65fb4ab30aca40d76\": container with ID starting with f1fee98fec7a45b64284962d0f5ea124ca899e15fc4bb4e65fb4ab30aca40d76 not found: ID does not exist" Nov 02 07:01:33.092994 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:33.092934 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-cdfce9-5fwjj" Nov 02 07:01:33.150295 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:33.150267 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e2fdb61-3a2c-46df-92ff-27547f130b0f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e2fdb61-3a2c-46df-92ff-27547f130b0f-maas-api-xks-ca\") pod \"2e2fdb61-3a2c-46df-92ff-27547f130b0f\" (UID: \"2e2fdb61-3a2c-46df-92ff-27547f130b0f\") " Nov 02 07:01:33.150441 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:33.150356 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e2fdb61-3a2c-46df-92ff-27547f130b0f-kube-api-access-hp89v\" (UniqueName: \"kubernetes.io/projected/2e2fdb61-3a2c-46df-92ff-27547f130b0f-kube-api-access-hp89v\") pod \"2e2fdb61-3a2c-46df-92ff-27547f130b0f\" (UID: \"2e2fdb61-3a2c-46df-92ff-27547f130b0f\") " Nov 02 07:01:33.150511 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:33.150447 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e2fdb61-3a2c-46df-92ff-27547f130b0f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e2fdb61-3a2c-46df-92ff-27547f130b0f-maas-api-openshift-ca\") pod \"2e2fdb61-3a2c-46df-92ff-27547f130b0f\" (UID: \"2e2fdb61-3a2c-46df-92ff-27547f130b0f\") " Nov 02 07:01:33.150814 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:33.150793 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e2fdb61-3a2c-46df-92ff-27547f130b0f-maas-api-openshift-ca" pod "2e2fdb61-3a2c-46df-92ff-27547f130b0f" (UID: "2e2fdb61-3a2c-46df-92ff-27547f130b0f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:01:33.152178 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:33.152154 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e2fdb61-3a2c-46df-92ff-27547f130b0f-maas-api-xks-ca" pod "2e2fdb61-3a2c-46df-92ff-27547f130b0f" (UID: "2e2fdb61-3a2c-46df-92ff-27547f130b0f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 07:01:33.152308 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:33.152294 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e2fdb61-3a2c-46df-92ff-27547f130b0f-kube-api-access-hp89v" pod "2e2fdb61-3a2c-46df-92ff-27547f130b0f" (UID: "2e2fdb61-3a2c-46df-92ff-27547f130b0f"). InnerVolumeSpecName "kube-api-access-hp89v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:01:33.232224 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:33.232197 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-9mj97"] Nov 02 07:01:33.251427 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:33.251397 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e2fdb61-3a2c-46df-92ff-27547f130b0f-maas-api-openshift-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:01:33.251427 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:33.251422 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e2fdb61-3a2c-46df-92ff-27547f130b0f-maas-api-xks-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:01:33.251612 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:33.251436 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hp89v\" (UniqueName: \"kubernetes.io/projected/2e2fdb61-3a2c-46df-92ff-27547f130b0f-kube-api-access-hp89v\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:01:33.281973 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:33.281942 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-9mj97"] Nov 02 07:01:33.852377 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:33.852344 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ee541ef-78e0-4075-9780-eb53b607ddcc" path="/var/lib/kubelet/pods/9ee541ef-78e0-4075-9780-eb53b607ddcc/volumes" Nov 02 07:01:33.962305 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:33.962268 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-cdfce9-5fwjj" Nov 02 07:01:33.962804 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:33.962276 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-cdfce9-5fwjj" event={"ID":"2e2fdb61-3a2c-46df-92ff-27547f130b0f","Type":"ContainerDied","Data":"55388ab792da142f7553bfcc93e22cca795b1865677a164a7cf7ba222d14ed57"} Nov 02 07:01:33.962804 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:33.962400 5916 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="55388ab792da142f7553bfcc93e22cca795b1865677a164a7cf7ba222d14ed57" Nov 02 07:01:33.993485 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:33.993446 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-cdfce9/premium-w1-cdfce9-kserve-598cb9d57f-mrzrz"] Nov 02 07:01:34.002661 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:34.002631 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-cdfce9/simulator-w1-cdfce9-kserve-7ffd68f78-5vwl2"] Nov 02 07:01:34.125019 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:34.124979 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-3e96be-skgc9" Nov 02 07:01:34.159003 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:34.158973 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74e1873c-18db-4d6e-ad41-44ca49a732bd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74e1873c-18db-4d6e-ad41-44ca49a732bd-maas-api-openshift-ca\") pod \"74e1873c-18db-4d6e-ad41-44ca49a732bd\" (UID: \"74e1873c-18db-4d6e-ad41-44ca49a732bd\") " Nov 02 07:01:34.159151 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:34.159101 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74e1873c-18db-4d6e-ad41-44ca49a732bd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74e1873c-18db-4d6e-ad41-44ca49a732bd-maas-api-xks-ca\") pod \"74e1873c-18db-4d6e-ad41-44ca49a732bd\" (UID: \"74e1873c-18db-4d6e-ad41-44ca49a732bd\") " Nov 02 07:01:34.159212 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:34.159193 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74e1873c-18db-4d6e-ad41-44ca49a732bd-kube-api-access-xz7r4\" (UniqueName: \"kubernetes.io/projected/74e1873c-18db-4d6e-ad41-44ca49a732bd-kube-api-access-xz7r4\") pod \"74e1873c-18db-4d6e-ad41-44ca49a732bd\" (UID: \"74e1873c-18db-4d6e-ad41-44ca49a732bd\") " Nov 02 07:01:34.159506 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:34.159481 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74e1873c-18db-4d6e-ad41-44ca49a732bd-maas-api-openshift-ca" pod "74e1873c-18db-4d6e-ad41-44ca49a732bd" (UID: "74e1873c-18db-4d6e-ad41-44ca49a732bd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:01:34.161279 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:34.161235 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74e1873c-18db-4d6e-ad41-44ca49a732bd-maas-api-xks-ca" pod "74e1873c-18db-4d6e-ad41-44ca49a732bd" (UID: "74e1873c-18db-4d6e-ad41-44ca49a732bd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 07:01:34.161386 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:34.161282 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74e1873c-18db-4d6e-ad41-44ca49a732bd-kube-api-access-xz7r4" pod "74e1873c-18db-4d6e-ad41-44ca49a732bd" (UID: "74e1873c-18db-4d6e-ad41-44ca49a732bd"). InnerVolumeSpecName "kube-api-access-xz7r4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:01:34.260881 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:34.260854 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74e1873c-18db-4d6e-ad41-44ca49a732bd-maas-api-xks-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:01:34.260881 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:34.260877 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xz7r4\" (UniqueName: \"kubernetes.io/projected/74e1873c-18db-4d6e-ad41-44ca49a732bd-kube-api-access-xz7r4\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:01:34.260881 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:34.260888 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74e1873c-18db-4d6e-ad41-44ca49a732bd-maas-api-openshift-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:01:34.972101 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:34.972072 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-3e96be-skgc9" Nov 02 07:01:34.972513 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:34.972072 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-3e96be-skgc9" event={"ID":"74e1873c-18db-4d6e-ad41-44ca49a732bd","Type":"ContainerDied","Data":"014241a3640cd7d03df14113724a6d49c66efeb4420ce45241dec6ea43d22307"} Nov 02 07:01:34.972513 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:34.972206 5916 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="014241a3640cd7d03df14113724a6d49c66efeb4420ce45241dec6ea43d22307" Nov 02 07:01:41.071802 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:41.071769 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa9aa932-ztxp2"] Nov 02 07:01:41.072210 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:41.072184 5916 cpu_manager.go:414] "containerMap: removing container" podUID="9ee541ef-78e0-4075-9780-eb53b607ddcc" containerName="istio-proxy" Nov 02 07:01:41.072210 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:41.072196 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ee541ef-78e0-4075-9780-eb53b607ddcc" containerName="istio-proxy" Nov 02 07:01:41.072289 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:41.072219 5916 cpu_manager.go:414] "containerMap: removing container" podUID="2e2fdb61-3a2c-46df-92ff-27547f130b0f" containerName="revoke-keys" Nov 02 07:01:41.072289 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:41.072225 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e2fdb61-3a2c-46df-92ff-27547f130b0f" containerName="revoke-keys" Nov 02 07:01:41.072289 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:41.072248 5916 cpu_manager.go:414] "containerMap: removing container" podUID="74e1873c-18db-4d6e-ad41-44ca49a732bd" containerName="revoke-keys" Nov 02 07:01:41.072289 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:41.072253 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74e1873c-18db-4d6e-ad41-44ca49a732bd" containerName="revoke-keys" Nov 02 07:01:41.072417 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:41.072317 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="74e1873c-18db-4d6e-ad41-44ca49a732bd" containerName="revoke-keys" Nov 02 07:01:41.072417 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:41.072325 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ee541ef-78e0-4075-9780-eb53b607ddcc" containerName="istio-proxy" Nov 02 07:01:41.072417 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:41.072336 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e2fdb61-3a2c-46df-92ff-27547f130b0f" containerName="revoke-keys" Nov 02 07:01:41.075734 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:41.075718 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa9aa932-ztxp2" Nov 02 07:01:41.124664 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:41.124642 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36233f6d-5ba6-4f5d-b148-da1ff598286b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa9aa932-ztxp2\" (UID: \"36233f6d-5ba6-4f5d-b148-da1ff598286b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa9aa932-ztxp2" Nov 02 07:01:41.124787 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:41.124690 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmctq\" (UniqueName: \"kubernetes.io/projected/36233f6d-5ba6-4f5d-b148-da1ff598286b-kube-api-access-gmctq\") pod \"maas-api-revoke-sub-aa9aa932-ztxp2\" (UID: \"36233f6d-5ba6-4f5d-b148-da1ff598286b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa9aa932-ztxp2" Nov 02 07:01:41.124787 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:41.124709 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36233f6d-5ba6-4f5d-b148-da1ff598286b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa9aa932-ztxp2\" (UID: \"36233f6d-5ba6-4f5d-b148-da1ff598286b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa9aa932-ztxp2" Nov 02 07:01:41.125997 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:41.125977 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f4e008a-8n8kr"] Nov 02 07:01:41.129601 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:41.129587 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f4e008a-8n8kr" Nov 02 07:01:41.173322 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:41.173295 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 07:01:41.173322 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:41.173311 5916 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 07:01:41.178484 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:41.178457 5916 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 07:01:41.178597 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:41.178547 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-68l4r\"" Nov 02 07:01:41.181264 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:41.181242 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa9aa932-ztxp2"] Nov 02 07:01:41.231691 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:41.231657 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83ec9f9d-b0b5-4eaa-aa0d-69c43c283c2d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5f4e008a-8n8kr\" (UID: \"83ec9f9d-b0b5-4eaa-aa0d-69c43c283c2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f4e008a-8n8kr" Nov 02 07:01:41.231859 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:41.231786 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36233f6d-5ba6-4f5d-b148-da1ff598286b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa9aa932-ztxp2\" (UID: \"36233f6d-5ba6-4f5d-b148-da1ff598286b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa9aa932-ztxp2" Nov 02 07:01:41.231944 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:41.231884 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmctq\" (UniqueName: \"kubernetes.io/projected/36233f6d-5ba6-4f5d-b148-da1ff598286b-kube-api-access-gmctq\") pod \"maas-api-revoke-sub-aa9aa932-ztxp2\" (UID: \"36233f6d-5ba6-4f5d-b148-da1ff598286b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa9aa932-ztxp2" Nov 02 07:01:41.232002 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:41.231940 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36233f6d-5ba6-4f5d-b148-da1ff598286b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa9aa932-ztxp2\" (UID: \"36233f6d-5ba6-4f5d-b148-da1ff598286b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa9aa932-ztxp2" Nov 02 07:01:41.232002 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:41.231971 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83ec9f9d-b0b5-4eaa-aa0d-69c43c283c2d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5f4e008a-8n8kr\" (UID: \"83ec9f9d-b0b5-4eaa-aa0d-69c43c283c2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f4e008a-8n8kr" Nov 02 07:01:41.232120 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:41.232032 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmfnh\" (UniqueName: \"kubernetes.io/projected/83ec9f9d-b0b5-4eaa-aa0d-69c43c283c2d-kube-api-access-mmfnh\") pod \"maas-api-revoke-sub-5f4e008a-8n8kr\" (UID: \"83ec9f9d-b0b5-4eaa-aa0d-69c43c283c2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f4e008a-8n8kr" Nov 02 07:01:41.232977 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:41.232952 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36233f6d-5ba6-4f5d-b148-da1ff598286b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa9aa932-ztxp2\" (UID: \"36233f6d-5ba6-4f5d-b148-da1ff598286b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa9aa932-ztxp2" Nov 02 07:01:41.236144 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:41.236122 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36233f6d-5ba6-4f5d-b148-da1ff598286b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa9aa932-ztxp2\" (UID: \"36233f6d-5ba6-4f5d-b148-da1ff598286b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa9aa932-ztxp2" Nov 02 07:01:41.239497 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:41.239475 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f4e008a-8n8kr"] Nov 02 07:01:41.289391 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:41.289365 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmctq\" (UniqueName: \"kubernetes.io/projected/36233f6d-5ba6-4f5d-b148-da1ff598286b-kube-api-access-gmctq\") pod \"maas-api-revoke-sub-aa9aa932-ztxp2\" (UID: \"36233f6d-5ba6-4f5d-b148-da1ff598286b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa9aa932-ztxp2" Nov 02 07:01:41.333283 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:41.333204 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83ec9f9d-b0b5-4eaa-aa0d-69c43c283c2d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5f4e008a-8n8kr\" (UID: \"83ec9f9d-b0b5-4eaa-aa0d-69c43c283c2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f4e008a-8n8kr" Nov 02 07:01:41.333426 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:41.333346 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83ec9f9d-b0b5-4eaa-aa0d-69c43c283c2d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5f4e008a-8n8kr\" (UID: \"83ec9f9d-b0b5-4eaa-aa0d-69c43c283c2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f4e008a-8n8kr" Nov 02 07:01:41.333426 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:41.333390 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmfnh\" (UniqueName: \"kubernetes.io/projected/83ec9f9d-b0b5-4eaa-aa0d-69c43c283c2d-kube-api-access-mmfnh\") pod \"maas-api-revoke-sub-5f4e008a-8n8kr\" (UID: \"83ec9f9d-b0b5-4eaa-aa0d-69c43c283c2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f4e008a-8n8kr" Nov 02 07:01:41.334139 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:41.334116 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83ec9f9d-b0b5-4eaa-aa0d-69c43c283c2d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5f4e008a-8n8kr\" (UID: \"83ec9f9d-b0b5-4eaa-aa0d-69c43c283c2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f4e008a-8n8kr" Nov 02 07:01:41.336825 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:41.336800 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83ec9f9d-b0b5-4eaa-aa0d-69c43c283c2d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5f4e008a-8n8kr\" (UID: \"83ec9f9d-b0b5-4eaa-aa0d-69c43c283c2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f4e008a-8n8kr" Nov 02 07:01:41.388625 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:41.388597 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa9aa932-ztxp2" Nov 02 07:01:41.391119 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:41.391094 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmfnh\" (UniqueName: \"kubernetes.io/projected/83ec9f9d-b0b5-4eaa-aa0d-69c43c283c2d-kube-api-access-mmfnh\") pod \"maas-api-revoke-sub-5f4e008a-8n8kr\" (UID: \"83ec9f9d-b0b5-4eaa-aa0d-69c43c283c2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f4e008a-8n8kr" Nov 02 07:01:41.442342 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:41.442173 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f4e008a-8n8kr" Nov 02 07:01:41.769547 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:41.769518 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa9aa932-ztxp2"] Nov 02 07:01:41.770686 ip-10-0-130-93 kubenswrapper[5916]: W1102 07:01:41.770662 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36233f6d_5ba6_4f5d_b148_da1ff598286b.slice/crio-3f72d42102e86a1d2f5e64ee4685584dd71301164eebc7bd37641ade704cd082 WatchSource:0}: Error finding container 3f72d42102e86a1d2f5e64ee4685584dd71301164eebc7bd37641ade704cd082: Status 404 returned error can't find the container with id 3f72d42102e86a1d2f5e64ee4685584dd71301164eebc7bd37641ade704cd082 Nov 02 07:01:41.825535 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:41.825506 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f4e008a-8n8kr"] Nov 02 07:01:41.826297 ip-10-0-130-93 kubenswrapper[5916]: W1102 07:01:41.826270 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83ec9f9d_b0b5_4eaa_aa0d_69c43c283c2d.slice/crio-477b4f68bd51e55a8cda9cb1a53ba1ef2f91b42f7cf3a9154d77aa21f869960e WatchSource:0}: Error finding container 477b4f68bd51e55a8cda9cb1a53ba1ef2f91b42f7cf3a9154d77aa21f869960e: Status 404 returned error can't find the container with id 477b4f68bd51e55a8cda9cb1a53ba1ef2f91b42f7cf3a9154d77aa21f869960e Nov 02 07:01:42.024386 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:42.024305 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa9aa932-ztxp2" event={"ID":"36233f6d-5ba6-4f5d-b148-da1ff598286b","Type":"ContainerStarted","Data":"3f72d42102e86a1d2f5e64ee4685584dd71301164eebc7bd37641ade704cd082"} Nov 02 07:01:42.025693 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:42.025672 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f4e008a-8n8kr" event={"ID":"83ec9f9d-b0b5-4eaa-aa0d-69c43c283c2d","Type":"ContainerStarted","Data":"477b4f68bd51e55a8cda9cb1a53ba1ef2f91b42f7cf3a9154d77aa21f869960e"} Nov 02 07:01:43.033992 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:43.033960 5916 generic.go:356] "Generic (PLEG): container finished" podID="83ec9f9d-b0b5-4eaa-aa0d-69c43c283c2d" containerID="7d72d4477c27747c2c0461881a22f853e9908752a11ed0bdeedd357063e3f180" exitCode=0 Nov 02 07:01:43.034453 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:43.034045 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f4e008a-8n8kr" event={"ID":"83ec9f9d-b0b5-4eaa-aa0d-69c43c283c2d","Type":"ContainerDied","Data":"7d72d4477c27747c2c0461881a22f853e9908752a11ed0bdeedd357063e3f180"} Nov 02 07:01:43.035809 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:43.035784 5916 generic.go:356] "Generic (PLEG): container finished" podID="36233f6d-5ba6-4f5d-b148-da1ff598286b" containerID="5ec8838f066f6852066dfe62f7003c995a5a612f4e98f3f65495f5c4ac783c90" exitCode=0 Nov 02 07:01:43.035940 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:43.035822 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa9aa932-ztxp2" event={"ID":"36233f6d-5ba6-4f5d-b148-da1ff598286b","Type":"ContainerDied","Data":"5ec8838f066f6852066dfe62f7003c995a5a612f4e98f3f65495f5c4ac783c90"} Nov 02 07:01:44.210729 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:44.210706 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa9aa932-ztxp2" Nov 02 07:01:44.213479 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:44.213459 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f4e008a-8n8kr" Nov 02 07:01:44.263167 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:44.263136 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83ec9f9d-b0b5-4eaa-aa0d-69c43c283c2d-kube-api-access-mmfnh\" (UniqueName: \"kubernetes.io/projected/83ec9f9d-b0b5-4eaa-aa0d-69c43c283c2d-kube-api-access-mmfnh\") pod \"83ec9f9d-b0b5-4eaa-aa0d-69c43c283c2d\" (UID: \"83ec9f9d-b0b5-4eaa-aa0d-69c43c283c2d\") " Nov 02 07:01:44.263342 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:44.263241 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36233f6d-5ba6-4f5d-b148-da1ff598286b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36233f6d-5ba6-4f5d-b148-da1ff598286b-maas-api-openshift-ca\") pod \"36233f6d-5ba6-4f5d-b148-da1ff598286b\" (UID: \"36233f6d-5ba6-4f5d-b148-da1ff598286b\") " Nov 02 07:01:44.263342 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:44.263300 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36233f6d-5ba6-4f5d-b148-da1ff598286b-kube-api-access-gmctq\" (UniqueName: \"kubernetes.io/projected/36233f6d-5ba6-4f5d-b148-da1ff598286b-kube-api-access-gmctq\") pod \"36233f6d-5ba6-4f5d-b148-da1ff598286b\" (UID: \"36233f6d-5ba6-4f5d-b148-da1ff598286b\") " Nov 02 07:01:44.263454 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:44.263358 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/83ec9f9d-b0b5-4eaa-aa0d-69c43c283c2d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83ec9f9d-b0b5-4eaa-aa0d-69c43c283c2d-maas-api-xks-ca\") pod \"83ec9f9d-b0b5-4eaa-aa0d-69c43c283c2d\" (UID: \"83ec9f9d-b0b5-4eaa-aa0d-69c43c283c2d\") " Nov 02 07:01:44.263975 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:44.263827 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/36233f6d-5ba6-4f5d-b148-da1ff598286b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36233f6d-5ba6-4f5d-b148-da1ff598286b-maas-api-xks-ca\") pod \"36233f6d-5ba6-4f5d-b148-da1ff598286b\" (UID: \"36233f6d-5ba6-4f5d-b148-da1ff598286b\") " Nov 02 07:01:44.263975 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:44.263870 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83ec9f9d-b0b5-4eaa-aa0d-69c43c283c2d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83ec9f9d-b0b5-4eaa-aa0d-69c43c283c2d-maas-api-openshift-ca\") pod \"83ec9f9d-b0b5-4eaa-aa0d-69c43c283c2d\" (UID: \"83ec9f9d-b0b5-4eaa-aa0d-69c43c283c2d\") " Nov 02 07:01:44.264323 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:44.264294 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83ec9f9d-b0b5-4eaa-aa0d-69c43c283c2d-maas-api-openshift-ca" pod "83ec9f9d-b0b5-4eaa-aa0d-69c43c283c2d" (UID: "83ec9f9d-b0b5-4eaa-aa0d-69c43c283c2d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:01:44.264614 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:44.264586 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36233f6d-5ba6-4f5d-b148-da1ff598286b-maas-api-openshift-ca" pod "36233f6d-5ba6-4f5d-b148-da1ff598286b" (UID: "36233f6d-5ba6-4f5d-b148-da1ff598286b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:01:44.265882 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:44.265856 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83ec9f9d-b0b5-4eaa-aa0d-69c43c283c2d-kube-api-access-mmfnh" pod "83ec9f9d-b0b5-4eaa-aa0d-69c43c283c2d" (UID: "83ec9f9d-b0b5-4eaa-aa0d-69c43c283c2d"). InnerVolumeSpecName "kube-api-access-mmfnh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:01:44.266006 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:44.265972 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83ec9f9d-b0b5-4eaa-aa0d-69c43c283c2d-maas-api-xks-ca" pod "83ec9f9d-b0b5-4eaa-aa0d-69c43c283c2d" (UID: "83ec9f9d-b0b5-4eaa-aa0d-69c43c283c2d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 07:01:44.266119 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:44.266100 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36233f6d-5ba6-4f5d-b148-da1ff598286b-maas-api-xks-ca" pod "36233f6d-5ba6-4f5d-b148-da1ff598286b" (UID: "36233f6d-5ba6-4f5d-b148-da1ff598286b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 07:01:44.266182 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:44.266136 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36233f6d-5ba6-4f5d-b148-da1ff598286b-kube-api-access-gmctq" pod "36233f6d-5ba6-4f5d-b148-da1ff598286b" (UID: "36233f6d-5ba6-4f5d-b148-da1ff598286b"). InnerVolumeSpecName "kube-api-access-gmctq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:01:44.365843 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:44.365762 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmfnh\" (UniqueName: \"kubernetes.io/projected/83ec9f9d-b0b5-4eaa-aa0d-69c43c283c2d-kube-api-access-mmfnh\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:01:44.365843 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:44.365799 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36233f6d-5ba6-4f5d-b148-da1ff598286b-maas-api-openshift-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:01:44.365843 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:44.365813 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmctq\" (UniqueName: \"kubernetes.io/projected/36233f6d-5ba6-4f5d-b148-da1ff598286b-kube-api-access-gmctq\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:01:44.365843 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:44.365827 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83ec9f9d-b0b5-4eaa-aa0d-69c43c283c2d-maas-api-xks-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:01:44.365843 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:44.365841 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36233f6d-5ba6-4f5d-b148-da1ff598286b-maas-api-xks-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:01:44.366231 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:44.365853 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83ec9f9d-b0b5-4eaa-aa0d-69c43c283c2d-maas-api-openshift-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:01:45.050643 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:45.050602 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa9aa932-ztxp2" Nov 02 07:01:45.050643 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:45.050629 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa9aa932-ztxp2" event={"ID":"36233f6d-5ba6-4f5d-b148-da1ff598286b","Type":"ContainerDied","Data":"3f72d42102e86a1d2f5e64ee4685584dd71301164eebc7bd37641ade704cd082"} Nov 02 07:01:45.050887 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:45.050679 5916 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3f72d42102e86a1d2f5e64ee4685584dd71301164eebc7bd37641ade704cd082" Nov 02 07:01:45.052251 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:45.052227 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f4e008a-8n8kr" Nov 02 07:01:45.052251 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:45.052231 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f4e008a-8n8kr" event={"ID":"83ec9f9d-b0b5-4eaa-aa0d-69c43c283c2d","Type":"ContainerDied","Data":"477b4f68bd51e55a8cda9cb1a53ba1ef2f91b42f7cf3a9154d77aa21f869960e"} Nov 02 07:01:45.052466 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:45.052276 5916 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="477b4f68bd51e55a8cda9cb1a53ba1ef2f91b42f7cf3a9154d77aa21f869960e" Nov 02 07:01:49.013899 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:49.013847 5916 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-cdfce9/premium-w1-cdfce9-kserve-598cb9d57f-mrzrz" podUID="f9c5c1f7-cdca-46d6-8573-c85af1cd1946" containerName="main" containerID="cri-o://5796d3c345467f547f1950bdca52fa5d1c016b58d45968d3677d97eb7ecfc443" gracePeriod=45 Nov 02 07:01:49.029226 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:49.029167 5916 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-cdfce9/simulator-w1-cdfce9-kserve-7ffd68f78-5vwl2" podUID="b8b219aa-6a59-4d31-96f6-8d50ab910aca" containerName="main" containerID="cri-o://58d6ff44bea63b532b79c7d9be036bc26050edc4e322bb88a11615a417f77dd4" gracePeriod=45 Nov 02 07:01:49.284963 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:49.284939 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-cdfce9/premium-w1-cdfce9-kserve-598cb9d57f-mrzrz" Nov 02 07:01:49.326375 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:49.326351 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-cdfce9/simulator-w1-cdfce9-kserve-7ffd68f78-5vwl2" Nov 02 07:01:49.413158 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:49.413124 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8b219aa-6a59-4d31-96f6-8d50ab910aca-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8b219aa-6a59-4d31-96f6-8d50ab910aca-model-cache\") pod \"b8b219aa-6a59-4d31-96f6-8d50ab910aca\" (UID: \"b8b219aa-6a59-4d31-96f6-8d50ab910aca\") " Nov 02 07:01:49.413343 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:49.413190 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9c5c1f7-cdca-46d6-8573-c85af1cd1946-kube-api-access-wmlm2\" (UniqueName: \"kubernetes.io/projected/f9c5c1f7-cdca-46d6-8573-c85af1cd1946-kube-api-access-wmlm2\") pod \"f9c5c1f7-cdca-46d6-8573-c85af1cd1946\" (UID: \"f9c5c1f7-cdca-46d6-8573-c85af1cd1946\") " Nov 02 07:01:49.413343 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:49.413237 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9c5c1f7-cdca-46d6-8573-c85af1cd1946-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9c5c1f7-cdca-46d6-8573-c85af1cd1946-model-cache\") pod \"f9c5c1f7-cdca-46d6-8573-c85af1cd1946\" (UID: \"f9c5c1f7-cdca-46d6-8573-c85af1cd1946\") " Nov 02 07:01:49.413343 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:49.413301 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9c5c1f7-cdca-46d6-8573-c85af1cd1946-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9c5c1f7-cdca-46d6-8573-c85af1cd1946-dshm\") pod \"f9c5c1f7-cdca-46d6-8573-c85af1cd1946\" (UID: \"f9c5c1f7-cdca-46d6-8573-c85af1cd1946\") " Nov 02 07:01:49.413343 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:49.413332 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8b219aa-6a59-4d31-96f6-8d50ab910aca-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8b219aa-6a59-4d31-96f6-8d50ab910aca-tmp-dir\") pod \"b8b219aa-6a59-4d31-96f6-8d50ab910aca\" (UID: \"b8b219aa-6a59-4d31-96f6-8d50ab910aca\") " Nov 02 07:01:49.413589 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:49.413354 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b8b219aa-6a59-4d31-96f6-8d50ab910aca-tls-certs\" (UniqueName: \"kubernetes.io/secret/b8b219aa-6a59-4d31-96f6-8d50ab910aca-tls-certs\") pod \"b8b219aa-6a59-4d31-96f6-8d50ab910aca\" (UID: \"b8b219aa-6a59-4d31-96f6-8d50ab910aca\") " Nov 02 07:01:49.413589 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:49.413391 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9c5c1f7-cdca-46d6-8573-c85af1cd1946-home\" (UniqueName: \"kubernetes.io/empty-dir/f9c5c1f7-cdca-46d6-8573-c85af1cd1946-home\") pod \"f9c5c1f7-cdca-46d6-8573-c85af1cd1946\" (UID: \"f9c5c1f7-cdca-46d6-8573-c85af1cd1946\") " Nov 02 07:01:49.413589 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:49.413412 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f9c5c1f7-cdca-46d6-8573-c85af1cd1946-tls-certs\" (UniqueName: \"kubernetes.io/secret/f9c5c1f7-cdca-46d6-8573-c85af1cd1946-tls-certs\") pod \"f9c5c1f7-cdca-46d6-8573-c85af1cd1946\" (UID: \"f9c5c1f7-cdca-46d6-8573-c85af1cd1946\") " Nov 02 07:01:49.413589 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:49.413479 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8b219aa-6a59-4d31-96f6-8d50ab910aca-kube-api-access-zvwlv\" (UniqueName: \"kubernetes.io/projected/b8b219aa-6a59-4d31-96f6-8d50ab910aca-kube-api-access-zvwlv\") pod \"b8b219aa-6a59-4d31-96f6-8d50ab910aca\" (UID: \"b8b219aa-6a59-4d31-96f6-8d50ab910aca\") " Nov 02 07:01:49.413589 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:49.413505 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9c5c1f7-cdca-46d6-8573-c85af1cd1946-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9c5c1f7-cdca-46d6-8573-c85af1cd1946-tmp-dir\") pod \"f9c5c1f7-cdca-46d6-8573-c85af1cd1946\" (UID: \"f9c5c1f7-cdca-46d6-8573-c85af1cd1946\") " Nov 02 07:01:49.413589 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:49.413531 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8b219aa-6a59-4d31-96f6-8d50ab910aca-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8b219aa-6a59-4d31-96f6-8d50ab910aca-dshm\") pod \"b8b219aa-6a59-4d31-96f6-8d50ab910aca\" (UID: \"b8b219aa-6a59-4d31-96f6-8d50ab910aca\") " Nov 02 07:01:49.413589 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:49.413553 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8b219aa-6a59-4d31-96f6-8d50ab910aca-home\" (UniqueName: \"kubernetes.io/empty-dir/b8b219aa-6a59-4d31-96f6-8d50ab910aca-home\") pod \"b8b219aa-6a59-4d31-96f6-8d50ab910aca\" (UID: \"b8b219aa-6a59-4d31-96f6-8d50ab910aca\") " Nov 02 07:01:49.414000 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:49.413966 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8b219aa-6a59-4d31-96f6-8d50ab910aca-home" pod "b8b219aa-6a59-4d31-96f6-8d50ab910aca" (UID: "b8b219aa-6a59-4d31-96f6-8d50ab910aca"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:01:49.414129 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:49.414112 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8b219aa-6a59-4d31-96f6-8d50ab910aca-model-cache" pod "b8b219aa-6a59-4d31-96f6-8d50ab910aca" (UID: "b8b219aa-6a59-4d31-96f6-8d50ab910aca"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:01:49.414723 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:49.414606 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9c5c1f7-cdca-46d6-8573-c85af1cd1946-home" pod "f9c5c1f7-cdca-46d6-8573-c85af1cd1946" (UID: "f9c5c1f7-cdca-46d6-8573-c85af1cd1946"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:01:49.414835 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:49.414806 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8b219aa-6a59-4d31-96f6-8d50ab910aca-tmp-dir" pod "b8b219aa-6a59-4d31-96f6-8d50ab910aca" (UID: "b8b219aa-6a59-4d31-96f6-8d50ab910aca"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:01:49.415501 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:49.415432 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9c5c1f7-cdca-46d6-8573-c85af1cd1946-tmp-dir" pod "f9c5c1f7-cdca-46d6-8573-c85af1cd1946" (UID: "f9c5c1f7-cdca-46d6-8573-c85af1cd1946"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:01:49.415687 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:49.415657 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9c5c1f7-cdca-46d6-8573-c85af1cd1946-model-cache" pod "f9c5c1f7-cdca-46d6-8573-c85af1cd1946" (UID: "f9c5c1f7-cdca-46d6-8573-c85af1cd1946"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:01:49.417267 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:49.417242 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9c5c1f7-cdca-46d6-8573-c85af1cd1946-kube-api-access-wmlm2" pod "f9c5c1f7-cdca-46d6-8573-c85af1cd1946" (UID: "f9c5c1f7-cdca-46d6-8573-c85af1cd1946"). InnerVolumeSpecName "kube-api-access-wmlm2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:01:49.417575 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:49.417319 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8b219aa-6a59-4d31-96f6-8d50ab910aca-kube-api-access-zvwlv" pod "b8b219aa-6a59-4d31-96f6-8d50ab910aca" (UID: "b8b219aa-6a59-4d31-96f6-8d50ab910aca"). InnerVolumeSpecName "kube-api-access-zvwlv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:01:49.417575 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:49.417366 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9c5c1f7-cdca-46d6-8573-c85af1cd1946-tls-certs" pod "f9c5c1f7-cdca-46d6-8573-c85af1cd1946" (UID: "f9c5c1f7-cdca-46d6-8573-c85af1cd1946"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 07:01:49.417575 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:49.417381 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9c5c1f7-cdca-46d6-8573-c85af1cd1946-dshm" pod "f9c5c1f7-cdca-46d6-8573-c85af1cd1946" (UID: "f9c5c1f7-cdca-46d6-8573-c85af1cd1946"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:01:49.417781 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:49.417752 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8b219aa-6a59-4d31-96f6-8d50ab910aca-dshm" pod "b8b219aa-6a59-4d31-96f6-8d50ab910aca" (UID: "b8b219aa-6a59-4d31-96f6-8d50ab910aca"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:01:49.419026 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:49.418984 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8b219aa-6a59-4d31-96f6-8d50ab910aca-tls-certs" pod "b8b219aa-6a59-4d31-96f6-8d50ab910aca" (UID: "b8b219aa-6a59-4d31-96f6-8d50ab910aca"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 07:01:49.514697 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:49.514663 5916 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9c5c1f7-cdca-46d6-8573-c85af1cd1946-home\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:01:49.514697 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:49.514695 5916 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9c5c1f7-cdca-46d6-8573-c85af1cd1946-tls-certs\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:01:49.514978 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:49.514707 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvwlv\" (UniqueName: \"kubernetes.io/projected/b8b219aa-6a59-4d31-96f6-8d50ab910aca-kube-api-access-zvwlv\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:01:49.514978 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:49.514716 5916 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9c5c1f7-cdca-46d6-8573-c85af1cd1946-tmp-dir\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:01:49.514978 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:49.514725 5916 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8b219aa-6a59-4d31-96f6-8d50ab910aca-dshm\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:01:49.514978 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:49.514733 5916 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8b219aa-6a59-4d31-96f6-8d50ab910aca-home\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:01:49.514978 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:49.514746 5916 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8b219aa-6a59-4d31-96f6-8d50ab910aca-model-cache\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:01:49.514978 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:49.514758 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmlm2\" (UniqueName: \"kubernetes.io/projected/f9c5c1f7-cdca-46d6-8573-c85af1cd1946-kube-api-access-wmlm2\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:01:49.514978 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:49.514767 5916 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9c5c1f7-cdca-46d6-8573-c85af1cd1946-model-cache\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:01:49.514978 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:49.514776 5916 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9c5c1f7-cdca-46d6-8573-c85af1cd1946-dshm\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:01:49.514978 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:49.514788 5916 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8b219aa-6a59-4d31-96f6-8d50ab910aca-tmp-dir\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:01:49.514978 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:49.514798 5916 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8b219aa-6a59-4d31-96f6-8d50ab910aca-tls-certs\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:01:49.892486 ip-10-0-130-93 kubenswrapper[5916]: E1102 07:01:49.892442 5916 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p98cq" containerName="extractor" Nov 02 07:01:50.087695 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:50.087662 5916 generic.go:356] "Generic (PLEG): container finished" podID="b8b219aa-6a59-4d31-96f6-8d50ab910aca" containerID="58d6ff44bea63b532b79c7d9be036bc26050edc4e322bb88a11615a417f77dd4" exitCode=0 Nov 02 07:01:50.088192 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:50.087703 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-cdfce9/simulator-w1-cdfce9-kserve-7ffd68f78-5vwl2" Nov 02 07:01:50.088192 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:50.087742 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-cdfce9/simulator-w1-cdfce9-kserve-7ffd68f78-5vwl2" event={"ID":"b8b219aa-6a59-4d31-96f6-8d50ab910aca","Type":"ContainerDied","Data":"58d6ff44bea63b532b79c7d9be036bc26050edc4e322bb88a11615a417f77dd4"} Nov 02 07:01:50.088192 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:50.087786 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-cdfce9/simulator-w1-cdfce9-kserve-7ffd68f78-5vwl2" event={"ID":"b8b219aa-6a59-4d31-96f6-8d50ab910aca","Type":"ContainerDied","Data":"3aa4436b28ac1b05648bc671ff17e9e22d93972aa56abf7c7ba4065bcd91dd8a"} Nov 02 07:01:50.088192 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:50.087816 5916 scope.go:122] "RemoveContainer" containerID="58d6ff44bea63b532b79c7d9be036bc26050edc4e322bb88a11615a417f77dd4" Nov 02 07:01:50.089357 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:50.089333 5916 generic.go:356] "Generic (PLEG): container finished" podID="f9c5c1f7-cdca-46d6-8573-c85af1cd1946" containerID="5796d3c345467f547f1950bdca52fa5d1c016b58d45968d3677d97eb7ecfc443" exitCode=0 Nov 02 07:01:50.089464 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:50.089367 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-cdfce9/premium-w1-cdfce9-kserve-598cb9d57f-mrzrz" event={"ID":"f9c5c1f7-cdca-46d6-8573-c85af1cd1946","Type":"ContainerDied","Data":"5796d3c345467f547f1950bdca52fa5d1c016b58d45968d3677d97eb7ecfc443"} Nov 02 07:01:50.089464 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:50.089396 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-cdfce9/premium-w1-cdfce9-kserve-598cb9d57f-mrzrz" Nov 02 07:01:50.089464 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:50.089414 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-cdfce9/premium-w1-cdfce9-kserve-598cb9d57f-mrzrz" event={"ID":"f9c5c1f7-cdca-46d6-8573-c85af1cd1946","Type":"ContainerDied","Data":"407fada787d0e6ed28a7f0688473ad4b7f49737ff236bbb640eab23936fee5ee"} Nov 02 07:01:50.097549 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:50.097527 5916 scope.go:122] "RemoveContainer" containerID="58d6ff44bea63b532b79c7d9be036bc26050edc4e322bb88a11615a417f77dd4" Nov 02 07:01:50.097834 ip-10-0-130-93 kubenswrapper[5916]: E1102 07:01:50.097816 5916 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58d6ff44bea63b532b79c7d9be036bc26050edc4e322bb88a11615a417f77dd4\": container with ID starting with 58d6ff44bea63b532b79c7d9be036bc26050edc4e322bb88a11615a417f77dd4 not found: ID does not exist" containerID="58d6ff44bea63b532b79c7d9be036bc26050edc4e322bb88a11615a417f77dd4" Nov 02 07:01:50.097882 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:50.097844 5916 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58d6ff44bea63b532b79c7d9be036bc26050edc4e322bb88a11615a417f77dd4"} err="failed to get container status \"58d6ff44bea63b532b79c7d9be036bc26050edc4e322bb88a11615a417f77dd4\": rpc error: code = NotFound desc = could not find container \"58d6ff44bea63b532b79c7d9be036bc26050edc4e322bb88a11615a417f77dd4\": container with ID starting with 58d6ff44bea63b532b79c7d9be036bc26050edc4e322bb88a11615a417f77dd4 not found: ID does not exist" Nov 02 07:01:50.097882 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:50.097861 5916 scope.go:122] "RemoveContainer" containerID="5796d3c345467f547f1950bdca52fa5d1c016b58d45968d3677d97eb7ecfc443" Nov 02 07:01:50.106870 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:50.106849 5916 scope.go:122] "RemoveContainer" containerID="5796d3c345467f547f1950bdca52fa5d1c016b58d45968d3677d97eb7ecfc443" Nov 02 07:01:50.107155 ip-10-0-130-93 kubenswrapper[5916]: E1102 07:01:50.107133 5916 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5796d3c345467f547f1950bdca52fa5d1c016b58d45968d3677d97eb7ecfc443\": container with ID starting with 5796d3c345467f547f1950bdca52fa5d1c016b58d45968d3677d97eb7ecfc443 not found: ID does not exist" containerID="5796d3c345467f547f1950bdca52fa5d1c016b58d45968d3677d97eb7ecfc443" Nov 02 07:01:50.107226 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:50.107163 5916 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5796d3c345467f547f1950bdca52fa5d1c016b58d45968d3677d97eb7ecfc443"} err="failed to get container status \"5796d3c345467f547f1950bdca52fa5d1c016b58d45968d3677d97eb7ecfc443\": rpc error: code = NotFound desc = could not find container \"5796d3c345467f547f1950bdca52fa5d1c016b58d45968d3677d97eb7ecfc443\": container with ID starting with 5796d3c345467f547f1950bdca52fa5d1c016b58d45968d3677d97eb7ecfc443 not found: ID does not exist" Nov 02 07:01:50.254629 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:50.254586 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-cdfce9/premium-w1-cdfce9-kserve-598cb9d57f-mrzrz"] Nov 02 07:01:50.257989 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:50.257965 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-cdfce9/premium-w1-cdfce9-kserve-598cb9d57f-mrzrz"] Nov 02 07:01:50.368012 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:50.367896 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-cdfce9/simulator-w1-cdfce9-kserve-7ffd68f78-5vwl2"] Nov 02 07:01:50.371264 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:50.371235 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-cdfce9/simulator-w1-cdfce9-kserve-7ffd68f78-5vwl2"] Nov 02 07:01:51.842839 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:51.842807 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8b219aa-6a59-4d31-96f6-8d50ab910aca" path="/var/lib/kubelet/pods/b8b219aa-6a59-4d31-96f6-8d50ab910aca/volumes" Nov 02 07:01:51.843489 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:51.843243 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9c5c1f7-cdca-46d6-8573-c85af1cd1946" path="/var/lib/kubelet/pods/f9c5c1f7-cdca-46d6-8573-c85af1cd1946/volumes" Nov 02 07:01:53.270876 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:53.270850 5916 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-csfhg_17dee72b-2f47-4016-8ffe-3a89934e66ee/console-operator/2.log" Nov 02 07:01:53.281962 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:53.281941 5916 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-csfhg_17dee72b-2f47-4016-8ffe-3a89934e66ee/console-operator/2.log" Nov 02 07:01:55.651336 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:55.651299 5916 scope.go:122] "RemoveContainer" containerID="9f736606ebb823adf23b5f5075bc38b3de3a188e6592a490290732db3f028e00" Nov 02 07:01:55.665096 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:01:55.665065 5916 scope.go:122] "RemoveContainer" containerID="9db1cab1f81d6480af2d5503bce80d922517edb60cf9e1ed18ae0109fad92035" Nov 02 07:02:01.536122 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:01.536043 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-3e96be-skgc9"] Nov 02 07:02:01.585655 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:01.585622 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-3e96be-skgc9"] Nov 02 07:02:01.841820 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:01.841741 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74e1873c-18db-4d6e-ad41-44ca49a732bd" path="/var/lib/kubelet/pods/74e1873c-18db-4d6e-ad41-44ca49a732bd/volumes" Nov 02 07:02:02.764491 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:02.764460 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-cdfce9-5fwjj"] Nov 02 07:02:02.814001 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:02.813973 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-cdfce9-5fwjj"] Nov 02 07:02:03.842440 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:03.842407 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e2fdb61-3a2c-46df-92ff-27547f130b0f" path="/var/lib/kubelet/pods/2e2fdb61-3a2c-46df-92ff-27547f130b0f/volumes" Nov 02 07:02:09.095175 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:09.095138 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-6e2657-fg5fq"] Nov 02 07:02:09.095612 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:09.095603 5916 cpu_manager.go:414] "containerMap: removing container" podUID="f9c5c1f7-cdca-46d6-8573-c85af1cd1946" containerName="main" Nov 02 07:02:09.095655 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:09.095616 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9c5c1f7-cdca-46d6-8573-c85af1cd1946" containerName="main" Nov 02 07:02:09.095655 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:09.095629 5916 cpu_manager.go:414] "containerMap: removing container" podUID="83ec9f9d-b0b5-4eaa-aa0d-69c43c283c2d" containerName="revoke-keys" Nov 02 07:02:09.095655 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:09.095633 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83ec9f9d-b0b5-4eaa-aa0d-69c43c283c2d" containerName="revoke-keys" Nov 02 07:02:09.095655 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:09.095641 5916 cpu_manager.go:414] "containerMap: removing container" podUID="b8b219aa-6a59-4d31-96f6-8d50ab910aca" containerName="main" Nov 02 07:02:09.095655 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:09.095646 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8b219aa-6a59-4d31-96f6-8d50ab910aca" containerName="main" Nov 02 07:02:09.095820 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:09.095657 5916 cpu_manager.go:414] "containerMap: removing container" podUID="36233f6d-5ba6-4f5d-b148-da1ff598286b" containerName="revoke-keys" Nov 02 07:02:09.095820 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:09.095665 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36233f6d-5ba6-4f5d-b148-da1ff598286b" containerName="revoke-keys" Nov 02 07:02:09.095820 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:09.095733 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9c5c1f7-cdca-46d6-8573-c85af1cd1946" containerName="main" Nov 02 07:02:09.095820 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:09.095745 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="83ec9f9d-b0b5-4eaa-aa0d-69c43c283c2d" containerName="revoke-keys" Nov 02 07:02:09.095820 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:09.095757 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8b219aa-6a59-4d31-96f6-8d50ab910aca" containerName="main" Nov 02 07:02:09.095820 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:09.095763 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="36233f6d-5ba6-4f5d-b148-da1ff598286b" containerName="revoke-keys" Nov 02 07:02:09.100945 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:09.100921 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-6e2657-fg5fq" Nov 02 07:02:09.150853 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:09.150820 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-6e2657-fg5fq"] Nov 02 07:02:09.153576 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:09.153549 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 07:02:09.198071 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:09.198033 5916 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 07:02:09.198071 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:09.198033 5916 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 07:02:09.203261 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:09.203240 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-68l4r\"" Nov 02 07:02:09.214770 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:09.214741 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-978p7\" (UniqueName: \"kubernetes.io/projected/40f214a9-dcfa-4c23-9373-405b405c3d1f-kube-api-access-978p7\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-6e2657-fg5fq\" (UID: \"40f214a9-dcfa-4c23-9373-405b405c3d1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-6e2657-fg5fq" Nov 02 07:02:09.214972 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:09.214952 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40f214a9-dcfa-4c23-9373-405b405c3d1f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-6e2657-fg5fq\" (UID: \"40f214a9-dcfa-4c23-9373-405b405c3d1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-6e2657-fg5fq" Nov 02 07:02:09.215038 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:09.214996 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40f214a9-dcfa-4c23-9373-405b405c3d1f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-6e2657-fg5fq\" (UID: \"40f214a9-dcfa-4c23-9373-405b405c3d1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-6e2657-fg5fq" Nov 02 07:02:09.315500 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:09.315461 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40f214a9-dcfa-4c23-9373-405b405c3d1f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-6e2657-fg5fq\" (UID: \"40f214a9-dcfa-4c23-9373-405b405c3d1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-6e2657-fg5fq" Nov 02 07:02:09.315695 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:09.315506 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40f214a9-dcfa-4c23-9373-405b405c3d1f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-6e2657-fg5fq\" (UID: \"40f214a9-dcfa-4c23-9373-405b405c3d1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-6e2657-fg5fq" Nov 02 07:02:09.315695 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:09.315553 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-978p7\" (UniqueName: \"kubernetes.io/projected/40f214a9-dcfa-4c23-9373-405b405c3d1f-kube-api-access-978p7\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-6e2657-fg5fq\" (UID: \"40f214a9-dcfa-4c23-9373-405b405c3d1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-6e2657-fg5fq" Nov 02 07:02:09.316259 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:09.316238 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40f214a9-dcfa-4c23-9373-405b405c3d1f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-6e2657-fg5fq\" (UID: \"40f214a9-dcfa-4c23-9373-405b405c3d1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-6e2657-fg5fq" Nov 02 07:02:09.317960 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:09.317941 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40f214a9-dcfa-4c23-9373-405b405c3d1f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-6e2657-fg5fq\" (UID: \"40f214a9-dcfa-4c23-9373-405b405c3d1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-6e2657-fg5fq" Nov 02 07:02:09.373529 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:09.373454 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-978p7\" (UniqueName: \"kubernetes.io/projected/40f214a9-dcfa-4c23-9373-405b405c3d1f-kube-api-access-978p7\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-6e2657-fg5fq\" (UID: \"40f214a9-dcfa-4c23-9373-405b405c3d1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-6e2657-fg5fq" Nov 02 07:02:09.414242 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:09.414216 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-6e2657-fg5fq" Nov 02 07:02:09.787082 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:09.787053 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-6e2657-fg5fq"] Nov 02 07:02:09.788515 ip-10-0-130-93 kubenswrapper[5916]: W1102 07:02:09.788481 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40f214a9_dcfa_4c23_9373_405b405c3d1f.slice/crio-5329c47d61c9e1638b02c1bf5e8fcbdf9d841e42b6e0a7ecef89cdfd99aaa50e WatchSource:0}: Error finding container 5329c47d61c9e1638b02c1bf5e8fcbdf9d841e42b6e0a7ecef89cdfd99aaa50e: Status 404 returned error can't find the container with id 5329c47d61c9e1638b02c1bf5e8fcbdf9d841e42b6e0a7ecef89cdfd99aaa50e Nov 02 07:02:10.082993 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:10.082960 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-flk8r"] Nov 02 07:02:10.083187 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:10.083162 5916 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-flk8r" podUID="4c10335d-e121-4ccd-949d-964cf21650ca" containerName="istio-proxy" containerID="cri-o://c11f67bbf7a865f85db82cdeae7847c747172a05959bbe9cd68e14ba886a53b3" gracePeriod=30 Nov 02 07:02:10.229437 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:10.229404 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-6e2657-fg5fq" event={"ID":"40f214a9-dcfa-4c23-9373-405b405c3d1f","Type":"ContainerStarted","Data":"5329c47d61c9e1638b02c1bf5e8fcbdf9d841e42b6e0a7ecef89cdfd99aaa50e"} Nov 02 07:02:11.237515 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:11.237484 5916 generic.go:356] "Generic (PLEG): container finished" podID="40f214a9-dcfa-4c23-9373-405b405c3d1f" containerID="b7a0a1239daa6b1ee3633d5ff24889c1b09671af56de03dc167087976cf1a4d9" exitCode=0 Nov 02 07:02:11.237958 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:11.237532 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-6e2657-fg5fq" event={"ID":"40f214a9-dcfa-4c23-9373-405b405c3d1f","Type":"ContainerDied","Data":"b7a0a1239daa6b1ee3633d5ff24889c1b09671af56de03dc167087976cf1a4d9"} Nov 02 07:02:12.378085 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:12.378061 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-6e2657-fg5fq" Nov 02 07:02:12.443674 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:12.443636 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/40f214a9-dcfa-4c23-9373-405b405c3d1f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40f214a9-dcfa-4c23-9373-405b405c3d1f-maas-api-xks-ca\") pod \"40f214a9-dcfa-4c23-9373-405b405c3d1f\" (UID: \"40f214a9-dcfa-4c23-9373-405b405c3d1f\") " Nov 02 07:02:12.443850 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:12.443699 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40f214a9-dcfa-4c23-9373-405b405c3d1f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40f214a9-dcfa-4c23-9373-405b405c3d1f-maas-api-openshift-ca\") pod \"40f214a9-dcfa-4c23-9373-405b405c3d1f\" (UID: \"40f214a9-dcfa-4c23-9373-405b405c3d1f\") " Nov 02 07:02:12.443850 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:12.443811 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40f214a9-dcfa-4c23-9373-405b405c3d1f-kube-api-access-978p7\" (UniqueName: \"kubernetes.io/projected/40f214a9-dcfa-4c23-9373-405b405c3d1f-kube-api-access-978p7\") pod \"40f214a9-dcfa-4c23-9373-405b405c3d1f\" (UID: \"40f214a9-dcfa-4c23-9373-405b405c3d1f\") " Nov 02 07:02:12.444145 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:12.444115 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40f214a9-dcfa-4c23-9373-405b405c3d1f-maas-api-openshift-ca" pod "40f214a9-dcfa-4c23-9373-405b405c3d1f" (UID: "40f214a9-dcfa-4c23-9373-405b405c3d1f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:02:12.445954 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:12.445917 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40f214a9-dcfa-4c23-9373-405b405c3d1f-maas-api-xks-ca" pod "40f214a9-dcfa-4c23-9373-405b405c3d1f" (UID: "40f214a9-dcfa-4c23-9373-405b405c3d1f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 07:02:12.446058 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:12.446038 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40f214a9-dcfa-4c23-9373-405b405c3d1f-kube-api-access-978p7" pod "40f214a9-dcfa-4c23-9373-405b405c3d1f" (UID: "40f214a9-dcfa-4c23-9373-405b405c3d1f"). InnerVolumeSpecName "kube-api-access-978p7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:02:12.544604 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:12.544533 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40f214a9-dcfa-4c23-9373-405b405c3d1f-maas-api-openshift-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:02:12.544604 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:12.544560 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-978p7\" (UniqueName: \"kubernetes.io/projected/40f214a9-dcfa-4c23-9373-405b405c3d1f-kube-api-access-978p7\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:02:12.544604 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:12.544571 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40f214a9-dcfa-4c23-9373-405b405c3d1f-maas-api-xks-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:02:13.253305 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:13.253270 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-6e2657-fg5fq" Nov 02 07:02:13.253305 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:13.253281 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-6e2657-fg5fq" event={"ID":"40f214a9-dcfa-4c23-9373-405b405c3d1f","Type":"ContainerDied","Data":"5329c47d61c9e1638b02c1bf5e8fcbdf9d841e42b6e0a7ecef89cdfd99aaa50e"} Nov 02 07:02:13.253575 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:13.253332 5916 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5329c47d61c9e1638b02c1bf5e8fcbdf9d841e42b6e0a7ecef89cdfd99aaa50e" Nov 02 07:02:13.481958 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:13.481917 5916 patch_prober.go:29] interesting pod/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-flk8r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 07:02:13.482351 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:13.481980 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-flk8r" podUID="4c10335d-e121-4ccd-949d-964cf21650ca" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 07:02:15.270266 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:15.270236 5916 generic.go:356] "Generic (PLEG): container finished" podID="4c10335d-e121-4ccd-949d-964cf21650ca" containerID="c11f67bbf7a865f85db82cdeae7847c747172a05959bbe9cd68e14ba886a53b3" exitCode=0 Nov 02 07:02:15.270622 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:15.270312 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-flk8r" event={"ID":"4c10335d-e121-4ccd-949d-964cf21650ca","Type":"ContainerDied","Data":"c11f67bbf7a865f85db82cdeae7847c747172a05959bbe9cd68e14ba886a53b3"} Nov 02 07:02:15.338782 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:15.338762 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-flk8r" Nov 02 07:02:15.364976 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:15.364922 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c10335d-e121-4ccd-949d-964cf21650ca-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c10335d-e121-4ccd-949d-964cf21650ca-workload-socket\") pod \"4c10335d-e121-4ccd-949d-964cf21650ca\" (UID: \"4c10335d-e121-4ccd-949d-964cf21650ca\") " Nov 02 07:02:15.364976 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:15.364959 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c10335d-e121-4ccd-949d-964cf21650ca-istio-token\" (UniqueName: \"kubernetes.io/projected/4c10335d-e121-4ccd-949d-964cf21650ca-istio-token\") pod \"4c10335d-e121-4ccd-949d-964cf21650ca\" (UID: \"4c10335d-e121-4ccd-949d-964cf21650ca\") " Nov 02 07:02:15.365133 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:15.364986 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c10335d-e121-4ccd-949d-964cf21650ca-kube-api-access-5wrdx\" (UniqueName: \"kubernetes.io/projected/4c10335d-e121-4ccd-949d-964cf21650ca-kube-api-access-5wrdx\") pod \"4c10335d-e121-4ccd-949d-964cf21650ca\" (UID: \"4c10335d-e121-4ccd-949d-964cf21650ca\") " Nov 02 07:02:15.365133 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:15.365024 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c10335d-e121-4ccd-949d-964cf21650ca-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c10335d-e121-4ccd-949d-964cf21650ca-workload-certs\") pod \"4c10335d-e121-4ccd-949d-964cf21650ca\" (UID: \"4c10335d-e121-4ccd-949d-964cf21650ca\") " Nov 02 07:02:15.365133 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:15.365060 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c10335d-e121-4ccd-949d-964cf21650ca-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c10335d-e121-4ccd-949d-964cf21650ca-credential-socket\") pod \"4c10335d-e121-4ccd-949d-964cf21650ca\" (UID: \"4c10335d-e121-4ccd-949d-964cf21650ca\") " Nov 02 07:02:15.365133 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:15.365093 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c10335d-e121-4ccd-949d-964cf21650ca-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c10335d-e121-4ccd-949d-964cf21650ca-istio-data\") pod \"4c10335d-e121-4ccd-949d-964cf21650ca\" (UID: \"4c10335d-e121-4ccd-949d-964cf21650ca\") " Nov 02 07:02:15.365133 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:15.365130 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4c10335d-e121-4ccd-949d-964cf21650ca-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c10335d-e121-4ccd-949d-964cf21650ca-istio-podinfo\") pod \"4c10335d-e121-4ccd-949d-964cf21650ca\" (UID: \"4c10335d-e121-4ccd-949d-964cf21650ca\") " Nov 02 07:02:15.365405 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:15.365206 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c10335d-e121-4ccd-949d-964cf21650ca-workload-socket" pod "4c10335d-e121-4ccd-949d-964cf21650ca" (UID: "4c10335d-e121-4ccd-949d-964cf21650ca"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:02:15.365405 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:15.365233 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c10335d-e121-4ccd-949d-964cf21650ca-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c10335d-e121-4ccd-949d-964cf21650ca-istiod-ca-cert\") pod \"4c10335d-e121-4ccd-949d-964cf21650ca\" (UID: \"4c10335d-e121-4ccd-949d-964cf21650ca\") " Nov 02 07:02:15.365405 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:15.365263 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c10335d-e121-4ccd-949d-964cf21650ca-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c10335d-e121-4ccd-949d-964cf21650ca-istio-envoy\") pod \"4c10335d-e121-4ccd-949d-964cf21650ca\" (UID: \"4c10335d-e121-4ccd-949d-964cf21650ca\") " Nov 02 07:02:15.365405 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:15.365315 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c10335d-e121-4ccd-949d-964cf21650ca-workload-certs" pod "4c10335d-e121-4ccd-949d-964cf21650ca" (UID: "4c10335d-e121-4ccd-949d-964cf21650ca"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:02:15.365405 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:15.365358 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c10335d-e121-4ccd-949d-964cf21650ca-credential-socket" pod "4c10335d-e121-4ccd-949d-964cf21650ca" (UID: "4c10335d-e121-4ccd-949d-964cf21650ca"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:02:15.365405 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:15.365401 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c10335d-e121-4ccd-949d-964cf21650ca-istio-data" pod "4c10335d-e121-4ccd-949d-964cf21650ca" (UID: "4c10335d-e121-4ccd-949d-964cf21650ca"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:02:15.365737 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:15.365568 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c10335d-e121-4ccd-949d-964cf21650ca-istiod-ca-cert" pod "4c10335d-e121-4ccd-949d-964cf21650ca" (UID: "4c10335d-e121-4ccd-949d-964cf21650ca"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:02:15.365737 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:15.365596 5916 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c10335d-e121-4ccd-949d-964cf21650ca-workload-socket\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:02:15.365737 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:15.365611 5916 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c10335d-e121-4ccd-949d-964cf21650ca-workload-certs\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:02:15.365737 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:15.365622 5916 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c10335d-e121-4ccd-949d-964cf21650ca-credential-socket\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:02:15.365737 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:15.365630 5916 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c10335d-e121-4ccd-949d-964cf21650ca-istio-data\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:02:15.367253 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:15.367231 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c10335d-e121-4ccd-949d-964cf21650ca-istio-envoy" pod "4c10335d-e121-4ccd-949d-964cf21650ca" (UID: "4c10335d-e121-4ccd-949d-964cf21650ca"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:02:15.367344 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:15.367321 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c10335d-e121-4ccd-949d-964cf21650ca-kube-api-access-5wrdx" pod "4c10335d-e121-4ccd-949d-964cf21650ca" (UID: "4c10335d-e121-4ccd-949d-964cf21650ca"). InnerVolumeSpecName "kube-api-access-5wrdx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:02:15.367508 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:15.367491 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c10335d-e121-4ccd-949d-964cf21650ca-istio-token" pod "4c10335d-e121-4ccd-949d-964cf21650ca" (UID: "4c10335d-e121-4ccd-949d-964cf21650ca"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:02:15.367577 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:15.367560 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4c10335d-e121-4ccd-949d-964cf21650ca-istio-podinfo" pod "4c10335d-e121-4ccd-949d-964cf21650ca" (UID: "4c10335d-e121-4ccd-949d-964cf21650ca"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 07:02:15.466650 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:15.466629 5916 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c10335d-e121-4ccd-949d-964cf21650ca-istio-token\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:02:15.466759 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:15.466651 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wrdx\" (UniqueName: \"kubernetes.io/projected/4c10335d-e121-4ccd-949d-964cf21650ca-kube-api-access-5wrdx\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:02:15.466759 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:15.466661 5916 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c10335d-e121-4ccd-949d-964cf21650ca-istio-podinfo\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:02:15.466759 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:15.466669 5916 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c10335d-e121-4ccd-949d-964cf21650ca-istiod-ca-cert\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:02:15.466759 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:15.466679 5916 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c10335d-e121-4ccd-949d-964cf21650ca-istio-envoy\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:02:16.278121 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:16.278091 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-flk8r" Nov 02 07:02:16.278551 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:16.278094 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-flk8r" event={"ID":"4c10335d-e121-4ccd-949d-964cf21650ca","Type":"ContainerDied","Data":"b69bc230833ed96783908f62494c00f475d26791f1ad6b0ba14836afb6935c61"} Nov 02 07:02:16.278551 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:16.278225 5916 scope.go:122] "RemoveContainer" containerID="c11f67bbf7a865f85db82cdeae7847c747172a05959bbe9cd68e14ba886a53b3" Nov 02 07:02:16.494522 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:16.494489 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-flk8r"] Nov 02 07:02:16.544122 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:16.544055 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-flk8r"] Nov 02 07:02:17.840723 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:17.840683 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c10335d-e121-4ccd-949d-964cf21650ca" path="/var/lib/kubelet/pods/4c10335d-e121-4ccd-949d-964cf21650ca/volumes" Nov 02 07:02:19.892831 ip-10-0-130-93 kubenswrapper[5916]: E1102 07:02:19.892799 5916 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p98cq" containerName="extractor" Nov 02 07:02:25.078451 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:25.078417 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-6e2657-fg5fq"] Nov 02 07:02:25.127946 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:25.127898 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-6e2657-fg5fq"] Nov 02 07:02:25.841917 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:25.841869 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40f214a9-dcfa-4c23-9373-405b405c3d1f" path="/var/lib/kubelet/pods/40f214a9-dcfa-4c23-9373-405b405c3d1f/volumes" Nov 02 07:02:32.737338 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:32.737303 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-6ff85e-zpz86"] Nov 02 07:02:32.737735 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:32.737704 5916 cpu_manager.go:414] "containerMap: removing container" podUID="4c10335d-e121-4ccd-949d-964cf21650ca" containerName="istio-proxy" Nov 02 07:02:32.737735 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:32.737717 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c10335d-e121-4ccd-949d-964cf21650ca" containerName="istio-proxy" Nov 02 07:02:32.737818 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:32.737750 5916 cpu_manager.go:414] "containerMap: removing container" podUID="40f214a9-dcfa-4c23-9373-405b405c3d1f" containerName="revoke-keys" Nov 02 07:02:32.737818 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:32.737755 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40f214a9-dcfa-4c23-9373-405b405c3d1f" containerName="revoke-keys" Nov 02 07:02:32.737885 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:32.737826 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="40f214a9-dcfa-4c23-9373-405b405c3d1f" containerName="revoke-keys" Nov 02 07:02:32.737885 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:32.737838 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c10335d-e121-4ccd-949d-964cf21650ca" containerName="istio-proxy" Nov 02 07:02:32.741369 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:32.741350 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-6ff85e-zpz86" Nov 02 07:02:32.793074 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:32.793046 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-6ff85e-zpz86"] Nov 02 07:02:32.794189 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:32.794165 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 07:02:32.794331 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:32.794247 5916 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 07:02:32.794404 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:32.794362 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-68l4r\"" Nov 02 07:02:32.794470 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:32.794452 5916 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 07:02:32.818270 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:32.818246 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0da5ba01-4d64-4344-a6fc-3918297eb4f7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-6ff85e-zpz86\" (UID: \"0da5ba01-4d64-4344-a6fc-3918297eb4f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-6ff85e-zpz86" Nov 02 07:02:32.818368 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:32.818289 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6p59\" (UniqueName: \"kubernetes.io/projected/0da5ba01-4d64-4344-a6fc-3918297eb4f7-kube-api-access-b6p59\") pod \"maas-api-revoke-keys-e2e-worker-w6-6ff85e-zpz86\" (UID: \"0da5ba01-4d64-4344-a6fc-3918297eb4f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-6ff85e-zpz86" Nov 02 07:02:32.818368 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:32.818332 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0da5ba01-4d64-4344-a6fc-3918297eb4f7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-6ff85e-zpz86\" (UID: \"0da5ba01-4d64-4344-a6fc-3918297eb4f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-6ff85e-zpz86" Nov 02 07:02:32.918834 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:32.918807 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0da5ba01-4d64-4344-a6fc-3918297eb4f7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-6ff85e-zpz86\" (UID: \"0da5ba01-4d64-4344-a6fc-3918297eb4f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-6ff85e-zpz86" Nov 02 07:02:32.918969 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:32.918856 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6p59\" (UniqueName: \"kubernetes.io/projected/0da5ba01-4d64-4344-a6fc-3918297eb4f7-kube-api-access-b6p59\") pod \"maas-api-revoke-keys-e2e-worker-w6-6ff85e-zpz86\" (UID: \"0da5ba01-4d64-4344-a6fc-3918297eb4f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-6ff85e-zpz86" Nov 02 07:02:32.919069 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:32.919050 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0da5ba01-4d64-4344-a6fc-3918297eb4f7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-6ff85e-zpz86\" (UID: \"0da5ba01-4d64-4344-a6fc-3918297eb4f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-6ff85e-zpz86" Nov 02 07:02:32.919658 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:32.919633 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0da5ba01-4d64-4344-a6fc-3918297eb4f7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-6ff85e-zpz86\" (UID: \"0da5ba01-4d64-4344-a6fc-3918297eb4f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-6ff85e-zpz86" Nov 02 07:02:32.921161 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:32.921144 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0da5ba01-4d64-4344-a6fc-3918297eb4f7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-6ff85e-zpz86\" (UID: \"0da5ba01-4d64-4344-a6fc-3918297eb4f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-6ff85e-zpz86" Nov 02 07:02:32.975561 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:32.975531 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6p59\" (UniqueName: \"kubernetes.io/projected/0da5ba01-4d64-4344-a6fc-3918297eb4f7-kube-api-access-b6p59\") pod \"maas-api-revoke-keys-e2e-worker-w6-6ff85e-zpz86\" (UID: \"0da5ba01-4d64-4344-a6fc-3918297eb4f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-6ff85e-zpz86" Nov 02 07:02:33.054777 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:33.054708 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-6ff85e-zpz86" Nov 02 07:02:33.426219 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:33.426189 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-6ff85e-zpz86"] Nov 02 07:02:33.427776 ip-10-0-130-93 kubenswrapper[5916]: W1102 07:02:33.427750 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0da5ba01_4d64_4344_a6fc_3918297eb4f7.slice/crio-f857b77ab1cdc2f933d763b9d4804325c66b2c623c65df38e78df6dc58fde3cd WatchSource:0}: Error finding container f857b77ab1cdc2f933d763b9d4804325c66b2c623c65df38e78df6dc58fde3cd: Status 404 returned error can't find the container with id f857b77ab1cdc2f933d763b9d4804325c66b2c623c65df38e78df6dc58fde3cd Nov 02 07:02:34.400268 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:34.400227 5916 generic.go:356] "Generic (PLEG): container finished" podID="0da5ba01-4d64-4344-a6fc-3918297eb4f7" containerID="eb1c5ed06cfdb6920145dcd563a297fe5426bdb5c849824fd57a18a2ee2504c3" exitCode=0 Nov 02 07:02:34.400694 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:34.400296 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-6ff85e-zpz86" event={"ID":"0da5ba01-4d64-4344-a6fc-3918297eb4f7","Type":"ContainerDied","Data":"eb1c5ed06cfdb6920145dcd563a297fe5426bdb5c849824fd57a18a2ee2504c3"} Nov 02 07:02:34.400694 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:34.400343 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-6ff85e-zpz86" event={"ID":"0da5ba01-4d64-4344-a6fc-3918297eb4f7","Type":"ContainerStarted","Data":"f857b77ab1cdc2f933d763b9d4804325c66b2c623c65df38e78df6dc58fde3cd"} Nov 02 07:02:35.539636 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:35.539608 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-6ff85e-zpz86" Nov 02 07:02:35.643727 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:35.643702 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0da5ba01-4d64-4344-a6fc-3918297eb4f7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0da5ba01-4d64-4344-a6fc-3918297eb4f7-maas-api-xks-ca\") pod \"0da5ba01-4d64-4344-a6fc-3918297eb4f7\" (UID: \"0da5ba01-4d64-4344-a6fc-3918297eb4f7\") " Nov 02 07:02:35.643877 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:35.643754 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0da5ba01-4d64-4344-a6fc-3918297eb4f7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0da5ba01-4d64-4344-a6fc-3918297eb4f7-maas-api-openshift-ca\") pod \"0da5ba01-4d64-4344-a6fc-3918297eb4f7\" (UID: \"0da5ba01-4d64-4344-a6fc-3918297eb4f7\") " Nov 02 07:02:35.643877 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:35.643860 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0da5ba01-4d64-4344-a6fc-3918297eb4f7-kube-api-access-b6p59\" (UniqueName: \"kubernetes.io/projected/0da5ba01-4d64-4344-a6fc-3918297eb4f7-kube-api-access-b6p59\") pod \"0da5ba01-4d64-4344-a6fc-3918297eb4f7\" (UID: \"0da5ba01-4d64-4344-a6fc-3918297eb4f7\") " Nov 02 07:02:35.644078 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:35.644060 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0da5ba01-4d64-4344-a6fc-3918297eb4f7-maas-api-openshift-ca" pod "0da5ba01-4d64-4344-a6fc-3918297eb4f7" (UID: "0da5ba01-4d64-4344-a6fc-3918297eb4f7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:02:35.644205 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:35.644185 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0da5ba01-4d64-4344-a6fc-3918297eb4f7-maas-api-openshift-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:02:35.645834 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:35.645812 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0da5ba01-4d64-4344-a6fc-3918297eb4f7-kube-api-access-b6p59" pod "0da5ba01-4d64-4344-a6fc-3918297eb4f7" (UID: "0da5ba01-4d64-4344-a6fc-3918297eb4f7"). InnerVolumeSpecName "kube-api-access-b6p59". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:02:35.645923 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:35.645836 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0da5ba01-4d64-4344-a6fc-3918297eb4f7-maas-api-xks-ca" pod "0da5ba01-4d64-4344-a6fc-3918297eb4f7" (UID: "0da5ba01-4d64-4344-a6fc-3918297eb4f7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 07:02:35.745678 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:35.745617 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6p59\" (UniqueName: \"kubernetes.io/projected/0da5ba01-4d64-4344-a6fc-3918297eb4f7-kube-api-access-b6p59\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:02:35.745678 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:35.745643 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0da5ba01-4d64-4344-a6fc-3918297eb4f7-maas-api-xks-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:02:36.415438 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:36.415409 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-6ff85e-zpz86" Nov 02 07:02:36.415633 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:36.415445 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-6ff85e-zpz86" event={"ID":"0da5ba01-4d64-4344-a6fc-3918297eb4f7","Type":"ContainerDied","Data":"f857b77ab1cdc2f933d763b9d4804325c66b2c623c65df38e78df6dc58fde3cd"} Nov 02 07:02:36.415633 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:36.415499 5916 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f857b77ab1cdc2f933d763b9d4804325c66b2c623c65df38e78df6dc58fde3cd" Nov 02 07:02:37.195264 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:37.195230 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-6ff85e/premium-w6-6ff85e-kserve-59848ff6d4-szbxx"] Nov 02 07:02:37.244866 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:37.244833 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-6ff85e/simulator-w6-6ff85e-kserve-9b9fd978f-q76nl"] Nov 02 07:02:44.145353 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:44.145318 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-6ff85e-7c6f4dfd87-c5m8z"] Nov 02 07:02:44.249934 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:44.249871 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-6ff85e-7fd4b5f85f-4wcpn"] Nov 02 07:02:45.773777 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:45.773741 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d48fbdd4-5cfql"] Nov 02 07:02:45.774302 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:45.774243 5916 cpu_manager.go:414] "containerMap: removing container" podUID="0da5ba01-4d64-4344-a6fc-3918297eb4f7" containerName="revoke-keys" Nov 02 07:02:45.774302 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:45.774261 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0da5ba01-4d64-4344-a6fc-3918297eb4f7" containerName="revoke-keys" Nov 02 07:02:45.774404 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:45.774375 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="0da5ba01-4d64-4344-a6fc-3918297eb4f7" containerName="revoke-keys" Nov 02 07:02:45.778894 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:45.778871 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d48fbdd4-5cfql" Nov 02 07:02:45.833692 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:45.833662 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d48fbdd4-5cfql"] Nov 02 07:02:45.880833 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:45.880807 5916 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 07:02:45.880999 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:45.880915 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 07:02:45.880999 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:45.880935 5916 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 07:02:45.881541 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:45.881525 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-68l4r\"" Nov 02 07:02:45.933289 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:45.933251 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db3677b9-3aa0-415f-a6a0-e7e87e2aad7f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d48fbdd4-5cfql\" (UID: \"db3677b9-3aa0-415f-a6a0-e7e87e2aad7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d48fbdd4-5cfql" Nov 02 07:02:45.933462 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:45.933436 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db3677b9-3aa0-415f-a6a0-e7e87e2aad7f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d48fbdd4-5cfql\" (UID: \"db3677b9-3aa0-415f-a6a0-e7e87e2aad7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d48fbdd4-5cfql" Nov 02 07:02:45.933535 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:45.933517 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n79mn\" (UniqueName: \"kubernetes.io/projected/db3677b9-3aa0-415f-a6a0-e7e87e2aad7f-kube-api-access-n79mn\") pod \"maas-api-revoke-sub-d48fbdd4-5cfql\" (UID: \"db3677b9-3aa0-415f-a6a0-e7e87e2aad7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d48fbdd4-5cfql" Nov 02 07:02:46.034620 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:46.034536 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n79mn\" (UniqueName: \"kubernetes.io/projected/db3677b9-3aa0-415f-a6a0-e7e87e2aad7f-kube-api-access-n79mn\") pod \"maas-api-revoke-sub-d48fbdd4-5cfql\" (UID: \"db3677b9-3aa0-415f-a6a0-e7e87e2aad7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d48fbdd4-5cfql" Nov 02 07:02:46.034620 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:46.034592 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db3677b9-3aa0-415f-a6a0-e7e87e2aad7f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d48fbdd4-5cfql\" (UID: \"db3677b9-3aa0-415f-a6a0-e7e87e2aad7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d48fbdd4-5cfql" Nov 02 07:02:46.034863 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:46.034657 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db3677b9-3aa0-415f-a6a0-e7e87e2aad7f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d48fbdd4-5cfql\" (UID: \"db3677b9-3aa0-415f-a6a0-e7e87e2aad7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d48fbdd4-5cfql" Nov 02 07:02:46.035371 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:46.035349 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db3677b9-3aa0-415f-a6a0-e7e87e2aad7f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d48fbdd4-5cfql\" (UID: \"db3677b9-3aa0-415f-a6a0-e7e87e2aad7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d48fbdd4-5cfql" Nov 02 07:02:46.037052 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:46.037031 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db3677b9-3aa0-415f-a6a0-e7e87e2aad7f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d48fbdd4-5cfql\" (UID: \"db3677b9-3aa0-415f-a6a0-e7e87e2aad7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d48fbdd4-5cfql" Nov 02 07:02:46.093127 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:46.093105 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n79mn\" (UniqueName: \"kubernetes.io/projected/db3677b9-3aa0-415f-a6a0-e7e87e2aad7f-kube-api-access-n79mn\") pod \"maas-api-revoke-sub-d48fbdd4-5cfql\" (UID: \"db3677b9-3aa0-415f-a6a0-e7e87e2aad7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d48fbdd4-5cfql" Nov 02 07:02:46.392225 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:46.392198 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d48fbdd4-5cfql" Nov 02 07:02:46.767302 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:46.767266 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d48fbdd4-5cfql"] Nov 02 07:02:46.767945 ip-10-0-130-93 kubenswrapper[5916]: W1102 07:02:46.767919 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb3677b9_3aa0_415f_a6a0_e7e87e2aad7f.slice/crio-5a99502453889354fb1761dcded003ddbfac356a31ccd552a72882604aca6b7d WatchSource:0}: Error finding container 5a99502453889354fb1761dcded003ddbfac356a31ccd552a72882604aca6b7d: Status 404 returned error can't find the container with id 5a99502453889354fb1761dcded003ddbfac356a31ccd552a72882604aca6b7d Nov 02 07:02:47.487757 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:47.487718 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d48fbdd4-5cfql" event={"ID":"db3677b9-3aa0-415f-a6a0-e7e87e2aad7f","Type":"ContainerStarted","Data":"dd9b9bc5ce94ad8319a4888a09ec6e69a660ad7e1634f6b3683c3f678275bb12"} Nov 02 07:02:47.488213 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:47.487779 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d48fbdd4-5cfql" event={"ID":"db3677b9-3aa0-415f-a6a0-e7e87e2aad7f","Type":"ContainerStarted","Data":"5a99502453889354fb1761dcded003ddbfac356a31ccd552a72882604aca6b7d"} Nov 02 07:02:47.601485 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:47.601439 5916 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d48fbdd4-5cfql" podStartSLOduration=2.601426511 podStartE2EDuration="2.601426511s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:02:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:02:47.600683933 +0000 UTC m=+2755.872504329" watchObservedRunningTime="2026-11-02 07:02:47.601426511 +0000 UTC m=+2755.873246904" Nov 02 07:02:48.495756 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:48.495719 5916 generic.go:356] "Generic (PLEG): container finished" podID="db3677b9-3aa0-415f-a6a0-e7e87e2aad7f" containerID="dd9b9bc5ce94ad8319a4888a09ec6e69a660ad7e1634f6b3683c3f678275bb12" exitCode=0 Nov 02 07:02:48.496171 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:48.495795 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d48fbdd4-5cfql" event={"ID":"db3677b9-3aa0-415f-a6a0-e7e87e2aad7f","Type":"ContainerDied","Data":"dd9b9bc5ce94ad8319a4888a09ec6e69a660ad7e1634f6b3683c3f678275bb12"} Nov 02 07:02:49.146396 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:49.146353 5916 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w6-6ff85e-7c6f4dfd87-c5m8z" podUID="ee6a9440-ee89-4c29-aa2b-eb34c10935bf" containerName="payload-processing" containerID="cri-o://aa49170d5fb55818a96f24ed50fe303d658eb084693107e35c5e6d9c0eded4d4" gracePeriod=25 Nov 02 07:02:49.250394 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:49.250349 5916 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-6ff85e-7fd4b5f85f-4wcpn" podUID="771371d6-10a2-4679-8a9e-3ca2e695f4db" containerName="payload-pre-processing" containerID="cri-o://bc5ad09cad1229e5799b0ec72209b54e2a2e835508faa0b019607cb64356d848" gracePeriod=25 Nov 02 07:02:49.640518 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:49.640493 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d48fbdd4-5cfql" Nov 02 07:02:49.766634 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:49.766550 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db3677b9-3aa0-415f-a6a0-e7e87e2aad7f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db3677b9-3aa0-415f-a6a0-e7e87e2aad7f-maas-api-openshift-ca\") pod \"db3677b9-3aa0-415f-a6a0-e7e87e2aad7f\" (UID: \"db3677b9-3aa0-415f-a6a0-e7e87e2aad7f\") " Nov 02 07:02:49.766786 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:49.766651 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db3677b9-3aa0-415f-a6a0-e7e87e2aad7f-kube-api-access-n79mn\" (UniqueName: \"kubernetes.io/projected/db3677b9-3aa0-415f-a6a0-e7e87e2aad7f-kube-api-access-n79mn\") pod \"db3677b9-3aa0-415f-a6a0-e7e87e2aad7f\" (UID: \"db3677b9-3aa0-415f-a6a0-e7e87e2aad7f\") " Nov 02 07:02:49.766786 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:49.766695 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db3677b9-3aa0-415f-a6a0-e7e87e2aad7f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db3677b9-3aa0-415f-a6a0-e7e87e2aad7f-maas-api-xks-ca\") pod \"db3677b9-3aa0-415f-a6a0-e7e87e2aad7f\" (UID: \"db3677b9-3aa0-415f-a6a0-e7e87e2aad7f\") " Nov 02 07:02:49.766970 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:49.766945 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db3677b9-3aa0-415f-a6a0-e7e87e2aad7f-maas-api-openshift-ca" pod "db3677b9-3aa0-415f-a6a0-e7e87e2aad7f" (UID: "db3677b9-3aa0-415f-a6a0-e7e87e2aad7f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:02:49.767124 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:49.767100 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db3677b9-3aa0-415f-a6a0-e7e87e2aad7f-maas-api-openshift-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:02:49.769415 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:49.769391 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db3677b9-3aa0-415f-a6a0-e7e87e2aad7f-kube-api-access-n79mn" pod "db3677b9-3aa0-415f-a6a0-e7e87e2aad7f" (UID: "db3677b9-3aa0-415f-a6a0-e7e87e2aad7f"). InnerVolumeSpecName "kube-api-access-n79mn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:02:49.769526 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:49.769454 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db3677b9-3aa0-415f-a6a0-e7e87e2aad7f-maas-api-xks-ca" pod "db3677b9-3aa0-415f-a6a0-e7e87e2aad7f" (UID: "db3677b9-3aa0-415f-a6a0-e7e87e2aad7f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 07:02:49.867577 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:49.867548 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n79mn\" (UniqueName: \"kubernetes.io/projected/db3677b9-3aa0-415f-a6a0-e7e87e2aad7f-kube-api-access-n79mn\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:02:49.867577 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:49.867578 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db3677b9-3aa0-415f-a6a0-e7e87e2aad7f-maas-api-xks-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:02:49.893800 ip-10-0-130-93 kubenswrapper[5916]: E1102 07:02:49.893766 5916 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p98cq" containerName="extractor" Nov 02 07:02:50.508889 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:50.508859 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d48fbdd4-5cfql" Nov 02 07:02:50.509090 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:50.508891 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d48fbdd4-5cfql" event={"ID":"db3677b9-3aa0-415f-a6a0-e7e87e2aad7f","Type":"ContainerDied","Data":"5a99502453889354fb1761dcded003ddbfac356a31ccd552a72882604aca6b7d"} Nov 02 07:02:50.509090 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:50.508955 5916 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5a99502453889354fb1761dcded003ddbfac356a31ccd552a72882604aca6b7d" Nov 02 07:02:52.215685 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:52.215638 5916 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-6ff85e/premium-w6-6ff85e-kserve-59848ff6d4-szbxx" podUID="eb386976-f898-4f1d-b27c-77bccc735a3b" containerName="main" containerID="cri-o://9d87b95c2d781f8e2b67a8a2cdb01b2e44a1ddd9bd7c0b2b5ee58c70a2973db7" gracePeriod=45 Nov 02 07:02:52.263818 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:52.263774 5916 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-6ff85e/simulator-w6-6ff85e-kserve-9b9fd978f-q76nl" podUID="30114285-d730-4abb-a618-6833d5746feb" containerName="main" containerID="cri-o://fc5bf7005819795b62cfed8b3b6ed7aaafa51099457a92ab763b25e22c585130" gracePeriod=45 Nov 02 07:02:52.523547 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:52.523521 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-6ff85e/premium-w6-6ff85e-kserve-59848ff6d4-szbxx" Nov 02 07:02:52.525258 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:52.525232 5916 generic.go:356] "Generic (PLEG): container finished" podID="eb386976-f898-4f1d-b27c-77bccc735a3b" containerID="9d87b95c2d781f8e2b67a8a2cdb01b2e44a1ddd9bd7c0b2b5ee58c70a2973db7" exitCode=0 Nov 02 07:02:52.525375 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:52.525277 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-6ff85e/premium-w6-6ff85e-kserve-59848ff6d4-szbxx" Nov 02 07:02:52.525375 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:52.525324 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-6ff85e/premium-w6-6ff85e-kserve-59848ff6d4-szbxx" event={"ID":"eb386976-f898-4f1d-b27c-77bccc735a3b","Type":"ContainerDied","Data":"9d87b95c2d781f8e2b67a8a2cdb01b2e44a1ddd9bd7c0b2b5ee58c70a2973db7"} Nov 02 07:02:52.525510 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:52.525373 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-6ff85e/premium-w6-6ff85e-kserve-59848ff6d4-szbxx" event={"ID":"eb386976-f898-4f1d-b27c-77bccc735a3b","Type":"ContainerDied","Data":"ffea19bfea5016d03ff383440b417e03d9599ab839b43625b2c688c451895b70"} Nov 02 07:02:52.525510 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:52.525397 5916 scope.go:122] "RemoveContainer" containerID="9d87b95c2d781f8e2b67a8a2cdb01b2e44a1ddd9bd7c0b2b5ee58c70a2973db7" Nov 02 07:02:52.527520 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:52.527353 5916 generic.go:356] "Generic (PLEG): container finished" podID="30114285-d730-4abb-a618-6833d5746feb" containerID="fc5bf7005819795b62cfed8b3b6ed7aaafa51099457a92ab763b25e22c585130" exitCode=0 Nov 02 07:02:52.527520 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:52.527468 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-6ff85e/simulator-w6-6ff85e-kserve-9b9fd978f-q76nl" event={"ID":"30114285-d730-4abb-a618-6833d5746feb","Type":"ContainerDied","Data":"fc5bf7005819795b62cfed8b3b6ed7aaafa51099457a92ab763b25e22c585130"} Nov 02 07:02:52.537079 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:52.537056 5916 scope.go:122] "RemoveContainer" containerID="9d87b95c2d781f8e2b67a8a2cdb01b2e44a1ddd9bd7c0b2b5ee58c70a2973db7" Nov 02 07:02:52.537408 ip-10-0-130-93 kubenswrapper[5916]: E1102 07:02:52.537383 5916 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d87b95c2d781f8e2b67a8a2cdb01b2e44a1ddd9bd7c0b2b5ee58c70a2973db7\": container with ID starting with 9d87b95c2d781f8e2b67a8a2cdb01b2e44a1ddd9bd7c0b2b5ee58c70a2973db7 not found: ID does not exist" containerID="9d87b95c2d781f8e2b67a8a2cdb01b2e44a1ddd9bd7c0b2b5ee58c70a2973db7" Nov 02 07:02:52.537510 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:52.537417 5916 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d87b95c2d781f8e2b67a8a2cdb01b2e44a1ddd9bd7c0b2b5ee58c70a2973db7"} err="failed to get container status \"9d87b95c2d781f8e2b67a8a2cdb01b2e44a1ddd9bd7c0b2b5ee58c70a2973db7\": rpc error: code = NotFound desc = could not find container \"9d87b95c2d781f8e2b67a8a2cdb01b2e44a1ddd9bd7c0b2b5ee58c70a2973db7\": container with ID starting with 9d87b95c2d781f8e2b67a8a2cdb01b2e44a1ddd9bd7c0b2b5ee58c70a2973db7 not found: ID does not exist" Nov 02 07:02:52.559841 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:52.559820 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-6ff85e/simulator-w6-6ff85e-kserve-9b9fd978f-q76nl" Nov 02 07:02:52.594685 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:52.594658 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb386976-f898-4f1d-b27c-77bccc735a3b-tls-certs\" (UniqueName: \"kubernetes.io/secret/eb386976-f898-4f1d-b27c-77bccc735a3b-tls-certs\") pod \"eb386976-f898-4f1d-b27c-77bccc735a3b\" (UID: \"eb386976-f898-4f1d-b27c-77bccc735a3b\") " Nov 02 07:02:52.594851 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:52.594700 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb386976-f898-4f1d-b27c-77bccc735a3b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb386976-f898-4f1d-b27c-77bccc735a3b-model-cache\") pod \"eb386976-f898-4f1d-b27c-77bccc735a3b\" (UID: \"eb386976-f898-4f1d-b27c-77bccc735a3b\") " Nov 02 07:02:52.594851 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:52.594721 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb386976-f898-4f1d-b27c-77bccc735a3b-home\" (UniqueName: \"kubernetes.io/empty-dir/eb386976-f898-4f1d-b27c-77bccc735a3b-home\") pod \"eb386976-f898-4f1d-b27c-77bccc735a3b\" (UID: \"eb386976-f898-4f1d-b27c-77bccc735a3b\") " Nov 02 07:02:52.594851 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:52.594737 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb386976-f898-4f1d-b27c-77bccc735a3b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb386976-f898-4f1d-b27c-77bccc735a3b-dshm\") pod \"eb386976-f898-4f1d-b27c-77bccc735a3b\" (UID: \"eb386976-f898-4f1d-b27c-77bccc735a3b\") " Nov 02 07:02:52.594851 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:52.594786 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb386976-f898-4f1d-b27c-77bccc735a3b-kube-api-access-9v5bm\" (UniqueName: \"kubernetes.io/projected/eb386976-f898-4f1d-b27c-77bccc735a3b-kube-api-access-9v5bm\") pod \"eb386976-f898-4f1d-b27c-77bccc735a3b\" (UID: \"eb386976-f898-4f1d-b27c-77bccc735a3b\") " Nov 02 07:02:52.594851 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:52.594804 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb386976-f898-4f1d-b27c-77bccc735a3b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb386976-f898-4f1d-b27c-77bccc735a3b-tmp-dir\") pod \"eb386976-f898-4f1d-b27c-77bccc735a3b\" (UID: \"eb386976-f898-4f1d-b27c-77bccc735a3b\") " Nov 02 07:02:52.595168 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:52.594969 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb386976-f898-4f1d-b27c-77bccc735a3b-model-cache" pod "eb386976-f898-4f1d-b27c-77bccc735a3b" (UID: "eb386976-f898-4f1d-b27c-77bccc735a3b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:02:52.595168 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:52.594987 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb386976-f898-4f1d-b27c-77bccc735a3b-home" pod "eb386976-f898-4f1d-b27c-77bccc735a3b" (UID: "eb386976-f898-4f1d-b27c-77bccc735a3b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:02:52.595168 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:52.595149 5916 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb386976-f898-4f1d-b27c-77bccc735a3b-model-cache\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:02:52.595290 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:52.595169 5916 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb386976-f898-4f1d-b27c-77bccc735a3b-home\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:02:52.595290 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:52.595166 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb386976-f898-4f1d-b27c-77bccc735a3b-tmp-dir" pod "eb386976-f898-4f1d-b27c-77bccc735a3b" (UID: "eb386976-f898-4f1d-b27c-77bccc735a3b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:02:52.596936 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:52.596886 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb386976-f898-4f1d-b27c-77bccc735a3b-dshm" pod "eb386976-f898-4f1d-b27c-77bccc735a3b" (UID: "eb386976-f898-4f1d-b27c-77bccc735a3b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:02:52.597044 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:52.597029 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb386976-f898-4f1d-b27c-77bccc735a3b-tls-certs" pod "eb386976-f898-4f1d-b27c-77bccc735a3b" (UID: "eb386976-f898-4f1d-b27c-77bccc735a3b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 07:02:52.598048 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:52.598024 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb386976-f898-4f1d-b27c-77bccc735a3b-kube-api-access-9v5bm" pod "eb386976-f898-4f1d-b27c-77bccc735a3b" (UID: "eb386976-f898-4f1d-b27c-77bccc735a3b"). InnerVolumeSpecName "kube-api-access-9v5bm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:02:52.696082 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:52.696047 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30114285-d730-4abb-a618-6833d5746feb-home\" (UniqueName: \"kubernetes.io/empty-dir/30114285-d730-4abb-a618-6833d5746feb-home\") pod \"30114285-d730-4abb-a618-6833d5746feb\" (UID: \"30114285-d730-4abb-a618-6833d5746feb\") " Nov 02 07:02:52.696275 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:52.696108 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30114285-d730-4abb-a618-6833d5746feb-kube-api-access-x5rtk\" (UniqueName: \"kubernetes.io/projected/30114285-d730-4abb-a618-6833d5746feb-kube-api-access-x5rtk\") pod \"30114285-d730-4abb-a618-6833d5746feb\" (UID: \"30114285-d730-4abb-a618-6833d5746feb\") " Nov 02 07:02:52.696275 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:52.696139 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/30114285-d730-4abb-a618-6833d5746feb-tls-certs\" (UniqueName: \"kubernetes.io/secret/30114285-d730-4abb-a618-6833d5746feb-tls-certs\") pod \"30114285-d730-4abb-a618-6833d5746feb\" (UID: \"30114285-d730-4abb-a618-6833d5746feb\") " Nov 02 07:02:52.696417 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:52.696296 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30114285-d730-4abb-a618-6833d5746feb-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30114285-d730-4abb-a618-6833d5746feb-tmp-dir\") pod \"30114285-d730-4abb-a618-6833d5746feb\" (UID: \"30114285-d730-4abb-a618-6833d5746feb\") " Nov 02 07:02:52.696417 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:52.696346 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30114285-d730-4abb-a618-6833d5746feb-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30114285-d730-4abb-a618-6833d5746feb-model-cache\") pod \"30114285-d730-4abb-a618-6833d5746feb\" (UID: \"30114285-d730-4abb-a618-6833d5746feb\") " Nov 02 07:02:52.696417 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:52.696353 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30114285-d730-4abb-a618-6833d5746feb-home" pod "30114285-d730-4abb-a618-6833d5746feb" (UID: "30114285-d730-4abb-a618-6833d5746feb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:02:52.696417 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:52.696380 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30114285-d730-4abb-a618-6833d5746feb-dshm\" (UniqueName: \"kubernetes.io/empty-dir/30114285-d730-4abb-a618-6833d5746feb-dshm\") pod \"30114285-d730-4abb-a618-6833d5746feb\" (UID: \"30114285-d730-4abb-a618-6833d5746feb\") " Nov 02 07:02:52.696632 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:52.696562 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30114285-d730-4abb-a618-6833d5746feb-tmp-dir" pod "30114285-d730-4abb-a618-6833d5746feb" (UID: "30114285-d730-4abb-a618-6833d5746feb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:02:52.696738 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:52.696721 5916 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb386976-f898-4f1d-b27c-77bccc735a3b-tls-certs\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:02:52.696799 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:52.696742 5916 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb386976-f898-4f1d-b27c-77bccc735a3b-dshm\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:02:52.696799 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:52.696755 5916 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30114285-d730-4abb-a618-6833d5746feb-tmp-dir\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:02:52.696799 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:52.696769 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9v5bm\" (UniqueName: \"kubernetes.io/projected/eb386976-f898-4f1d-b27c-77bccc735a3b-kube-api-access-9v5bm\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:02:52.696799 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:52.696783 5916 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb386976-f898-4f1d-b27c-77bccc735a3b-tmp-dir\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:02:52.696799 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:52.696797 5916 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30114285-d730-4abb-a618-6833d5746feb-home\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:02:52.697091 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:52.697067 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30114285-d730-4abb-a618-6833d5746feb-model-cache" pod "30114285-d730-4abb-a618-6833d5746feb" (UID: "30114285-d730-4abb-a618-6833d5746feb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:02:52.698705 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:52.698682 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30114285-d730-4abb-a618-6833d5746feb-dshm" pod "30114285-d730-4abb-a618-6833d5746feb" (UID: "30114285-d730-4abb-a618-6833d5746feb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:02:52.699155 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:52.699130 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30114285-d730-4abb-a618-6833d5746feb-tls-certs" pod "30114285-d730-4abb-a618-6833d5746feb" (UID: "30114285-d730-4abb-a618-6833d5746feb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 07:02:52.699283 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:52.699231 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30114285-d730-4abb-a618-6833d5746feb-kube-api-access-x5rtk" pod "30114285-d730-4abb-a618-6833d5746feb" (UID: "30114285-d730-4abb-a618-6833d5746feb"). InnerVolumeSpecName "kube-api-access-x5rtk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:02:52.837981 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:52.837372 5916 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30114285-d730-4abb-a618-6833d5746feb-tls-certs\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:02:52.837981 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:52.837419 5916 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30114285-d730-4abb-a618-6833d5746feb-model-cache\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:02:52.837981 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:52.837434 5916 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30114285-d730-4abb-a618-6833d5746feb-dshm\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:02:52.837981 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:52.837448 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5rtk\" (UniqueName: \"kubernetes.io/projected/30114285-d730-4abb-a618-6833d5746feb-kube-api-access-x5rtk\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:02:52.969977 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:52.969934 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-6ff85e/premium-w6-6ff85e-kserve-59848ff6d4-szbxx"] Nov 02 07:02:52.972834 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:52.972808 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-6ff85e/premium-w6-6ff85e-kserve-59848ff6d4-szbxx"] Nov 02 07:02:53.519524 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:53.519437 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6ff85e/simulator-w6-6ff85e-kserve-9b9fd978f-q76nl" podUID="30114285-d730-4abb-a618-6833d5746feb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Nov 02 07:02:53.541486 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:53.541458 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-6ff85e/simulator-w6-6ff85e-kserve-9b9fd978f-q76nl" Nov 02 07:02:53.541649 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:53.541500 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-6ff85e/simulator-w6-6ff85e-kserve-9b9fd978f-q76nl" event={"ID":"30114285-d730-4abb-a618-6833d5746feb","Type":"ContainerDied","Data":"9bc4e63c81f118f24274f025423f90bdb072a07484a9ded2613bba329a293d85"} Nov 02 07:02:53.541649 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:53.541551 5916 scope.go:122] "RemoveContainer" containerID="fc5bf7005819795b62cfed8b3b6ed7aaafa51099457a92ab763b25e22c585130" Nov 02 07:02:53.705897 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:53.705869 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-6ff85e/simulator-w6-6ff85e-kserve-9b9fd978f-q76nl"] Nov 02 07:02:53.709169 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:53.709144 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-6ff85e/simulator-w6-6ff85e-kserve-9b9fd978f-q76nl"] Nov 02 07:02:53.842247 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:53.842179 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30114285-d730-4abb-a618-6833d5746feb" path="/var/lib/kubelet/pods/30114285-d730-4abb-a618-6833d5746feb/volumes" Nov 02 07:02:53.842576 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:53.842563 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb386976-f898-4f1d-b27c-77bccc735a3b" path="/var/lib/kubelet/pods/eb386976-f898-4f1d-b27c-77bccc735a3b/volumes" Nov 02 07:02:55.706644 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:55.706605 5916 scope.go:122] "RemoveContainer" containerID="8da2b6040ea1cb63e340397628762e5772f58e5906372dc38798364ae84b2470" Nov 02 07:02:55.887063 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:55.887034 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-6ff85e-7fd4b5f85f-4wcpn" podUID="771371d6-10a2-4679-8a9e-3ca2e695f4db" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 07:02:55.887195 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:02:55.887069 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-6ff85e-7c6f4dfd87-c5m8z" podUID="ee6a9440-ee89-4c29-aa2b-eb34c10935bf" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 07:03:03.104392 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:03.104354 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-6ff85e-zpz86"] Nov 02 07:03:03.154077 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:03.154047 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-6ff85e-zpz86"] Nov 02 07:03:03.841349 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:03.841308 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0da5ba01-4d64-4344-a6fc-3918297eb4f7" path="/var/lib/kubelet/pods/0da5ba01-4d64-4344-a6fc-3918297eb4f7/volumes" Nov 02 07:03:05.887106 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:05.887070 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-6ff85e-7c6f4dfd87-c5m8z" podUID="ee6a9440-ee89-4c29-aa2b-eb34c10935bf" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 07:03:05.887544 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:05.887068 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-6ff85e-7fd4b5f85f-4wcpn" podUID="771371d6-10a2-4679-8a9e-3ca2e695f4db" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 07:03:09.411308 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:09.411249 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-6ff85e-7c6f4dfd87-c5m8z" Nov 02 07:03:09.482043 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:09.482013 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee6a9440-ee89-4c29-aa2b-eb34c10935bf-kube-api-access-l9282\" (UniqueName: \"kubernetes.io/projected/ee6a9440-ee89-4c29-aa2b-eb34c10935bf-kube-api-access-l9282\") pod \"ee6a9440-ee89-4c29-aa2b-eb34c10935bf\" (UID: \"ee6a9440-ee89-4c29-aa2b-eb34c10935bf\") " Nov 02 07:03:09.482221 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:09.482086 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee6a9440-ee89-4c29-aa2b-eb34c10935bf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee6a9440-ee89-4c29-aa2b-eb34c10935bf-plugins-config-volume\") pod \"ee6a9440-ee89-4c29-aa2b-eb34c10935bf\" (UID: \"ee6a9440-ee89-4c29-aa2b-eb34c10935bf\") " Nov 02 07:03:09.482446 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:09.482423 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee6a9440-ee89-4c29-aa2b-eb34c10935bf-plugins-config-volume" pod "ee6a9440-ee89-4c29-aa2b-eb34c10935bf" (UID: "ee6a9440-ee89-4c29-aa2b-eb34c10935bf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:03:09.484187 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:09.484149 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee6a9440-ee89-4c29-aa2b-eb34c10935bf-kube-api-access-l9282" pod "ee6a9440-ee89-4c29-aa2b-eb34c10935bf" (UID: "ee6a9440-ee89-4c29-aa2b-eb34c10935bf"). InnerVolumeSpecName "kube-api-access-l9282". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:03:09.499039 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:09.499018 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-6ff85e-7fd4b5f85f-4wcpn" Nov 02 07:03:09.582890 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:09.582861 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/771371d6-10a2-4679-8a9e-3ca2e695f4db-kube-api-access-zcnd2\" (UniqueName: \"kubernetes.io/projected/771371d6-10a2-4679-8a9e-3ca2e695f4db-kube-api-access-zcnd2\") pod \"771371d6-10a2-4679-8a9e-3ca2e695f4db\" (UID: \"771371d6-10a2-4679-8a9e-3ca2e695f4db\") " Nov 02 07:03:09.583085 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:09.582951 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/771371d6-10a2-4679-8a9e-3ca2e695f4db-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/771371d6-10a2-4679-8a9e-3ca2e695f4db-plugins-config-volume\") pod \"771371d6-10a2-4679-8a9e-3ca2e695f4db\" (UID: \"771371d6-10a2-4679-8a9e-3ca2e695f4db\") " Nov 02 07:03:09.583190 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:09.583173 5916 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee6a9440-ee89-4c29-aa2b-eb34c10935bf-plugins-config-volume\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:03:09.583190 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:09.583187 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9282\" (UniqueName: \"kubernetes.io/projected/ee6a9440-ee89-4c29-aa2b-eb34c10935bf-kube-api-access-l9282\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:03:09.583332 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:09.583291 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/771371d6-10a2-4679-8a9e-3ca2e695f4db-plugins-config-volume" pod "771371d6-10a2-4679-8a9e-3ca2e695f4db" (UID: "771371d6-10a2-4679-8a9e-3ca2e695f4db"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:03:09.584865 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:09.584845 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/771371d6-10a2-4679-8a9e-3ca2e695f4db-kube-api-access-zcnd2" pod "771371d6-10a2-4679-8a9e-3ca2e695f4db" (UID: "771371d6-10a2-4679-8a9e-3ca2e695f4db"). InnerVolumeSpecName "kube-api-access-zcnd2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:03:09.642298 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:09.642265 5916 generic.go:356] "Generic (PLEG): container finished" podID="771371d6-10a2-4679-8a9e-3ca2e695f4db" containerID="bc5ad09cad1229e5799b0ec72209b54e2a2e835508faa0b019607cb64356d848" exitCode=0 Nov 02 07:03:09.642473 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:09.642310 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-6ff85e-7fd4b5f85f-4wcpn" Nov 02 07:03:09.642473 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:09.642345 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-6ff85e-7fd4b5f85f-4wcpn" event={"ID":"771371d6-10a2-4679-8a9e-3ca2e695f4db","Type":"ContainerDied","Data":"bc5ad09cad1229e5799b0ec72209b54e2a2e835508faa0b019607cb64356d848"} Nov 02 07:03:09.642473 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:09.642400 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-6ff85e-7fd4b5f85f-4wcpn" event={"ID":"771371d6-10a2-4679-8a9e-3ca2e695f4db","Type":"ContainerDied","Data":"7c392b9efd24406d1261beda11cc9b1b23a73b14639b027458b0a42e9dcf2e42"} Nov 02 07:03:09.642473 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:09.642428 5916 scope.go:122] "RemoveContainer" containerID="bc5ad09cad1229e5799b0ec72209b54e2a2e835508faa0b019607cb64356d848" Nov 02 07:03:09.644041 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:09.644018 5916 generic.go:356] "Generic (PLEG): container finished" podID="ee6a9440-ee89-4c29-aa2b-eb34c10935bf" containerID="aa49170d5fb55818a96f24ed50fe303d658eb084693107e35c5e6d9c0eded4d4" exitCode=0 Nov 02 07:03:09.644158 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:09.644064 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-6ff85e-7c6f4dfd87-c5m8z" Nov 02 07:03:09.644158 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:09.644098 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-6ff85e-7c6f4dfd87-c5m8z" event={"ID":"ee6a9440-ee89-4c29-aa2b-eb34c10935bf","Type":"ContainerDied","Data":"aa49170d5fb55818a96f24ed50fe303d658eb084693107e35c5e6d9c0eded4d4"} Nov 02 07:03:09.644158 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:09.644144 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-6ff85e-7c6f4dfd87-c5m8z" event={"ID":"ee6a9440-ee89-4c29-aa2b-eb34c10935bf","Type":"ContainerDied","Data":"807b3c30a9be1e105bd2fad3a7dec2b764111187b3d3364cc1ae2f003f127547"} Nov 02 07:03:09.653637 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:09.653620 5916 scope.go:122] "RemoveContainer" containerID="bc5ad09cad1229e5799b0ec72209b54e2a2e835508faa0b019607cb64356d848" Nov 02 07:03:09.653896 ip-10-0-130-93 kubenswrapper[5916]: E1102 07:03:09.653876 5916 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc5ad09cad1229e5799b0ec72209b54e2a2e835508faa0b019607cb64356d848\": container with ID starting with bc5ad09cad1229e5799b0ec72209b54e2a2e835508faa0b019607cb64356d848 not found: ID does not exist" containerID="bc5ad09cad1229e5799b0ec72209b54e2a2e835508faa0b019607cb64356d848" Nov 02 07:03:09.653997 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:09.653900 5916 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc5ad09cad1229e5799b0ec72209b54e2a2e835508faa0b019607cb64356d848"} err="failed to get container status \"bc5ad09cad1229e5799b0ec72209b54e2a2e835508faa0b019607cb64356d848\": rpc error: code = NotFound desc = could not find container \"bc5ad09cad1229e5799b0ec72209b54e2a2e835508faa0b019607cb64356d848\": container with ID starting with bc5ad09cad1229e5799b0ec72209b54e2a2e835508faa0b019607cb64356d848 not found: ID does not exist" Nov 02 07:03:09.653997 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:09.653935 5916 scope.go:122] "RemoveContainer" containerID="aa49170d5fb55818a96f24ed50fe303d658eb084693107e35c5e6d9c0eded4d4" Nov 02 07:03:09.664694 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:09.664673 5916 scope.go:122] "RemoveContainer" containerID="aa49170d5fb55818a96f24ed50fe303d658eb084693107e35c5e6d9c0eded4d4" Nov 02 07:03:09.665387 ip-10-0-130-93 kubenswrapper[5916]: E1102 07:03:09.665369 5916 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa49170d5fb55818a96f24ed50fe303d658eb084693107e35c5e6d9c0eded4d4\": container with ID starting with aa49170d5fb55818a96f24ed50fe303d658eb084693107e35c5e6d9c0eded4d4 not found: ID does not exist" containerID="aa49170d5fb55818a96f24ed50fe303d658eb084693107e35c5e6d9c0eded4d4" Nov 02 07:03:09.665461 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:09.665391 5916 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa49170d5fb55818a96f24ed50fe303d658eb084693107e35c5e6d9c0eded4d4"} err="failed to get container status \"aa49170d5fb55818a96f24ed50fe303d658eb084693107e35c5e6d9c0eded4d4\": rpc error: code = NotFound desc = could not find container \"aa49170d5fb55818a96f24ed50fe303d658eb084693107e35c5e6d9c0eded4d4\": container with ID starting with aa49170d5fb55818a96f24ed50fe303d658eb084693107e35c5e6d9c0eded4d4 not found: ID does not exist" Nov 02 07:03:09.684002 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:09.683979 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zcnd2\" (UniqueName: \"kubernetes.io/projected/771371d6-10a2-4679-8a9e-3ca2e695f4db-kube-api-access-zcnd2\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:03:09.684002 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:09.684003 5916 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/771371d6-10a2-4679-8a9e-3ca2e695f4db-plugins-config-volume\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:03:09.806280 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:09.806253 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-6ff85e-7fd4b5f85f-4wcpn"] Nov 02 07:03:09.812227 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:09.812209 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-6ff85e-7fd4b5f85f-4wcpn"] Nov 02 07:03:09.841396 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:09.841369 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="771371d6-10a2-4679-8a9e-3ca2e695f4db" path="/var/lib/kubelet/pods/771371d6-10a2-4679-8a9e-3ca2e695f4db/volumes" Nov 02 07:03:09.922826 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:09.922765 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-6ff85e-7c6f4dfd87-c5m8z"] Nov 02 07:03:09.929346 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:09.929327 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-6ff85e-7c6f4dfd87-c5m8z"] Nov 02 07:03:10.226081 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:10.225697 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-rvhhk"] Nov 02 07:03:10.229168 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:10.228274 5916 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-rvhhk" podUID="8d74d644-1bbf-4522-8075-43b1f476f656" containerName="istio-proxy" containerID="cri-o://1d0cc66708f23899d1f02f3b1561f95c902e0fda996233051880a0383f05bc04" gracePeriod=30 Nov 02 07:03:11.841331 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:11.841295 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee6a9440-ee89-4c29-aa2b-eb34c10935bf" path="/var/lib/kubelet/pods/ee6a9440-ee89-4c29-aa2b-eb34c10935bf/volumes" Nov 02 07:03:13.599425 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:13.599373 5916 patch_prober.go:29] interesting pod/e2e-worker-w6-6ff85e-openshift-default-698585659c-rvhhk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 07:03:13.599856 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:13.599449 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-rvhhk" podUID="8d74d644-1bbf-4522-8075-43b1f476f656" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 07:03:15.480559 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:15.480295 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-59fb8fd585-xgqcq"] Nov 02 07:03:15.480559 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:15.480544 5916 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-59fb8fd585-xgqcq" podUID="7c414f8b-a817-405d-aedf-6c570540d01a" containerName="manager" containerID="cri-o://ac0ab7db0e98409adc796e615d2492a81d94a2b8dd1158d2a2127bcb7a5a58cc" gracePeriod=30 Nov 02 07:03:15.707087 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:15.707050 5916 generic.go:356] "Generic (PLEG): container finished" podID="8d74d644-1bbf-4522-8075-43b1f476f656" containerID="1d0cc66708f23899d1f02f3b1561f95c902e0fda996233051880a0383f05bc04" exitCode=0 Nov 02 07:03:15.707272 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:15.707100 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-rvhhk" event={"ID":"8d74d644-1bbf-4522-8075-43b1f476f656","Type":"ContainerDied","Data":"1d0cc66708f23899d1f02f3b1561f95c902e0fda996233051880a0383f05bc04"} Nov 02 07:03:15.707272 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:15.707178 5916 scope.go:122] "RemoveContainer" containerID="86a483e176dbc71802a13babaadd925ebc4b9c64710630e13f7af6c9079394e2" Nov 02 07:03:15.709057 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:15.709036 5916 generic.go:356] "Generic (PLEG): container finished" podID="7c414f8b-a817-405d-aedf-6c570540d01a" containerID="ac0ab7db0e98409adc796e615d2492a81d94a2b8dd1158d2a2127bcb7a5a58cc" exitCode=0 Nov 02 07:03:15.709166 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:15.709066 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-59fb8fd585-xgqcq" event={"ID":"7c414f8b-a817-405d-aedf-6c570540d01a","Type":"ContainerDied","Data":"ac0ab7db0e98409adc796e615d2492a81d94a2b8dd1158d2a2127bcb7a5a58cc"} Nov 02 07:03:15.753217 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:15.753186 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-59fb8fd585-xgqcq" Nov 02 07:03:15.841822 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:15.841790 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c414f8b-a817-405d-aedf-6c570540d01a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c414f8b-a817-405d-aedf-6c570540d01a-metrics-certs\") pod \"7c414f8b-a817-405d-aedf-6c570540d01a\" (UID: \"7c414f8b-a817-405d-aedf-6c570540d01a\") " Nov 02 07:03:15.842215 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:15.841871 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c414f8b-a817-405d-aedf-6c570540d01a-kube-api-access-hx8hp\" (UniqueName: \"kubernetes.io/projected/7c414f8b-a817-405d-aedf-6c570540d01a-kube-api-access-hx8hp\") pod \"7c414f8b-a817-405d-aedf-6c570540d01a\" (UID: \"7c414f8b-a817-405d-aedf-6c570540d01a\") " Nov 02 07:03:15.842215 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:15.841928 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c414f8b-a817-405d-aedf-6c570540d01a-webhook-cert\" (UniqueName: \"kubernetes.io/secret/7c414f8b-a817-405d-aedf-6c570540d01a-webhook-cert\") pod \"7c414f8b-a817-405d-aedf-6c570540d01a\" (UID: \"7c414f8b-a817-405d-aedf-6c570540d01a\") " Nov 02 07:03:15.846900 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:15.846830 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c414f8b-a817-405d-aedf-6c570540d01a-metrics-certs" pod "7c414f8b-a817-405d-aedf-6c570540d01a" (UID: "7c414f8b-a817-405d-aedf-6c570540d01a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 07:03:15.847338 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:15.847145 5916 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c414f8b-a817-405d-aedf-6c570540d01a-metrics-certs\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:03:15.850933 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:15.849849 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c414f8b-a817-405d-aedf-6c570540d01a-webhook-cert" pod "7c414f8b-a817-405d-aedf-6c570540d01a" (UID: "7c414f8b-a817-405d-aedf-6c570540d01a"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 07:03:15.850933 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:15.849999 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c414f8b-a817-405d-aedf-6c570540d01a-kube-api-access-hx8hp" pod "7c414f8b-a817-405d-aedf-6c570540d01a" (UID: "7c414f8b-a817-405d-aedf-6c570540d01a"). InnerVolumeSpecName "kube-api-access-hx8hp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:03:15.879427 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:15.879402 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-rvhhk" Nov 02 07:03:15.948083 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:15.948048 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d74d644-1bbf-4522-8075-43b1f476f656-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d74d644-1bbf-4522-8075-43b1f476f656-istio-data\") pod \"8d74d644-1bbf-4522-8075-43b1f476f656\" (UID: \"8d74d644-1bbf-4522-8075-43b1f476f656\") " Nov 02 07:03:15.948248 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:15.948110 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d74d644-1bbf-4522-8075-43b1f476f656-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d74d644-1bbf-4522-8075-43b1f476f656-workload-socket\") pod \"8d74d644-1bbf-4522-8075-43b1f476f656\" (UID: \"8d74d644-1bbf-4522-8075-43b1f476f656\") " Nov 02 07:03:15.948248 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:15.948172 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d74d644-1bbf-4522-8075-43b1f476f656-kube-api-access-mlj2p\" (UniqueName: \"kubernetes.io/projected/8d74d644-1bbf-4522-8075-43b1f476f656-kube-api-access-mlj2p\") pod \"8d74d644-1bbf-4522-8075-43b1f476f656\" (UID: \"8d74d644-1bbf-4522-8075-43b1f476f656\") " Nov 02 07:03:15.948248 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:15.948211 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d74d644-1bbf-4522-8075-43b1f476f656-istio-token\" (UniqueName: \"kubernetes.io/projected/8d74d644-1bbf-4522-8075-43b1f476f656-istio-token\") pod \"8d74d644-1bbf-4522-8075-43b1f476f656\" (UID: \"8d74d644-1bbf-4522-8075-43b1f476f656\") " Nov 02 07:03:15.948248 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:15.948244 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d74d644-1bbf-4522-8075-43b1f476f656-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d74d644-1bbf-4522-8075-43b1f476f656-istiod-ca-cert\") pod \"8d74d644-1bbf-4522-8075-43b1f476f656\" (UID: \"8d74d644-1bbf-4522-8075-43b1f476f656\") " Nov 02 07:03:15.948530 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:15.948276 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d74d644-1bbf-4522-8075-43b1f476f656-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d74d644-1bbf-4522-8075-43b1f476f656-workload-certs\") pod \"8d74d644-1bbf-4522-8075-43b1f476f656\" (UID: \"8d74d644-1bbf-4522-8075-43b1f476f656\") " Nov 02 07:03:15.948530 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:15.948314 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d74d644-1bbf-4522-8075-43b1f476f656-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d74d644-1bbf-4522-8075-43b1f476f656-credential-socket\") pod \"8d74d644-1bbf-4522-8075-43b1f476f656\" (UID: \"8d74d644-1bbf-4522-8075-43b1f476f656\") " Nov 02 07:03:15.948530 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:15.948345 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d74d644-1bbf-4522-8075-43b1f476f656-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d74d644-1bbf-4522-8075-43b1f476f656-istio-envoy\") pod \"8d74d644-1bbf-4522-8075-43b1f476f656\" (UID: \"8d74d644-1bbf-4522-8075-43b1f476f656\") " Nov 02 07:03:15.948530 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:15.948354 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d74d644-1bbf-4522-8075-43b1f476f656-workload-socket" pod "8d74d644-1bbf-4522-8075-43b1f476f656" (UID: "8d74d644-1bbf-4522-8075-43b1f476f656"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:03:15.948530 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:15.948361 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d74d644-1bbf-4522-8075-43b1f476f656-istio-data" pod "8d74d644-1bbf-4522-8075-43b1f476f656" (UID: "8d74d644-1bbf-4522-8075-43b1f476f656"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:03:15.948530 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:15.948459 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8d74d644-1bbf-4522-8075-43b1f476f656-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d74d644-1bbf-4522-8075-43b1f476f656-istio-podinfo\") pod \"8d74d644-1bbf-4522-8075-43b1f476f656\" (UID: \"8d74d644-1bbf-4522-8075-43b1f476f656\") " Nov 02 07:03:15.948941 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:15.948588 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d74d644-1bbf-4522-8075-43b1f476f656-credential-socket" pod "8d74d644-1bbf-4522-8075-43b1f476f656" (UID: "8d74d644-1bbf-4522-8075-43b1f476f656"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:03:15.948941 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:15.948603 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d74d644-1bbf-4522-8075-43b1f476f656-workload-certs" pod "8d74d644-1bbf-4522-8075-43b1f476f656" (UID: "8d74d644-1bbf-4522-8075-43b1f476f656"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:03:15.948941 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:15.948886 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d74d644-1bbf-4522-8075-43b1f476f656-istiod-ca-cert" pod "8d74d644-1bbf-4522-8075-43b1f476f656" (UID: "8d74d644-1bbf-4522-8075-43b1f476f656"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:03:15.949453 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:15.949426 5916 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d74d644-1bbf-4522-8075-43b1f476f656-istiod-ca-cert\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:03:15.949453 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:15.949449 5916 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d74d644-1bbf-4522-8075-43b1f476f656-workload-certs\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:03:15.949633 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:15.949463 5916 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d74d644-1bbf-4522-8075-43b1f476f656-credential-socket\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:03:15.949633 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:15.949477 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hx8hp\" (UniqueName: \"kubernetes.io/projected/7c414f8b-a817-405d-aedf-6c570540d01a-kube-api-access-hx8hp\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:03:15.949633 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:15.949491 5916 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7c414f8b-a817-405d-aedf-6c570540d01a-webhook-cert\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:03:15.949633 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:15.949503 5916 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d74d644-1bbf-4522-8075-43b1f476f656-istio-data\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:03:15.949633 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:15.949516 5916 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d74d644-1bbf-4522-8075-43b1f476f656-workload-socket\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:03:15.951079 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:15.951050 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d74d644-1bbf-4522-8075-43b1f476f656-istio-envoy" pod "8d74d644-1bbf-4522-8075-43b1f476f656" (UID: "8d74d644-1bbf-4522-8075-43b1f476f656"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:03:15.951237 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:15.951212 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d74d644-1bbf-4522-8075-43b1f476f656-kube-api-access-mlj2p" pod "8d74d644-1bbf-4522-8075-43b1f476f656" (UID: "8d74d644-1bbf-4522-8075-43b1f476f656"). InnerVolumeSpecName "kube-api-access-mlj2p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:03:15.951477 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:15.951451 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8d74d644-1bbf-4522-8075-43b1f476f656-istio-podinfo" pod "8d74d644-1bbf-4522-8075-43b1f476f656" (UID: "8d74d644-1bbf-4522-8075-43b1f476f656"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 07:03:15.951640 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:15.951618 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d74d644-1bbf-4522-8075-43b1f476f656-istio-token" pod "8d74d644-1bbf-4522-8075-43b1f476f656" (UID: "8d74d644-1bbf-4522-8075-43b1f476f656"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:03:16.050470 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:16.050432 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mlj2p\" (UniqueName: \"kubernetes.io/projected/8d74d644-1bbf-4522-8075-43b1f476f656-kube-api-access-mlj2p\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:03:16.050470 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:16.050471 5916 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d74d644-1bbf-4522-8075-43b1f476f656-istio-token\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:03:16.050692 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:16.050485 5916 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d74d644-1bbf-4522-8075-43b1f476f656-istio-envoy\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:03:16.050692 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:16.050498 5916 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d74d644-1bbf-4522-8075-43b1f476f656-istio-podinfo\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:03:16.716948 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:16.716888 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-rvhhk" Nov 02 07:03:16.716948 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:16.716925 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-rvhhk" event={"ID":"8d74d644-1bbf-4522-8075-43b1f476f656","Type":"ContainerDied","Data":"cce13f840ee4dfcbc5871eb317067e269c8f9da3127842317467be21cf1a084d"} Nov 02 07:03:16.717541 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:16.716988 5916 scope.go:122] "RemoveContainer" containerID="1d0cc66708f23899d1f02f3b1561f95c902e0fda996233051880a0383f05bc04" Nov 02 07:03:16.718436 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:16.718406 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-59fb8fd585-xgqcq" event={"ID":"7c414f8b-a817-405d-aedf-6c570540d01a","Type":"ContainerDied","Data":"7183917691cb7747fa9902c61323b5ff3245f5661cd3e8d636d3d09eb8007281"} Nov 02 07:03:16.718524 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:16.718404 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-59fb8fd585-xgqcq" Nov 02 07:03:16.728281 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:16.728263 5916 scope.go:122] "RemoveContainer" containerID="ac0ab7db0e98409adc796e615d2492a81d94a2b8dd1158d2a2127bcb7a5a58cc" Nov 02 07:03:16.932348 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:16.932317 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-rvhhk"] Nov 02 07:03:16.983843 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:16.983756 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-rvhhk"] Nov 02 07:03:17.049538 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:17.049508 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-59fb8fd585-xgqcq"] Nov 02 07:03:17.052902 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:17.052876 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-59fb8fd585-xgqcq"] Nov 02 07:03:17.842774 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:17.842740 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c414f8b-a817-405d-aedf-6c570540d01a" path="/var/lib/kubelet/pods/7c414f8b-a817-405d-aedf-6c570540d01a/volumes" Nov 02 07:03:17.843330 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:17.843206 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d74d644-1bbf-4522-8075-43b1f476f656" path="/var/lib/kubelet/pods/8d74d644-1bbf-4522-8075-43b1f476f656/volumes" Nov 02 07:03:19.509096 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:19.509058 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-59fb8fd585-9s9g9"] Nov 02 07:03:19.509577 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:19.509563 5916 cpu_manager.go:414] "containerMap: removing container" podUID="7c414f8b-a817-405d-aedf-6c570540d01a" containerName="manager" Nov 02 07:03:19.509617 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:19.509579 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c414f8b-a817-405d-aedf-6c570540d01a" containerName="manager" Nov 02 07:03:19.509617 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:19.509594 5916 cpu_manager.go:414] "containerMap: removing container" podUID="30114285-d730-4abb-a618-6833d5746feb" containerName="main" Nov 02 07:03:19.509617 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:19.509602 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30114285-d730-4abb-a618-6833d5746feb" containerName="main" Nov 02 07:03:19.509736 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:19.509617 5916 cpu_manager.go:414] "containerMap: removing container" podUID="ee6a9440-ee89-4c29-aa2b-eb34c10935bf" containerName="payload-processing" Nov 02 07:03:19.509736 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:19.509624 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee6a9440-ee89-4c29-aa2b-eb34c10935bf" containerName="payload-processing" Nov 02 07:03:19.509736 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:19.509635 5916 cpu_manager.go:414] "containerMap: removing container" podUID="db3677b9-3aa0-415f-a6a0-e7e87e2aad7f" containerName="revoke-keys" Nov 02 07:03:19.509736 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:19.509642 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db3677b9-3aa0-415f-a6a0-e7e87e2aad7f" containerName="revoke-keys" Nov 02 07:03:19.509736 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:19.509655 5916 cpu_manager.go:414] "containerMap: removing container" podUID="8d74d644-1bbf-4522-8075-43b1f476f656" containerName="istio-proxy" Nov 02 07:03:19.509736 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:19.509662 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d74d644-1bbf-4522-8075-43b1f476f656" containerName="istio-proxy" Nov 02 07:03:19.509736 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:19.509701 5916 cpu_manager.go:414] "containerMap: removing container" podUID="8d74d644-1bbf-4522-8075-43b1f476f656" containerName="istio-proxy" Nov 02 07:03:19.509736 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:19.509708 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d74d644-1bbf-4522-8075-43b1f476f656" containerName="istio-proxy" Nov 02 07:03:19.509736 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:19.509718 5916 cpu_manager.go:414] "containerMap: removing container" podUID="771371d6-10a2-4679-8a9e-3ca2e695f4db" containerName="payload-pre-processing" Nov 02 07:03:19.509736 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:19.509725 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="771371d6-10a2-4679-8a9e-3ca2e695f4db" containerName="payload-pre-processing" Nov 02 07:03:19.509736 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:19.509734 5916 cpu_manager.go:414] "containerMap: removing container" podUID="eb386976-f898-4f1d-b27c-77bccc735a3b" containerName="main" Nov 02 07:03:19.510105 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:19.509742 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb386976-f898-4f1d-b27c-77bccc735a3b" containerName="main" Nov 02 07:03:19.510105 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:19.509837 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c414f8b-a817-405d-aedf-6c570540d01a" containerName="manager" Nov 02 07:03:19.510105 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:19.509849 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="30114285-d730-4abb-a618-6833d5746feb" containerName="main" Nov 02 07:03:19.510105 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:19.509858 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee6a9440-ee89-4c29-aa2b-eb34c10935bf" containerName="payload-processing" Nov 02 07:03:19.510105 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:19.509866 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="db3677b9-3aa0-415f-a6a0-e7e87e2aad7f" containerName="revoke-keys" Nov 02 07:03:19.510105 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:19.509874 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d74d644-1bbf-4522-8075-43b1f476f656" containerName="istio-proxy" Nov 02 07:03:19.510105 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:19.509887 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="771371d6-10a2-4679-8a9e-3ca2e695f4db" containerName="payload-pre-processing" Nov 02 07:03:19.510105 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:19.509894 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb386976-f898-4f1d-b27c-77bccc735a3b" containerName="main" Nov 02 07:03:19.513359 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:19.513342 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-59fb8fd585-9s9g9" Nov 02 07:03:19.583230 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:19.583202 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24a75a7e-78f3-421a-ab7e-77fdb5a1279d-metrics-certs\") pod \"maas-controller-59fb8fd585-9s9g9\" (UID: \"24a75a7e-78f3-421a-ab7e-77fdb5a1279d\") " pod="opendatahub/maas-controller-59fb8fd585-9s9g9" Nov 02 07:03:19.583365 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:19.583250 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/24a75a7e-78f3-421a-ab7e-77fdb5a1279d-webhook-cert\") pod \"maas-controller-59fb8fd585-9s9g9\" (UID: \"24a75a7e-78f3-421a-ab7e-77fdb5a1279d\") " pod="opendatahub/maas-controller-59fb8fd585-9s9g9" Nov 02 07:03:19.583365 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:19.583268 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xd7v4\" (UniqueName: \"kubernetes.io/projected/24a75a7e-78f3-421a-ab7e-77fdb5a1279d-kube-api-access-xd7v4\") pod \"maas-controller-59fb8fd585-9s9g9\" (UID: \"24a75a7e-78f3-421a-ab7e-77fdb5a1279d\") " pod="opendatahub/maas-controller-59fb8fd585-9s9g9" Nov 02 07:03:19.615495 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:19.615462 5916 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 02 07:03:19.615495 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:19.615485 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-prjsd\"" Nov 02 07:03:19.615699 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:19.615555 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 02 07:03:19.615763 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:19.615714 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 02 07:03:19.617965 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:19.617947 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-59fb8fd585-9s9g9"] Nov 02 07:03:19.684241 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:19.684217 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24a75a7e-78f3-421a-ab7e-77fdb5a1279d-metrics-certs\") pod \"maas-controller-59fb8fd585-9s9g9\" (UID: \"24a75a7e-78f3-421a-ab7e-77fdb5a1279d\") " pod="opendatahub/maas-controller-59fb8fd585-9s9g9" Nov 02 07:03:19.684357 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:19.684265 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/24a75a7e-78f3-421a-ab7e-77fdb5a1279d-webhook-cert\") pod \"maas-controller-59fb8fd585-9s9g9\" (UID: \"24a75a7e-78f3-421a-ab7e-77fdb5a1279d\") " pod="opendatahub/maas-controller-59fb8fd585-9s9g9" Nov 02 07:03:19.684357 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:19.684284 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xd7v4\" (UniqueName: \"kubernetes.io/projected/24a75a7e-78f3-421a-ab7e-77fdb5a1279d-kube-api-access-xd7v4\") pod \"maas-controller-59fb8fd585-9s9g9\" (UID: \"24a75a7e-78f3-421a-ab7e-77fdb5a1279d\") " pod="opendatahub/maas-controller-59fb8fd585-9s9g9" Nov 02 07:03:19.686592 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:19.686572 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/24a75a7e-78f3-421a-ab7e-77fdb5a1279d-webhook-cert\") pod \"maas-controller-59fb8fd585-9s9g9\" (UID: \"24a75a7e-78f3-421a-ab7e-77fdb5a1279d\") " pod="opendatahub/maas-controller-59fb8fd585-9s9g9" Nov 02 07:03:19.686719 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:19.686661 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24a75a7e-78f3-421a-ab7e-77fdb5a1279d-metrics-certs\") pod \"maas-controller-59fb8fd585-9s9g9\" (UID: \"24a75a7e-78f3-421a-ab7e-77fdb5a1279d\") " pod="opendatahub/maas-controller-59fb8fd585-9s9g9" Nov 02 07:03:19.741061 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:19.741034 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xd7v4\" (UniqueName: \"kubernetes.io/projected/24a75a7e-78f3-421a-ab7e-77fdb5a1279d-kube-api-access-xd7v4\") pod \"maas-controller-59fb8fd585-9s9g9\" (UID: \"24a75a7e-78f3-421a-ab7e-77fdb5a1279d\") " pod="opendatahub/maas-controller-59fb8fd585-9s9g9" Nov 02 07:03:19.826478 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:19.826396 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-59fb8fd585-9s9g9" Nov 02 07:03:19.900184 ip-10-0-130-93 kubenswrapper[5916]: E1102 07:03:19.900150 5916 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p98cq" containerName="extractor" Nov 02 07:03:20.201018 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:20.200992 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-59fb8fd585-9s9g9"] Nov 02 07:03:20.202999 ip-10-0-130-93 kubenswrapper[5916]: W1102 07:03:20.202969 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24a75a7e_78f3_421a_ab7e_77fdb5a1279d.slice/crio-121746f9ab5736bccf4ac5079323b8008a1c4b8fb7ef3732e1826e073c1cbd15 WatchSource:0}: Error finding container 121746f9ab5736bccf4ac5079323b8008a1c4b8fb7ef3732e1826e073c1cbd15: Status 404 returned error can't find the container with id 121746f9ab5736bccf4ac5079323b8008a1c4b8fb7ef3732e1826e073c1cbd15 Nov 02 07:03:20.204217 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:20.204201 5916 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 07:03:20.749596 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:20.749562 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-59fb8fd585-9s9g9" event={"ID":"24a75a7e-78f3-421a-ab7e-77fdb5a1279d","Type":"ContainerStarted","Data":"121746f9ab5736bccf4ac5079323b8008a1c4b8fb7ef3732e1826e073c1cbd15"} Nov 02 07:03:21.757062 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:21.757029 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-59fb8fd585-9s9g9" event={"ID":"24a75a7e-78f3-421a-ab7e-77fdb5a1279d","Type":"ContainerStarted","Data":"24da47a175b4efab88fab4d4a4284c4cab0ca8f7548d16880160c554992e3f2a"} Nov 02 07:03:21.757473 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:21.757101 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-59fb8fd585-9s9g9" Nov 02 07:03:21.882305 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:21.881234 5916 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-59fb8fd585-9s9g9" podStartSLOduration=1.956417177 podStartE2EDuration="2.881218019s" totalImagesPullingTime="924.800842ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:03:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:03:21.879213483 +0000 UTC m=+2790.151033877" watchObservedRunningTime="2026-11-02 07:03:21.881218019 +0000 UTC m=+2790.153038414" Nov 02 07:03:30.979969 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:30.979876 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tnptl"] Nov 02 07:03:30.980556 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:30.980535 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d74d644-1bbf-4522-8075-43b1f476f656" containerName="istio-proxy" Nov 02 07:03:30.983871 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:30.983850 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tnptl" Nov 02 07:03:31.029648 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:31.029623 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6527w"] Nov 02 07:03:31.033176 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:31.033162 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6527w" Nov 02 07:03:31.040828 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:31.040807 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-hptbf\"" Nov 02 07:03:31.074526 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:31.074500 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a24071b2-e468-4590-93ea-1be6ac017cf8-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-tnptl\" (UID: \"a24071b2-e468-4590-93ea-1be6ac017cf8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tnptl" Nov 02 07:03:31.074646 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:31.074539 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45rqf\" (UniqueName: \"kubernetes.io/projected/a24071b2-e468-4590-93ea-1be6ac017cf8-kube-api-access-45rqf\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-tnptl\" (UID: \"a24071b2-e468-4590-93ea-1be6ac017cf8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tnptl" Nov 02 07:03:31.074720 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:31.074659 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a24071b2-e468-4590-93ea-1be6ac017cf8-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-tnptl\" (UID: \"a24071b2-e468-4590-93ea-1be6ac017cf8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tnptl" Nov 02 07:03:31.074720 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:31.074693 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a24071b2-e468-4590-93ea-1be6ac017cf8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-tnptl\" (UID: \"a24071b2-e468-4590-93ea-1be6ac017cf8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tnptl" Nov 02 07:03:31.074816 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:31.074799 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a24071b2-e468-4590-93ea-1be6ac017cf8-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-tnptl\" (UID: \"a24071b2-e468-4590-93ea-1be6ac017cf8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tnptl" Nov 02 07:03:31.074859 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:31.074830 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a24071b2-e468-4590-93ea-1be6ac017cf8-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-tnptl\" (UID: \"a24071b2-e468-4590-93ea-1be6ac017cf8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tnptl" Nov 02 07:03:31.074944 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:31.074891 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a24071b2-e468-4590-93ea-1be6ac017cf8-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-tnptl\" (UID: \"a24071b2-e468-4590-93ea-1be6ac017cf8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tnptl" Nov 02 07:03:31.074998 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:31.074944 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a24071b2-e468-4590-93ea-1be6ac017cf8-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-tnptl\" (UID: \"a24071b2-e468-4590-93ea-1be6ac017cf8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tnptl" Nov 02 07:03:31.075081 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:31.075060 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a24071b2-e468-4590-93ea-1be6ac017cf8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-tnptl\" (UID: \"a24071b2-e468-4590-93ea-1be6ac017cf8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tnptl" Nov 02 07:03:31.081518 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:31.081497 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tnptl"] Nov 02 07:03:31.124190 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:31.124163 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6527w"] Nov 02 07:03:31.176255 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:31.176226 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/918ad956-7d56-4627-a1a4-e7c75694d1f9-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-6527w\" (UID: \"918ad956-7d56-4627-a1a4-e7c75694d1f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6527w" Nov 02 07:03:31.176376 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:31.176258 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/918ad956-7d56-4627-a1a4-e7c75694d1f9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-6527w\" (UID: \"918ad956-7d56-4627-a1a4-e7c75694d1f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6527w" Nov 02 07:03:31.176376 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:31.176280 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a24071b2-e468-4590-93ea-1be6ac017cf8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-tnptl\" (UID: \"a24071b2-e468-4590-93ea-1be6ac017cf8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tnptl" Nov 02 07:03:31.176452 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:31.176391 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/918ad956-7d56-4627-a1a4-e7c75694d1f9-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-6527w\" (UID: \"918ad956-7d56-4627-a1a4-e7c75694d1f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6527w" Nov 02 07:03:31.176452 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:31.176418 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/918ad956-7d56-4627-a1a4-e7c75694d1f9-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-6527w\" (UID: \"918ad956-7d56-4627-a1a4-e7c75694d1f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6527w" Nov 02 07:03:31.176534 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:31.176449 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a24071b2-e468-4590-93ea-1be6ac017cf8-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-tnptl\" (UID: \"a24071b2-e468-4590-93ea-1be6ac017cf8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tnptl" Nov 02 07:03:31.176647 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:31.176625 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45rqf\" (UniqueName: \"kubernetes.io/projected/a24071b2-e468-4590-93ea-1be6ac017cf8-kube-api-access-45rqf\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-tnptl\" (UID: \"a24071b2-e468-4590-93ea-1be6ac017cf8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tnptl" Nov 02 07:03:31.176719 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:31.176665 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a24071b2-e468-4590-93ea-1be6ac017cf8-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-tnptl\" (UID: \"a24071b2-e468-4590-93ea-1be6ac017cf8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tnptl" Nov 02 07:03:31.176719 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:31.176696 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a24071b2-e468-4590-93ea-1be6ac017cf8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-tnptl\" (UID: \"a24071b2-e468-4590-93ea-1be6ac017cf8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tnptl" Nov 02 07:03:31.176830 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:31.176726 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/918ad956-7d56-4627-a1a4-e7c75694d1f9-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-6527w\" (UID: \"918ad956-7d56-4627-a1a4-e7c75694d1f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6527w" Nov 02 07:03:31.176830 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:31.176745 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a24071b2-e468-4590-93ea-1be6ac017cf8-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-tnptl\" (UID: \"a24071b2-e468-4590-93ea-1be6ac017cf8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tnptl" Nov 02 07:03:31.176830 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:31.176777 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a24071b2-e468-4590-93ea-1be6ac017cf8-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-tnptl\" (UID: \"a24071b2-e468-4590-93ea-1be6ac017cf8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tnptl" Nov 02 07:03:31.177011 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:31.176948 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxnr8\" (UniqueName: \"kubernetes.io/projected/918ad956-7d56-4627-a1a4-e7c75694d1f9-kube-api-access-kxnr8\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-6527w\" (UID: \"918ad956-7d56-4627-a1a4-e7c75694d1f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6527w" Nov 02 07:03:31.177011 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:31.176983 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a24071b2-e468-4590-93ea-1be6ac017cf8-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-tnptl\" (UID: \"a24071b2-e468-4590-93ea-1be6ac017cf8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tnptl" Nov 02 07:03:31.177103 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:31.177051 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/918ad956-7d56-4627-a1a4-e7c75694d1f9-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-6527w\" (UID: \"918ad956-7d56-4627-a1a4-e7c75694d1f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6527w" Nov 02 07:03:31.177103 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:31.177088 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/918ad956-7d56-4627-a1a4-e7c75694d1f9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-6527w\" (UID: \"918ad956-7d56-4627-a1a4-e7c75694d1f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6527w" Nov 02 07:03:31.177212 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:31.177115 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a24071b2-e468-4590-93ea-1be6ac017cf8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-tnptl\" (UID: \"a24071b2-e468-4590-93ea-1be6ac017cf8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tnptl" Nov 02 07:03:31.177212 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:31.177130 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a24071b2-e468-4590-93ea-1be6ac017cf8-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-tnptl\" (UID: \"a24071b2-e468-4590-93ea-1be6ac017cf8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tnptl" Nov 02 07:03:31.177212 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:31.177135 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a24071b2-e468-4590-93ea-1be6ac017cf8-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-tnptl\" (UID: \"a24071b2-e468-4590-93ea-1be6ac017cf8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tnptl" Nov 02 07:03:31.177354 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:31.177245 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/918ad956-7d56-4627-a1a4-e7c75694d1f9-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-6527w\" (UID: \"918ad956-7d56-4627-a1a4-e7c75694d1f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6527w" Nov 02 07:03:31.177354 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:31.177272 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a24071b2-e468-4590-93ea-1be6ac017cf8-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-tnptl\" (UID: \"a24071b2-e468-4590-93ea-1be6ac017cf8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tnptl" Nov 02 07:03:31.177460 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:31.177436 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a24071b2-e468-4590-93ea-1be6ac017cf8-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-tnptl\" (UID: \"a24071b2-e468-4590-93ea-1be6ac017cf8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tnptl" Nov 02 07:03:31.177524 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:31.177506 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a24071b2-e468-4590-93ea-1be6ac017cf8-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-tnptl\" (UID: \"a24071b2-e468-4590-93ea-1be6ac017cf8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tnptl" Nov 02 07:03:31.179177 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:31.179156 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a24071b2-e468-4590-93ea-1be6ac017cf8-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-tnptl\" (UID: \"a24071b2-e468-4590-93ea-1be6ac017cf8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tnptl" Nov 02 07:03:31.179177 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:31.179167 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a24071b2-e468-4590-93ea-1be6ac017cf8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-tnptl\" (UID: \"a24071b2-e468-4590-93ea-1be6ac017cf8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tnptl" Nov 02 07:03:31.236068 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:31.236007 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a24071b2-e468-4590-93ea-1be6ac017cf8-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-tnptl\" (UID: \"a24071b2-e468-4590-93ea-1be6ac017cf8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tnptl" Nov 02 07:03:31.236699 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:31.236678 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45rqf\" (UniqueName: \"kubernetes.io/projected/a24071b2-e468-4590-93ea-1be6ac017cf8-kube-api-access-45rqf\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-tnptl\" (UID: \"a24071b2-e468-4590-93ea-1be6ac017cf8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tnptl" Nov 02 07:03:31.277964 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:31.277926 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/918ad956-7d56-4627-a1a4-e7c75694d1f9-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-6527w\" (UID: \"918ad956-7d56-4627-a1a4-e7c75694d1f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6527w" Nov 02 07:03:31.278104 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:31.277994 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxnr8\" (UniqueName: \"kubernetes.io/projected/918ad956-7d56-4627-a1a4-e7c75694d1f9-kube-api-access-kxnr8\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-6527w\" (UID: \"918ad956-7d56-4627-a1a4-e7c75694d1f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6527w" Nov 02 07:03:31.278104 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:31.278018 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/918ad956-7d56-4627-a1a4-e7c75694d1f9-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-6527w\" (UID: \"918ad956-7d56-4627-a1a4-e7c75694d1f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6527w" Nov 02 07:03:31.278104 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:31.278033 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/918ad956-7d56-4627-a1a4-e7c75694d1f9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-6527w\" (UID: \"918ad956-7d56-4627-a1a4-e7c75694d1f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6527w" Nov 02 07:03:31.278104 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:31.278076 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/918ad956-7d56-4627-a1a4-e7c75694d1f9-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-6527w\" (UID: \"918ad956-7d56-4627-a1a4-e7c75694d1f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6527w" Nov 02 07:03:31.278396 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:31.278124 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/918ad956-7d56-4627-a1a4-e7c75694d1f9-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-6527w\" (UID: \"918ad956-7d56-4627-a1a4-e7c75694d1f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6527w" Nov 02 07:03:31.278396 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:31.278152 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/918ad956-7d56-4627-a1a4-e7c75694d1f9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-6527w\" (UID: \"918ad956-7d56-4627-a1a4-e7c75694d1f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6527w" Nov 02 07:03:31.278396 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:31.278202 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/918ad956-7d56-4627-a1a4-e7c75694d1f9-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-6527w\" (UID: \"918ad956-7d56-4627-a1a4-e7c75694d1f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6527w" Nov 02 07:03:31.278396 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:31.278229 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/918ad956-7d56-4627-a1a4-e7c75694d1f9-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-6527w\" (UID: \"918ad956-7d56-4627-a1a4-e7c75694d1f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6527w" Nov 02 07:03:31.278624 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:31.278484 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/918ad956-7d56-4627-a1a4-e7c75694d1f9-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-6527w\" (UID: \"918ad956-7d56-4627-a1a4-e7c75694d1f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6527w" Nov 02 07:03:31.278624 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:31.278556 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/918ad956-7d56-4627-a1a4-e7c75694d1f9-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-6527w\" (UID: \"918ad956-7d56-4627-a1a4-e7c75694d1f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6527w" Nov 02 07:03:31.278734 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:31.278640 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/918ad956-7d56-4627-a1a4-e7c75694d1f9-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-6527w\" (UID: \"918ad956-7d56-4627-a1a4-e7c75694d1f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6527w" Nov 02 07:03:31.278734 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:31.278670 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/918ad956-7d56-4627-a1a4-e7c75694d1f9-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-6527w\" (UID: \"918ad956-7d56-4627-a1a4-e7c75694d1f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6527w" Nov 02 07:03:31.278825 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:31.278763 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/918ad956-7d56-4627-a1a4-e7c75694d1f9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-6527w\" (UID: \"918ad956-7d56-4627-a1a4-e7c75694d1f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6527w" Nov 02 07:03:31.280476 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:31.280453 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/918ad956-7d56-4627-a1a4-e7c75694d1f9-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-6527w\" (UID: \"918ad956-7d56-4627-a1a4-e7c75694d1f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6527w" Nov 02 07:03:31.280682 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:31.280665 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/918ad956-7d56-4627-a1a4-e7c75694d1f9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-6527w\" (UID: \"918ad956-7d56-4627-a1a4-e7c75694d1f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6527w" Nov 02 07:03:31.296719 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:31.296695 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tnptl" Nov 02 07:03:31.334623 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:31.334590 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/918ad956-7d56-4627-a1a4-e7c75694d1f9-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-6527w\" (UID: \"918ad956-7d56-4627-a1a4-e7c75694d1f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6527w" Nov 02 07:03:31.334887 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:31.334866 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxnr8\" (UniqueName: \"kubernetes.io/projected/918ad956-7d56-4627-a1a4-e7c75694d1f9-kube-api-access-kxnr8\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-6527w\" (UID: \"918ad956-7d56-4627-a1a4-e7c75694d1f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6527w" Nov 02 07:03:31.345529 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:31.345504 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6527w" Nov 02 07:03:31.724510 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:31.724480 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tnptl"] Nov 02 07:03:31.764749 ip-10-0-130-93 kubenswrapper[5916]: W1102 07:03:31.764716 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda24071b2_e468_4590_93ea_1be6ac017cf8.slice/crio-4ff538d74062dc7d622fa6ec28e01234bc1c79337e32d3b069059539ac6c1ea4 WatchSource:0}: Error finding container 4ff538d74062dc7d622fa6ec28e01234bc1c79337e32d3b069059539ac6c1ea4: Status 404 returned error can't find the container with id 4ff538d74062dc7d622fa6ec28e01234bc1c79337e32d3b069059539ac6c1ea4 Nov 02 07:03:31.766839 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:31.766800 5916 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 07:03:31.766937 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:31.766870 5916 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 07:03:31.766991 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:31.766897 5916 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 07:03:31.778885 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:31.778862 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6527w"] Nov 02 07:03:31.782591 ip-10-0-130-93 kubenswrapper[5916]: W1102 07:03:31.782570 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod918ad956_7d56_4627_a1a4_e7c75694d1f9.slice/crio-4b9ddeb41120e1504d9f3a8d5bf9f5fb46f5537c2b133cfb76ae854612bdf0af WatchSource:0}: Error finding container 4b9ddeb41120e1504d9f3a8d5bf9f5fb46f5537c2b133cfb76ae854612bdf0af: Status 404 returned error can't find the container with id 4b9ddeb41120e1504d9f3a8d5bf9f5fb46f5537c2b133cfb76ae854612bdf0af Nov 02 07:03:31.784572 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:31.784545 5916 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 07:03:31.784687 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:31.784600 5916 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 07:03:31.784687 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:31.784629 5916 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 07:03:31.816774 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:31.816746 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tnptl" event={"ID":"a24071b2-e468-4590-93ea-1be6ac017cf8","Type":"ContainerStarted","Data":"4ff538d74062dc7d622fa6ec28e01234bc1c79337e32d3b069059539ac6c1ea4"} Nov 02 07:03:31.818227 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:31.818201 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6527w" event={"ID":"918ad956-7d56-4627-a1a4-e7c75694d1f9","Type":"ContainerStarted","Data":"4b9ddeb41120e1504d9f3a8d5bf9f5fb46f5537c2b133cfb76ae854612bdf0af"} Nov 02 07:03:32.772929 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:32.772180 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-59fb8fd585-9s9g9" Nov 02 07:03:32.844928 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:32.839012 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6527w" event={"ID":"918ad956-7d56-4627-a1a4-e7c75694d1f9","Type":"ContainerStarted","Data":"c6d2bf0c34a0fdf453206e75ee9c422c52910c1e4151983816bdd20729c20022"} Nov 02 07:03:32.844928 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:32.844534 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tnptl" event={"ID":"a24071b2-e468-4590-93ea-1be6ac017cf8","Type":"ContainerStarted","Data":"a7042da8a8463360228ce5be700f8dce2c8336782ced747263b625645f613e2e"} Nov 02 07:03:33.078989 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:33.078417 5916 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tnptl" podStartSLOduration=3.078402409 podStartE2EDuration="3.078402409s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:03:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:03:33.077460007 +0000 UTC m=+2801.349280401" watchObservedRunningTime="2026-11-02 07:03:33.078402409 +0000 UTC m=+2801.350222802" Nov 02 07:03:33.201843 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:33.201783 5916 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6527w" podStartSLOduration=3.201770226 podStartE2EDuration="3.201770226s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:03:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:03:33.199569941 +0000 UTC m=+2801.471390335" watchObservedRunningTime="2026-11-02 07:03:33.201770226 +0000 UTC m=+2801.473590619" Nov 02 07:03:33.297157 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:33.297124 5916 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tnptl" Nov 02 07:03:33.298294 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:33.298273 5916 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-tnptl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body= Nov 02 07:03:33.298359 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:33.298317 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tnptl" podUID="a24071b2-e468-4590-93ea-1be6ac017cf8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" Nov 02 07:03:33.354259 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:33.352256 5916 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6527w" Nov 02 07:03:33.373967 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:33.366390 5916 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-6527w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" start-of-body= Nov 02 07:03:33.373967 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:33.366448 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6527w" podUID="918ad956-7d56-4627-a1a4-e7c75694d1f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" Nov 02 07:03:34.297574 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:34.297538 5916 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-tnptl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body= Nov 02 07:03:34.298097 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:34.297588 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tnptl" podUID="a24071b2-e468-4590-93ea-1be6ac017cf8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" Nov 02 07:03:34.346927 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:34.346500 5916 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-6527w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" start-of-body= Nov 02 07:03:34.346927 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:34.346574 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6527w" podUID="918ad956-7d56-4627-a1a4-e7c75694d1f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" Nov 02 07:03:35.297743 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:35.297703 5916 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-tnptl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body= Nov 02 07:03:35.298274 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:35.297771 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tnptl" podUID="a24071b2-e468-4590-93ea-1be6ac017cf8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" Nov 02 07:03:35.346926 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:35.346292 5916 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-6527w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" start-of-body= Nov 02 07:03:35.346926 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:35.346345 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6527w" podUID="918ad956-7d56-4627-a1a4-e7c75694d1f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" Nov 02 07:03:36.297855 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:36.297819 5916 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-tnptl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body= Nov 02 07:03:36.298331 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:36.297868 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tnptl" podUID="a24071b2-e468-4590-93ea-1be6ac017cf8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" Nov 02 07:03:36.346929 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:36.346305 5916 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-6527w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" start-of-body= Nov 02 07:03:36.346929 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:36.346369 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6527w" podUID="918ad956-7d56-4627-a1a4-e7c75694d1f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" Nov 02 07:03:37.297665 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:37.297628 5916 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-tnptl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body= Nov 02 07:03:37.297852 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:37.297678 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tnptl" podUID="a24071b2-e468-4590-93ea-1be6ac017cf8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" Nov 02 07:03:37.346927 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:37.346248 5916 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-6527w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" start-of-body= Nov 02 07:03:37.346927 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:37.346303 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6527w" podUID="918ad956-7d56-4627-a1a4-e7c75694d1f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" Nov 02 07:03:38.297071 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:38.297035 5916 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-tnptl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body= Nov 02 07:03:38.297291 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:38.297100 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tnptl" podUID="a24071b2-e468-4590-93ea-1be6ac017cf8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" Nov 02 07:03:38.346637 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:38.346595 5916 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-6527w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" start-of-body= Nov 02 07:03:38.346793 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:38.346666 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6527w" podUID="918ad956-7d56-4627-a1a4-e7c75694d1f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" Nov 02 07:03:39.939759 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:39.939705 5916 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6527w" Nov 02 07:03:39.940316 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:39.940148 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6527w" Nov 02 07:03:39.940834 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:39.940810 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6527w" Nov 02 07:03:39.988417 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:39.988384 5916 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tnptl" Nov 02 07:03:39.988800 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:39.988778 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tnptl" Nov 02 07:03:39.989455 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:39.989439 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tnptl" Nov 02 07:03:46.106388 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:46.106347 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-px5bb"] Nov 02 07:03:46.113050 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:46.113020 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-px5bb" Nov 02 07:03:46.219988 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:46.219952 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-px5bb"] Nov 02 07:03:46.254467 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:46.254429 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6566515d-9e15-4679-b56f-87c483940ba6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-px5bb\" (UID: \"6566515d-9e15-4679-b56f-87c483940ba6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-px5bb" Nov 02 07:03:46.254765 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:46.254739 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6566515d-9e15-4679-b56f-87c483940ba6-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-px5bb\" (UID: \"6566515d-9e15-4679-b56f-87c483940ba6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-px5bb" Nov 02 07:03:46.254955 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:46.254932 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6566515d-9e15-4679-b56f-87c483940ba6-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-px5bb\" (UID: \"6566515d-9e15-4679-b56f-87c483940ba6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-px5bb" Nov 02 07:03:46.255087 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:46.255070 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6566515d-9e15-4679-b56f-87c483940ba6-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-px5bb\" (UID: \"6566515d-9e15-4679-b56f-87c483940ba6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-px5bb" Nov 02 07:03:46.255239 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:46.255223 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6566515d-9e15-4679-b56f-87c483940ba6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-px5bb\" (UID: \"6566515d-9e15-4679-b56f-87c483940ba6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-px5bb" Nov 02 07:03:46.255370 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:46.255354 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6566515d-9e15-4679-b56f-87c483940ba6-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-px5bb\" (UID: \"6566515d-9e15-4679-b56f-87c483940ba6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-px5bb" Nov 02 07:03:46.255500 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:46.255475 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6566515d-9e15-4679-b56f-87c483940ba6-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-px5bb\" (UID: \"6566515d-9e15-4679-b56f-87c483940ba6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-px5bb" Nov 02 07:03:46.255634 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:46.255526 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5fcq\" (UniqueName: \"kubernetes.io/projected/6566515d-9e15-4679-b56f-87c483940ba6-kube-api-access-n5fcq\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-px5bb\" (UID: \"6566515d-9e15-4679-b56f-87c483940ba6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-px5bb" Nov 02 07:03:46.255634 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:46.255554 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6566515d-9e15-4679-b56f-87c483940ba6-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-px5bb\" (UID: \"6566515d-9e15-4679-b56f-87c483940ba6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-px5bb" Nov 02 07:03:46.356882 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:46.356767 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6566515d-9e15-4679-b56f-87c483940ba6-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-px5bb\" (UID: \"6566515d-9e15-4679-b56f-87c483940ba6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-px5bb" Nov 02 07:03:46.356882 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:46.356839 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5fcq\" (UniqueName: \"kubernetes.io/projected/6566515d-9e15-4679-b56f-87c483940ba6-kube-api-access-n5fcq\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-px5bb\" (UID: \"6566515d-9e15-4679-b56f-87c483940ba6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-px5bb" Nov 02 07:03:46.356882 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:46.356867 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6566515d-9e15-4679-b56f-87c483940ba6-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-px5bb\" (UID: \"6566515d-9e15-4679-b56f-87c483940ba6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-px5bb" Nov 02 07:03:46.357228 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:46.356953 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6566515d-9e15-4679-b56f-87c483940ba6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-px5bb\" (UID: \"6566515d-9e15-4679-b56f-87c483940ba6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-px5bb" Nov 02 07:03:46.357228 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:46.356999 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6566515d-9e15-4679-b56f-87c483940ba6-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-px5bb\" (UID: \"6566515d-9e15-4679-b56f-87c483940ba6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-px5bb" Nov 02 07:03:46.357228 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:46.357037 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6566515d-9e15-4679-b56f-87c483940ba6-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-px5bb\" (UID: \"6566515d-9e15-4679-b56f-87c483940ba6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-px5bb" Nov 02 07:03:46.357228 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:46.357066 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6566515d-9e15-4679-b56f-87c483940ba6-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-px5bb\" (UID: \"6566515d-9e15-4679-b56f-87c483940ba6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-px5bb" Nov 02 07:03:46.357228 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:46.357115 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6566515d-9e15-4679-b56f-87c483940ba6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-px5bb\" (UID: \"6566515d-9e15-4679-b56f-87c483940ba6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-px5bb" Nov 02 07:03:46.357228 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:46.357147 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6566515d-9e15-4679-b56f-87c483940ba6-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-px5bb\" (UID: \"6566515d-9e15-4679-b56f-87c483940ba6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-px5bb" Nov 02 07:03:46.357639 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:46.357610 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6566515d-9e15-4679-b56f-87c483940ba6-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-px5bb\" (UID: \"6566515d-9e15-4679-b56f-87c483940ba6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-px5bb" Nov 02 07:03:46.357943 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:46.357892 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6566515d-9e15-4679-b56f-87c483940ba6-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-px5bb\" (UID: \"6566515d-9e15-4679-b56f-87c483940ba6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-px5bb" Nov 02 07:03:46.358207 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:46.358183 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6566515d-9e15-4679-b56f-87c483940ba6-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-px5bb\" (UID: \"6566515d-9e15-4679-b56f-87c483940ba6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-px5bb" Nov 02 07:03:46.358451 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:46.358428 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6566515d-9e15-4679-b56f-87c483940ba6-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-px5bb\" (UID: \"6566515d-9e15-4679-b56f-87c483940ba6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-px5bb" Nov 02 07:03:46.359855 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:46.359831 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6566515d-9e15-4679-b56f-87c483940ba6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-px5bb\" (UID: \"6566515d-9e15-4679-b56f-87c483940ba6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-px5bb" Nov 02 07:03:46.363128 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:46.363082 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6566515d-9e15-4679-b56f-87c483940ba6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-px5bb\" (UID: \"6566515d-9e15-4679-b56f-87c483940ba6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-px5bb" Nov 02 07:03:46.363701 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:46.363646 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6566515d-9e15-4679-b56f-87c483940ba6-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-px5bb\" (UID: \"6566515d-9e15-4679-b56f-87c483940ba6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-px5bb" Nov 02 07:03:46.416133 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:46.416099 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6566515d-9e15-4679-b56f-87c483940ba6-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-px5bb\" (UID: \"6566515d-9e15-4679-b56f-87c483940ba6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-px5bb" Nov 02 07:03:46.416282 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:46.416109 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5fcq\" (UniqueName: \"kubernetes.io/projected/6566515d-9e15-4679-b56f-87c483940ba6-kube-api-access-n5fcq\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-px5bb\" (UID: \"6566515d-9e15-4679-b56f-87c483940ba6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-px5bb" Nov 02 07:03:46.437321 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:46.437064 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-px5bb" Nov 02 07:03:46.885259 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:46.885222 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-px5bb"] Nov 02 07:03:46.893319 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:46.893290 5916 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 07:03:46.893423 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:46.893352 5916 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 07:03:46.893423 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:46.893379 5916 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 07:03:46.975180 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:46.975142 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-px5bb" event={"ID":"6566515d-9e15-4679-b56f-87c483940ba6","Type":"ContainerStarted","Data":"bdf3240a0dee5f4b3281f29839c9813b20486ffc185688379222aff848bba59b"} Nov 02 07:03:47.984285 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:47.984240 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-px5bb" event={"ID":"6566515d-9e15-4679-b56f-87c483940ba6","Type":"ContainerStarted","Data":"eb6b41e450bfdd62bed135653623a543f6c3a0a990b3dbf8b28e720651f45e71"} Nov 02 07:03:48.204729 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:48.204675 5916 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-px5bb" podStartSLOduration=2.204659851 podStartE2EDuration="2.204659851s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:03:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:03:48.153445294 +0000 UTC m=+2816.425265689" watchObservedRunningTime="2026-11-02 07:03:48.204659851 +0000 UTC m=+2816.476480244" Nov 02 07:03:48.444929 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:48.442233 5916 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-px5bb" Nov 02 07:03:48.446069 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:48.446041 5916 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-px5bb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" start-of-body= Nov 02 07:03:48.446276 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:48.446251 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-px5bb" podUID="6566515d-9e15-4679-b56f-87c483940ba6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" Nov 02 07:03:49.444401 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:49.444364 5916 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-px5bb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" start-of-body= Nov 02 07:03:49.445057 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:49.445027 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-px5bb" podUID="6566515d-9e15-4679-b56f-87c483940ba6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" Nov 02 07:03:49.892785 ip-10-0-130-93 kubenswrapper[5916]: E1102 07:03:49.892751 5916 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p98cq" containerName="extractor" Nov 02 07:03:50.444333 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:50.444293 5916 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-px5bb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" start-of-body= Nov 02 07:03:50.444680 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:50.444649 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-px5bb" podUID="6566515d-9e15-4679-b56f-87c483940ba6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" Nov 02 07:03:51.445260 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:51.445221 5916 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-px5bb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" start-of-body= Nov 02 07:03:51.445867 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:51.445835 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-px5bb" podUID="6566515d-9e15-4679-b56f-87c483940ba6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" Nov 02 07:03:52.445928 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:52.445229 5916 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-px5bb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" start-of-body= Nov 02 07:03:52.445928 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:52.445304 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-px5bb" podUID="6566515d-9e15-4679-b56f-87c483940ba6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" Nov 02 07:03:53.172597 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:53.172556 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-59fb8fd585-9s9g9"] Nov 02 07:03:53.172814 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:53.172752 5916 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-59fb8fd585-9s9g9" podUID="24a75a7e-78f3-421a-ab7e-77fdb5a1279d" containerName="manager" containerID="cri-o://24da47a175b4efab88fab4d4a4284c4cab0ca8f7548d16880160c554992e3f2a" gracePeriod=30 Nov 02 07:03:53.417843 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:53.417818 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-59fb8fd585-9s9g9" Nov 02 07:03:53.438204 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:53.438144 5916 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-px5bb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" start-of-body= Nov 02 07:03:53.438313 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:53.438200 5916 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-px5bb" podUID="6566515d-9e15-4679-b56f-87c483940ba6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" Nov 02 07:03:53.526045 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:53.526016 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24a75a7e-78f3-421a-ab7e-77fdb5a1279d-kube-api-access-xd7v4\" (UniqueName: \"kubernetes.io/projected/24a75a7e-78f3-421a-ab7e-77fdb5a1279d-kube-api-access-xd7v4\") pod \"24a75a7e-78f3-421a-ab7e-77fdb5a1279d\" (UID: \"24a75a7e-78f3-421a-ab7e-77fdb5a1279d\") " Nov 02 07:03:53.526446 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:53.526059 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/24a75a7e-78f3-421a-ab7e-77fdb5a1279d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/24a75a7e-78f3-421a-ab7e-77fdb5a1279d-metrics-certs\") pod \"24a75a7e-78f3-421a-ab7e-77fdb5a1279d\" (UID: \"24a75a7e-78f3-421a-ab7e-77fdb5a1279d\") " Nov 02 07:03:53.526446 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:53.526115 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/24a75a7e-78f3-421a-ab7e-77fdb5a1279d-webhook-cert\" (UniqueName: \"kubernetes.io/secret/24a75a7e-78f3-421a-ab7e-77fdb5a1279d-webhook-cert\") pod \"24a75a7e-78f3-421a-ab7e-77fdb5a1279d\" (UID: \"24a75a7e-78f3-421a-ab7e-77fdb5a1279d\") " Nov 02 07:03:53.528276 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:53.528246 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24a75a7e-78f3-421a-ab7e-77fdb5a1279d-webhook-cert" pod "24a75a7e-78f3-421a-ab7e-77fdb5a1279d" (UID: "24a75a7e-78f3-421a-ab7e-77fdb5a1279d"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 07:03:53.528384 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:53.528318 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24a75a7e-78f3-421a-ab7e-77fdb5a1279d-kube-api-access-xd7v4" pod "24a75a7e-78f3-421a-ab7e-77fdb5a1279d" (UID: "24a75a7e-78f3-421a-ab7e-77fdb5a1279d"). InnerVolumeSpecName "kube-api-access-xd7v4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:03:53.528384 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:53.528334 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24a75a7e-78f3-421a-ab7e-77fdb5a1279d-metrics-certs" pod "24a75a7e-78f3-421a-ab7e-77fdb5a1279d" (UID: "24a75a7e-78f3-421a-ab7e-77fdb5a1279d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 07:03:53.627594 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:53.627562 5916 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/24a75a7e-78f3-421a-ab7e-77fdb5a1279d-webhook-cert\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:03:53.627755 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:53.627610 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xd7v4\" (UniqueName: \"kubernetes.io/projected/24a75a7e-78f3-421a-ab7e-77fdb5a1279d-kube-api-access-xd7v4\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:03:53.627755 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:53.627620 5916 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24a75a7e-78f3-421a-ab7e-77fdb5a1279d-metrics-certs\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:03:54.022109 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:54.022069 5916 generic.go:356] "Generic (PLEG): container finished" podID="24a75a7e-78f3-421a-ab7e-77fdb5a1279d" containerID="24da47a175b4efab88fab4d4a4284c4cab0ca8f7548d16880160c554992e3f2a" exitCode=0 Nov 02 07:03:54.022266 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:54.022151 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-59fb8fd585-9s9g9" Nov 02 07:03:54.022266 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:54.022150 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-59fb8fd585-9s9g9" event={"ID":"24a75a7e-78f3-421a-ab7e-77fdb5a1279d","Type":"ContainerDied","Data":"24da47a175b4efab88fab4d4a4284c4cab0ca8f7548d16880160c554992e3f2a"} Nov 02 07:03:54.022383 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:54.022273 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-59fb8fd585-9s9g9" event={"ID":"24a75a7e-78f3-421a-ab7e-77fdb5a1279d","Type":"ContainerDied","Data":"121746f9ab5736bccf4ac5079323b8008a1c4b8fb7ef3732e1826e073c1cbd15"} Nov 02 07:03:54.022383 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:54.022305 5916 scope.go:122] "RemoveContainer" containerID="24da47a175b4efab88fab4d4a4284c4cab0ca8f7548d16880160c554992e3f2a" Nov 02 07:03:54.032036 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:54.032018 5916 scope.go:122] "RemoveContainer" containerID="24da47a175b4efab88fab4d4a4284c4cab0ca8f7548d16880160c554992e3f2a" Nov 02 07:03:54.032334 ip-10-0-130-93 kubenswrapper[5916]: E1102 07:03:54.032309 5916 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24da47a175b4efab88fab4d4a4284c4cab0ca8f7548d16880160c554992e3f2a\": container with ID starting with 24da47a175b4efab88fab4d4a4284c4cab0ca8f7548d16880160c554992e3f2a not found: ID does not exist" containerID="24da47a175b4efab88fab4d4a4284c4cab0ca8f7548d16880160c554992e3f2a" Nov 02 07:03:54.032426 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:54.032342 5916 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24da47a175b4efab88fab4d4a4284c4cab0ca8f7548d16880160c554992e3f2a"} err="failed to get container status \"24da47a175b4efab88fab4d4a4284c4cab0ca8f7548d16880160c554992e3f2a\": rpc error: code = NotFound desc = could not find container \"24da47a175b4efab88fab4d4a4284c4cab0ca8f7548d16880160c554992e3f2a\": container with ID starting with 24da47a175b4efab88fab4d4a4284c4cab0ca8f7548d16880160c554992e3f2a not found: ID does not exist" Nov 02 07:03:54.187654 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:54.187613 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-59fb8fd585-9s9g9"] Nov 02 07:03:54.237135 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:54.237104 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-59fb8fd585-9s9g9"] Nov 02 07:03:54.928780 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:54.928752 5916 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-px5bb" Nov 02 07:03:54.929524 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:54.929497 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-px5bb" Nov 02 07:03:54.930297 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:54.930281 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-px5bb" Nov 02 07:03:55.840656 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:03:55.840625 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24a75a7e-78f3-421a-ab7e-77fdb5a1279d" path="/var/lib/kubelet/pods/24a75a7e-78f3-421a-ab7e-77fdb5a1279d/volumes" Nov 02 07:04:07.047625 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:04:07.047590 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a1de0b6-rhk79"] Nov 02 07:04:07.097328 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:04:07.097294 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a1de0b6-rhk79"] Nov 02 07:04:07.841158 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:04:07.841122 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e55fd1b3-1171-4b06-a303-34952600343f" path="/var/lib/kubelet/pods/e55fd1b3-1171-4b06-a303-34952600343f/volumes" Nov 02 07:04:17.048177 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:04:17.048133 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb05257d-gwf8w"] Nov 02 07:04:17.097859 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:04:17.097826 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb05257d-gwf8w"] Nov 02 07:04:17.841584 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:04:17.841550 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca23ad83-835d-4120-94f8-2227afef03c5" path="/var/lib/kubelet/pods/ca23ad83-835d-4120-94f8-2227afef03c5/volumes" Nov 02 07:04:19.894217 ip-10-0-130-93 kubenswrapper[5916]: E1102 07:04:19.894180 5916 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p98cq" containerName="extractor" Nov 02 07:04:20.048225 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:04:20.048189 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f3f1755-4qrwx"] Nov 02 07:04:20.098220 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:04:20.098185 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f3f1755-4qrwx"] Nov 02 07:04:21.841550 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:04:21.841514 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e5389b4-2147-4e2f-b5e7-f16a760ca9a2" path="/var/lib/kubelet/pods/2e5389b4-2147-4e2f-b5e7-f16a760ca9a2/volumes" Nov 02 07:04:36.057366 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:04:36.057326 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44df87b4-sv978"] Nov 02 07:04:36.107118 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:04:36.107087 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44df87b4-sv978"] Nov 02 07:04:37.841079 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:04:37.841039 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8519ee5f-3fde-4c74-afdf-f30eca48ad21" path="/var/lib/kubelet/pods/8519ee5f-3fde-4c74-afdf-f30eca48ad21/volumes" Nov 02 07:04:49.892689 ip-10-0-130-93 kubenswrapper[5916]: E1102 07:04:49.892649 5916 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p98cq" containerName="extractor" Nov 02 07:04:55.775883 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:04:55.775848 5916 scope.go:122] "RemoveContainer" containerID="1ec051afde7eff79711cb9ac5990a09a0df215d1ac0145647336e8ccf7dd4d79" Nov 02 07:05:19.893314 ip-10-0-130-93 kubenswrapper[5916]: E1102 07:05:19.893274 5916 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p98cq" containerName="extractor" Nov 02 07:05:45.056403 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:05:45.056360 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b87d28f4-pbmdq"] Nov 02 07:05:45.106359 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:05:45.106321 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2212d36a-mjpqb"] Nov 02 07:05:45.117279 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:05:45.117238 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b87d28f4-pbmdq"] Nov 02 07:05:45.120182 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:05:45.120154 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2212d36a-mjpqb"] Nov 02 07:05:45.841713 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:05:45.841677 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17bfa708-9d69-4ca7-ac46-d987e81292a7" path="/var/lib/kubelet/pods/17bfa708-9d69-4ca7-ac46-d987e81292a7/volumes" Nov 02 07:05:45.842111 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:05:45.842097 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bdb5f812-0bc7-4d0a-a5aa-fe18615da989" path="/var/lib/kubelet/pods/bdb5f812-0bc7-4d0a-a5aa-fe18615da989/volumes" Nov 02 07:05:49.922839 ip-10-0-130-93 kubenswrapper[5916]: E1102 07:05:49.922804 5916 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p98cq" containerName="extractor" Nov 02 07:05:55.803645 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:05:55.803609 5916 scope.go:122] "RemoveContainer" containerID="36dba79bb7b8f68b18bfbe61433ac299162d04f9d54a7d42609084c756c589a4" Nov 02 07:05:55.812769 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:05:55.812750 5916 scope.go:122] "RemoveContainer" containerID="98e01d30edee97c7d8c7855e2be111d680d5be6e9b45dcedb3a2c145d776cd25" Nov 02 07:05:55.821206 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:05:55.821184 5916 scope.go:122] "RemoveContainer" containerID="24b51bc8e2a75779525c9f6e9aa06a8b20a5625ec055cc4c90b59226670bb3b8" Nov 02 07:05:55.829172 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:05:55.829151 5916 scope.go:122] "RemoveContainer" containerID="84324208460eb65a7eeaf5cd8d2fcbbc25b80698997b4b845d7514b3c5828820" Nov 02 07:06:19.896486 ip-10-0-130-93 kubenswrapper[5916]: E1102 07:06:19.896440 5916 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p98cq" containerName="extractor" Nov 02 07:06:45.056168 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:06:45.056082 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa9aa932-ztxp2"] Nov 02 07:06:45.081638 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:06:45.081608 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f4e008a-8n8kr"] Nov 02 07:06:45.105962 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:06:45.105933 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f4e008a-8n8kr"] Nov 02 07:06:45.107943 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:06:45.107923 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa9aa932-ztxp2"] Nov 02 07:06:45.842043 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:06:45.842008 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36233f6d-5ba6-4f5d-b148-da1ff598286b" path="/var/lib/kubelet/pods/36233f6d-5ba6-4f5d-b148-da1ff598286b/volumes" Nov 02 07:06:45.842471 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:06:45.842456 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83ec9f9d-b0b5-4eaa-aa0d-69c43c283c2d" path="/var/lib/kubelet/pods/83ec9f9d-b0b5-4eaa-aa0d-69c43c283c2d/volumes" Nov 02 07:06:49.893314 ip-10-0-130-93 kubenswrapper[5916]: E1102 07:06:49.893276 5916 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p98cq" containerName="extractor" Nov 02 07:06:53.312638 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:06:53.312610 5916 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-csfhg_17dee72b-2f47-4016-8ffe-3a89934e66ee/console-operator/2.log" Nov 02 07:06:53.324257 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:06:53.324235 5916 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-csfhg_17dee72b-2f47-4016-8ffe-3a89934e66ee/console-operator/2.log" Nov 02 07:06:55.876942 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:06:55.876894 5916 scope.go:122] "RemoveContainer" containerID="cf9ca3a606f9776f13687bc4e6d4d4b8111e358676eb0f33a090471dc24db776" Nov 02 07:06:55.886446 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:06:55.886426 5916 scope.go:122] "RemoveContainer" containerID="d12942b7208c3798c4d501220a72d41ae837aa329a81419557c944c20b983077" Nov 02 07:06:55.894804 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:06:55.894784 5916 scope.go:122] "RemoveContainer" containerID="0cf12270d11cde0e9f57f6b402c8fdf574c7a7e0a2e5db6e9a9c3b91390e6b27" Nov 02 07:06:55.902819 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:06:55.902799 5916 scope.go:122] "RemoveContainer" containerID="a654f013e0dc5dcf471c4e6daf93aa7848ed84524e93c3fc5fbba0625a979cb6" Nov 02 07:06:55.910683 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:06:55.910664 5916 scope.go:122] "RemoveContainer" containerID="d9d1973197793ed88b68766b1e282f0eaf612a4d2194a36d74738c82a2d9b833" Nov 02 07:06:55.918196 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:06:55.918179 5916 scope.go:122] "RemoveContainer" containerID="9a12ada894602c49ebede56bf7d086db1f63b4ba855ca830e48b4cb587d3ba97" Nov 02 07:06:55.925922 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:06:55.925891 5916 scope.go:122] "RemoveContainer" containerID="281d2cbfa8b190b7d462b680171eef0bc3227fa9f2e5fa4e46bef9413c5a9ef3" Nov 02 07:07:19.893003 ip-10-0-130-93 kubenswrapper[5916]: E1102 07:07:19.892962 5916 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p98cq" containerName="extractor" Nov 02 07:07:40.983045 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:07:40.982990 5916 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-px5bb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.222:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 07:07:40.983702 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:07:40.983652 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-px5bb" podUID="6566515d-9e15-4679-b56f-87c483940ba6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.222:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 07:07:40.988036 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:07:40.988004 5916 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-6527w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.221:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 07:07:40.988209 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:07:40.988187 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6527w" podUID="918ad956-7d56-4627-a1a4-e7c75694d1f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.221:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 07:07:41.042140 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:07:41.041627 5916 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-tnptl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 07:07:41.042383 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:07:41.042356 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tnptl" podUID="a24071b2-e468-4590-93ea-1be6ac017cf8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 07:07:49.055806 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:07:49.055769 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d48fbdd4-5cfql"] Nov 02 07:07:49.105847 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:07:49.105814 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d48fbdd4-5cfql"] Nov 02 07:07:49.841889 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:07:49.841855 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db3677b9-3aa0-415f-a6a0-e7e87e2aad7f" path="/var/lib/kubelet/pods/db3677b9-3aa0-415f-a6a0-e7e87e2aad7f/volumes" Nov 02 07:07:49.895289 ip-10-0-130-93 kubenswrapper[5916]: E1102 07:07:49.895249 5916 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p98cq" containerName="extractor" Nov 02 07:07:55.972606 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:07:55.972577 5916 scope.go:122] "RemoveContainer" containerID="5ec8838f066f6852066dfe62f7003c995a5a612f4e98f3f65495f5c4ac783c90" Nov 02 07:07:55.981341 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:07:55.981321 5916 scope.go:122] "RemoveContainer" containerID="a77e27297af4c2bfad519fa2596405e0714eb0dcd44511086a8db15d184a74f8" Nov 02 07:07:55.989241 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:07:55.989226 5916 scope.go:122] "RemoveContainer" containerID="c97181451da2384040f654c5060d6441f23ebf964c6fd8ad8a0be1849b0d59b7" Nov 02 07:07:55.996966 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:07:55.996950 5916 scope.go:122] "RemoveContainer" containerID="7d72d4477c27747c2c0461881a22f853e9908752a11ed0bdeedd357063e3f180" Nov 02 07:08:19.892333 ip-10-0-130-93 kubenswrapper[5916]: E1102 07:08:19.892293 5916 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p98cq" containerName="extractor" Nov 02 07:08:25.973114 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:08:25.973069 5916 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-px5bb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.222:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 07:08:25.973685 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:08:25.973146 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-px5bb" podUID="6566515d-9e15-4679-b56f-87c483940ba6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.222:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 07:08:25.974312 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:08:25.974283 5916 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-6527w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.221:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 07:08:25.974467 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:08:25.974330 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6527w" podUID="918ad956-7d56-4627-a1a4-e7c75694d1f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.221:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 07:08:49.893173 ip-10-0-130-93 kubenswrapper[5916]: E1102 07:08:49.893135 5916 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p98cq" containerName="extractor" Nov 02 07:08:56.036974 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:08:56.036946 5916 scope.go:122] "RemoveContainer" containerID="dd9b9bc5ce94ad8319a4888a09ec6e69a660ad7e1634f6b3683c3f678275bb12" Nov 02 07:08:56.046011 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:08:56.045987 5916 scope.go:122] "RemoveContainer" containerID="eb1c5ed06cfdb6920145dcd563a297fe5426bdb5c849824fd57a18a2ee2504c3" Nov 02 07:08:56.054012 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:08:56.053989 5916 scope.go:122] "RemoveContainer" containerID="b7a0a1239daa6b1ee3633d5ff24889c1b09671af56de03dc167087976cf1a4d9" Nov 02 07:09:19.893582 ip-10-0-130-93 kubenswrapper[5916]: E1102 07:09:19.893543 5916 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p98cq" containerName="extractor" Nov 02 07:09:49.892667 ip-10-0-130-93 kubenswrapper[5916]: E1102 07:09:49.892582 5916 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p98cq" containerName="extractor" Nov 02 07:10:19.896793 ip-10-0-130-93 kubenswrapper[5916]: E1102 07:10:19.896755 5916 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p98cq" containerName="extractor" Nov 02 07:10:49.894061 ip-10-0-130-93 kubenswrapper[5916]: E1102 07:10:49.894021 5916 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p98cq" containerName="extractor" Nov 02 07:11:19.893469 ip-10-0-130-93 kubenswrapper[5916]: E1102 07:11:19.893432 5916 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p98cq" containerName="extractor" Nov 02 07:11:49.893459 ip-10-0-130-93 kubenswrapper[5916]: E1102 07:11:49.893419 5916 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p98cq" containerName="extractor" Nov 02 07:11:53.358303 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:11:53.358273 5916 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-csfhg_17dee72b-2f47-4016-8ffe-3a89934e66ee/console-operator/2.log" Nov 02 07:11:53.372296 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:11:53.372267 5916 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-csfhg_17dee72b-2f47-4016-8ffe-3a89934e66ee/console-operator/2.log" Nov 02 07:11:57.264587 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:11:57.264552 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ee03e20-scgm4"] Nov 02 07:11:57.265129 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:11:57.265110 5916 cpu_manager.go:414] "containerMap: removing container" podUID="24a75a7e-78f3-421a-ab7e-77fdb5a1279d" containerName="manager" Nov 02 07:11:57.265213 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:11:57.265132 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24a75a7e-78f3-421a-ab7e-77fdb5a1279d" containerName="manager" Nov 02 07:11:57.265298 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:11:57.265285 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="24a75a7e-78f3-421a-ab7e-77fdb5a1279d" containerName="manager" Nov 02 07:11:57.268498 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:11:57.268478 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ee03e20-scgm4" Nov 02 07:11:57.326790 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:11:57.326766 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ee03e20-scgm4"] Nov 02 07:11:57.373753 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:11:57.373732 5916 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 07:11:57.373867 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:11:57.373767 5916 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 07:11:57.373867 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:11:57.373807 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-68l4r\"" Nov 02 07:11:57.373867 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:11:57.373807 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 07:11:57.444208 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:11:57.444185 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e7910e1-f0eb-4088-accb-7e59f06626f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8ee03e20-scgm4\" (UID: \"3e7910e1-f0eb-4088-accb-7e59f06626f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ee03e20-scgm4" Nov 02 07:11:57.444311 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:11:57.444231 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqfbk\" (UniqueName: \"kubernetes.io/projected/3e7910e1-f0eb-4088-accb-7e59f06626f9-kube-api-access-sqfbk\") pod \"maas-api-revoke-sub-8ee03e20-scgm4\" (UID: \"3e7910e1-f0eb-4088-accb-7e59f06626f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ee03e20-scgm4" Nov 02 07:11:57.444311 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:11:57.444250 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e7910e1-f0eb-4088-accb-7e59f06626f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8ee03e20-scgm4\" (UID: \"3e7910e1-f0eb-4088-accb-7e59f06626f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ee03e20-scgm4" Nov 02 07:11:57.545364 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:11:57.545313 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e7910e1-f0eb-4088-accb-7e59f06626f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8ee03e20-scgm4\" (UID: \"3e7910e1-f0eb-4088-accb-7e59f06626f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ee03e20-scgm4" Nov 02 07:11:57.545364 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:11:57.545358 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqfbk\" (UniqueName: \"kubernetes.io/projected/3e7910e1-f0eb-4088-accb-7e59f06626f9-kube-api-access-sqfbk\") pod \"maas-api-revoke-sub-8ee03e20-scgm4\" (UID: \"3e7910e1-f0eb-4088-accb-7e59f06626f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ee03e20-scgm4" Nov 02 07:11:57.545540 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:11:57.545379 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e7910e1-f0eb-4088-accb-7e59f06626f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8ee03e20-scgm4\" (UID: \"3e7910e1-f0eb-4088-accb-7e59f06626f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ee03e20-scgm4" Nov 02 07:11:57.546017 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:11:57.545999 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e7910e1-f0eb-4088-accb-7e59f06626f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8ee03e20-scgm4\" (UID: \"3e7910e1-f0eb-4088-accb-7e59f06626f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ee03e20-scgm4" Nov 02 07:11:57.547648 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:11:57.547626 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e7910e1-f0eb-4088-accb-7e59f06626f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8ee03e20-scgm4\" (UID: \"3e7910e1-f0eb-4088-accb-7e59f06626f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ee03e20-scgm4" Nov 02 07:11:57.602694 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:11:57.602670 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqfbk\" (UniqueName: \"kubernetes.io/projected/3e7910e1-f0eb-4088-accb-7e59f06626f9-kube-api-access-sqfbk\") pod \"maas-api-revoke-sub-8ee03e20-scgm4\" (UID: \"3e7910e1-f0eb-4088-accb-7e59f06626f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ee03e20-scgm4" Nov 02 07:11:57.880729 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:11:57.880660 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ee03e20-scgm4" Nov 02 07:11:58.254626 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:11:58.254602 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ee03e20-scgm4"] Nov 02 07:11:58.257311 ip-10-0-130-93 kubenswrapper[5916]: W1102 07:11:58.257281 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e7910e1_f0eb_4088_accb_7e59f06626f9.slice/crio-0aefd6f22fef4ab436de27fe3e8bed371686d85191bb3daf9004b9fa0456cfe9 WatchSource:0}: Error finding container 0aefd6f22fef4ab436de27fe3e8bed371686d85191bb3daf9004b9fa0456cfe9: Status 404 returned error can't find the container with id 0aefd6f22fef4ab436de27fe3e8bed371686d85191bb3daf9004b9fa0456cfe9 Nov 02 07:11:58.999771 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:11:58.999735 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ee03e20-scgm4" event={"ID":"3e7910e1-f0eb-4088-accb-7e59f06626f9","Type":"ContainerStarted","Data":"4f28148c448b0c9b1cc67570a2345a5f49ab8d9275fb152a160fecfc1325d9e8"} Nov 02 07:11:59.000215 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:11:58.999821 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ee03e20-scgm4" event={"ID":"3e7910e1-f0eb-4088-accb-7e59f06626f9","Type":"ContainerStarted","Data":"0aefd6f22fef4ab436de27fe3e8bed371686d85191bb3daf9004b9fa0456cfe9"} Nov 02 07:11:59.122290 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:11:59.122240 5916 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ee03e20-scgm4" podStartSLOduration=2.122228348 podStartE2EDuration="2.122228348s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:11:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:11:59.121810203 +0000 UTC m=+3307.393630597" watchObservedRunningTime="2026-11-02 07:11:59.122228348 +0000 UTC m=+3307.394048742" Nov 02 07:12:00.006202 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:12:00.006167 5916 generic.go:356] "Generic (PLEG): container finished" podID="3e7910e1-f0eb-4088-accb-7e59f06626f9" containerID="4f28148c448b0c9b1cc67570a2345a5f49ab8d9275fb152a160fecfc1325d9e8" exitCode=0 Nov 02 07:12:00.006622 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:12:00.006250 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ee03e20-scgm4" event={"ID":"3e7910e1-f0eb-4088-accb-7e59f06626f9","Type":"ContainerDied","Data":"4f28148c448b0c9b1cc67570a2345a5f49ab8d9275fb152a160fecfc1325d9e8"} Nov 02 07:12:01.142935 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:12:01.142890 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ee03e20-scgm4" Nov 02 07:12:01.279502 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:12:01.279432 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e7910e1-f0eb-4088-accb-7e59f06626f9-kube-api-access-sqfbk\" (UniqueName: \"kubernetes.io/projected/3e7910e1-f0eb-4088-accb-7e59f06626f9-kube-api-access-sqfbk\") pod \"3e7910e1-f0eb-4088-accb-7e59f06626f9\" (UID: \"3e7910e1-f0eb-4088-accb-7e59f06626f9\") " Nov 02 07:12:01.279653 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:12:01.279526 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3e7910e1-f0eb-4088-accb-7e59f06626f9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e7910e1-f0eb-4088-accb-7e59f06626f9-maas-api-xks-ca\") pod \"3e7910e1-f0eb-4088-accb-7e59f06626f9\" (UID: \"3e7910e1-f0eb-4088-accb-7e59f06626f9\") " Nov 02 07:12:01.279653 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:12:01.279557 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e7910e1-f0eb-4088-accb-7e59f06626f9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e7910e1-f0eb-4088-accb-7e59f06626f9-maas-api-openshift-ca\") pod \"3e7910e1-f0eb-4088-accb-7e59f06626f9\" (UID: \"3e7910e1-f0eb-4088-accb-7e59f06626f9\") " Nov 02 07:12:01.280052 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:12:01.280023 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e7910e1-f0eb-4088-accb-7e59f06626f9-maas-api-openshift-ca" pod "3e7910e1-f0eb-4088-accb-7e59f06626f9" (UID: "3e7910e1-f0eb-4088-accb-7e59f06626f9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:12:01.281507 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:12:01.281487 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e7910e1-f0eb-4088-accb-7e59f06626f9-maas-api-xks-ca" pod "3e7910e1-f0eb-4088-accb-7e59f06626f9" (UID: "3e7910e1-f0eb-4088-accb-7e59f06626f9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 07:12:01.281575 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:12:01.281522 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e7910e1-f0eb-4088-accb-7e59f06626f9-kube-api-access-sqfbk" pod "3e7910e1-f0eb-4088-accb-7e59f06626f9" (UID: "3e7910e1-f0eb-4088-accb-7e59f06626f9"). InnerVolumeSpecName "kube-api-access-sqfbk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:12:01.380225 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:12:01.380201 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqfbk\" (UniqueName: \"kubernetes.io/projected/3e7910e1-f0eb-4088-accb-7e59f06626f9-kube-api-access-sqfbk\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:12:01.380225 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:12:01.380223 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e7910e1-f0eb-4088-accb-7e59f06626f9-maas-api-xks-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:12:01.380362 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:12:01.380233 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e7910e1-f0eb-4088-accb-7e59f06626f9-maas-api-openshift-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:12:02.018982 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:12:02.018951 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ee03e20-scgm4" event={"ID":"3e7910e1-f0eb-4088-accb-7e59f06626f9","Type":"ContainerDied","Data":"0aefd6f22fef4ab436de27fe3e8bed371686d85191bb3daf9004b9fa0456cfe9"} Nov 02 07:12:02.018982 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:12:02.018966 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ee03e20-scgm4" Nov 02 07:12:02.019207 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:12:02.018997 5916 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0aefd6f22fef4ab436de27fe3e8bed371686d85191bb3daf9004b9fa0456cfe9" Nov 02 07:12:09.319003 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:12:09.318967 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-614951-5sbmh"] Nov 02 07:12:09.319406 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:12:09.319394 5916 cpu_manager.go:414] "containerMap: removing container" podUID="3e7910e1-f0eb-4088-accb-7e59f06626f9" containerName="revoke-keys" Nov 02 07:12:09.319449 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:12:09.319406 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e7910e1-f0eb-4088-accb-7e59f06626f9" containerName="revoke-keys" Nov 02 07:12:09.319485 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:12:09.319479 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e7910e1-f0eb-4088-accb-7e59f06626f9" containerName="revoke-keys" Nov 02 07:12:09.322775 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:12:09.322756 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-614951-5sbmh" Nov 02 07:12:09.343217 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:12:09.343195 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e67ff51-9e58-4ce1-bbff-940755b2020e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-614951-5sbmh\" (UID: \"0e67ff51-9e58-4ce1-bbff-940755b2020e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-614951-5sbmh" Nov 02 07:12:09.343342 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:12:09.343222 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwvgw\" (UniqueName: \"kubernetes.io/projected/0e67ff51-9e58-4ce1-bbff-940755b2020e-kube-api-access-fwvgw\") pod \"maas-api-revoke-keys-e2e-worker-single-main-614951-5sbmh\" (UID: \"0e67ff51-9e58-4ce1-bbff-940755b2020e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-614951-5sbmh" Nov 02 07:12:09.343342 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:12:09.343248 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e67ff51-9e58-4ce1-bbff-940755b2020e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-614951-5sbmh\" (UID: \"0e67ff51-9e58-4ce1-bbff-940755b2020e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-614951-5sbmh" Nov 02 07:12:09.379962 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:12:09.379897 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-614951-5sbmh"] Nov 02 07:12:09.427177 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:12:09.427154 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 07:12:09.427177 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:12:09.427171 5916 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 07:12:09.427328 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:12:09.427218 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-68l4r\"" Nov 02 07:12:09.427328 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:12:09.427287 5916 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 07:12:09.444659 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:12:09.444638 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e67ff51-9e58-4ce1-bbff-940755b2020e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-614951-5sbmh\" (UID: \"0e67ff51-9e58-4ce1-bbff-940755b2020e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-614951-5sbmh" Nov 02 07:12:09.444770 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:12:09.444663 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwvgw\" (UniqueName: \"kubernetes.io/projected/0e67ff51-9e58-4ce1-bbff-940755b2020e-kube-api-access-fwvgw\") pod \"maas-api-revoke-keys-e2e-worker-single-main-614951-5sbmh\" (UID: \"0e67ff51-9e58-4ce1-bbff-940755b2020e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-614951-5sbmh" Nov 02 07:12:09.444770 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:12:09.444689 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e67ff51-9e58-4ce1-bbff-940755b2020e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-614951-5sbmh\" (UID: \"0e67ff51-9e58-4ce1-bbff-940755b2020e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-614951-5sbmh" Nov 02 07:12:09.445289 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:12:09.445268 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e67ff51-9e58-4ce1-bbff-940755b2020e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-614951-5sbmh\" (UID: \"0e67ff51-9e58-4ce1-bbff-940755b2020e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-614951-5sbmh" Nov 02 07:12:09.446971 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:12:09.446949 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e67ff51-9e58-4ce1-bbff-940755b2020e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-614951-5sbmh\" (UID: \"0e67ff51-9e58-4ce1-bbff-940755b2020e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-614951-5sbmh" Nov 02 07:12:09.502094 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:12:09.502067 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwvgw\" (UniqueName: \"kubernetes.io/projected/0e67ff51-9e58-4ce1-bbff-940755b2020e-kube-api-access-fwvgw\") pod \"maas-api-revoke-keys-e2e-worker-single-main-614951-5sbmh\" (UID: \"0e67ff51-9e58-4ce1-bbff-940755b2020e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-614951-5sbmh" Nov 02 07:12:09.637844 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:12:09.637821 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-614951-5sbmh" Nov 02 07:12:10.008554 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:12:10.008527 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-614951-5sbmh"] Nov 02 07:12:10.011033 ip-10-0-130-93 kubenswrapper[5916]: W1102 07:12:10.011005 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e67ff51_9e58_4ce1_bbff_940755b2020e.slice/crio-ce6d56e4bfa6ce1e1c3036d294e7ed8ebf20de13c287ca1dcfe363d07808dc83 WatchSource:0}: Error finding container ce6d56e4bfa6ce1e1c3036d294e7ed8ebf20de13c287ca1dcfe363d07808dc83: Status 404 returned error can't find the container with id ce6d56e4bfa6ce1e1c3036d294e7ed8ebf20de13c287ca1dcfe363d07808dc83 Nov 02 07:12:10.063385 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:12:10.063354 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-614951-5sbmh" event={"ID":"0e67ff51-9e58-4ce1-bbff-940755b2020e","Type":"ContainerStarted","Data":"ce6d56e4bfa6ce1e1c3036d294e7ed8ebf20de13c287ca1dcfe363d07808dc83"} Nov 02 07:12:11.070953 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:12:11.070899 5916 generic.go:356] "Generic (PLEG): container finished" podID="0e67ff51-9e58-4ce1-bbff-940755b2020e" containerID="eb77b61a49a2df10cd047693747517b95e0123c2ac6d8e0c011d2f20c4128454" exitCode=0 Nov 02 07:12:11.071361 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:12:11.070943 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-614951-5sbmh" event={"ID":"0e67ff51-9e58-4ce1-bbff-940755b2020e","Type":"ContainerDied","Data":"eb77b61a49a2df10cd047693747517b95e0123c2ac6d8e0c011d2f20c4128454"} Nov 02 07:12:12.220839 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:12:12.220813 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-614951-5sbmh" Nov 02 07:12:12.269283 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:12:12.269255 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e67ff51-9e58-4ce1-bbff-940755b2020e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e67ff51-9e58-4ce1-bbff-940755b2020e-maas-api-openshift-ca\") pod \"0e67ff51-9e58-4ce1-bbff-940755b2020e\" (UID: \"0e67ff51-9e58-4ce1-bbff-940755b2020e\") " Nov 02 07:12:12.269449 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:12:12.269290 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e67ff51-9e58-4ce1-bbff-940755b2020e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e67ff51-9e58-4ce1-bbff-940755b2020e-maas-api-xks-ca\") pod \"0e67ff51-9e58-4ce1-bbff-940755b2020e\" (UID: \"0e67ff51-9e58-4ce1-bbff-940755b2020e\") " Nov 02 07:12:12.269449 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:12:12.269342 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e67ff51-9e58-4ce1-bbff-940755b2020e-kube-api-access-fwvgw\" (UniqueName: \"kubernetes.io/projected/0e67ff51-9e58-4ce1-bbff-940755b2020e-kube-api-access-fwvgw\") pod \"0e67ff51-9e58-4ce1-bbff-940755b2020e\" (UID: \"0e67ff51-9e58-4ce1-bbff-940755b2020e\") " Nov 02 07:12:12.269665 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:12:12.269636 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e67ff51-9e58-4ce1-bbff-940755b2020e-maas-api-openshift-ca" pod "0e67ff51-9e58-4ce1-bbff-940755b2020e" (UID: "0e67ff51-9e58-4ce1-bbff-940755b2020e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:12:12.271421 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:12:12.271397 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e67ff51-9e58-4ce1-bbff-940755b2020e-kube-api-access-fwvgw" pod "0e67ff51-9e58-4ce1-bbff-940755b2020e" (UID: "0e67ff51-9e58-4ce1-bbff-940755b2020e"). InnerVolumeSpecName "kube-api-access-fwvgw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:12:12.271535 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:12:12.271418 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e67ff51-9e58-4ce1-bbff-940755b2020e-maas-api-xks-ca" pod "0e67ff51-9e58-4ce1-bbff-940755b2020e" (UID: "0e67ff51-9e58-4ce1-bbff-940755b2020e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 07:12:12.370328 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:12:12.370241 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e67ff51-9e58-4ce1-bbff-940755b2020e-maas-api-openshift-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:12:12.370629 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:12:12.370601 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e67ff51-9e58-4ce1-bbff-940755b2020e-maas-api-xks-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:12:12.370728 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:12:12.370633 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwvgw\" (UniqueName: \"kubernetes.io/projected/0e67ff51-9e58-4ce1-bbff-940755b2020e-kube-api-access-fwvgw\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:12:13.085689 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:12:13.085648 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-614951-5sbmh" event={"ID":"0e67ff51-9e58-4ce1-bbff-940755b2020e","Type":"ContainerDied","Data":"ce6d56e4bfa6ce1e1c3036d294e7ed8ebf20de13c287ca1dcfe363d07808dc83"} Nov 02 07:12:13.085884 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:12:13.085714 5916 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ce6d56e4bfa6ce1e1c3036d294e7ed8ebf20de13c287ca1dcfe363d07808dc83" Nov 02 07:12:13.085884 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:12:13.085779 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-614951-5sbmh" Nov 02 07:12:19.893091 ip-10-0-130-93 kubenswrapper[5916]: E1102 07:12:19.893057 5916 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p98cq" containerName="extractor" Nov 02 07:12:30.520088 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:12:30.520050 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-614951-5sbmh"] Nov 02 07:12:30.569815 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:12:30.569787 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-614951-5sbmh"] Nov 02 07:12:31.840975 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:12:31.840942 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e67ff51-9e58-4ce1-bbff-940755b2020e" path="/var/lib/kubelet/pods/0e67ff51-9e58-4ce1-bbff-940755b2020e/volumes" Nov 02 07:12:49.892558 ip-10-0-130-93 kubenswrapper[5916]: E1102 07:12:49.892521 5916 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p98cq" containerName="extractor" Nov 02 07:13:13.231515 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:13:13.231478 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-hptdf"] Nov 02 07:13:13.232182 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:13:13.231675 5916 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-hptdf" podUID="0a731aff-55a4-45a9-8ce5-77dbcb2e7a04" containerName="manager" containerID="cri-o://6ffd49c73292e1de14463f6d339f9a9a5b03ec2ec468c726fed0e0dd326f4a24" gracePeriod=10 Nov 02 07:13:13.432711 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:13:13.432673 5916 generic.go:356] "Generic (PLEG): container finished" podID="0a731aff-55a4-45a9-8ce5-77dbcb2e7a04" containerID="6ffd49c73292e1de14463f6d339f9a9a5b03ec2ec468c726fed0e0dd326f4a24" exitCode=0 Nov 02 07:13:13.432887 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:13:13.432743 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-hptdf" event={"ID":"0a731aff-55a4-45a9-8ce5-77dbcb2e7a04","Type":"ContainerDied","Data":"6ffd49c73292e1de14463f6d339f9a9a5b03ec2ec468c726fed0e0dd326f4a24"} Nov 02 07:13:13.487309 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:13:13.487258 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-hptdf" Nov 02 07:13:13.534472 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:13:13.534449 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a731aff-55a4-45a9-8ce5-77dbcb2e7a04-kube-api-access-c4nmd\" (UniqueName: \"kubernetes.io/projected/0a731aff-55a4-45a9-8ce5-77dbcb2e7a04-kube-api-access-c4nmd\") pod \"0a731aff-55a4-45a9-8ce5-77dbcb2e7a04\" (UID: \"0a731aff-55a4-45a9-8ce5-77dbcb2e7a04\") " Nov 02 07:13:13.534573 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:13:13.534541 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a731aff-55a4-45a9-8ce5-77dbcb2e7a04-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0a731aff-55a4-45a9-8ce5-77dbcb2e7a04-extensions-socket-volume\") pod \"0a731aff-55a4-45a9-8ce5-77dbcb2e7a04\" (UID: \"0a731aff-55a4-45a9-8ce5-77dbcb2e7a04\") " Nov 02 07:13:13.534948 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:13:13.534927 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a731aff-55a4-45a9-8ce5-77dbcb2e7a04-extensions-socket-volume" pod "0a731aff-55a4-45a9-8ce5-77dbcb2e7a04" (UID: "0a731aff-55a4-45a9-8ce5-77dbcb2e7a04"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:13:13.536404 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:13:13.536384 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a731aff-55a4-45a9-8ce5-77dbcb2e7a04-kube-api-access-c4nmd" pod "0a731aff-55a4-45a9-8ce5-77dbcb2e7a04" (UID: "0a731aff-55a4-45a9-8ce5-77dbcb2e7a04"). InnerVolumeSpecName "kube-api-access-c4nmd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:13:13.635174 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:13:13.635146 5916 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0a731aff-55a4-45a9-8ce5-77dbcb2e7a04-extensions-socket-volume\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:13:13.635174 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:13:13.635174 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4nmd\" (UniqueName: \"kubernetes.io/projected/0a731aff-55a4-45a9-8ce5-77dbcb2e7a04-kube-api-access-c4nmd\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:13:14.439001 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:13:14.438970 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-hptdf" Nov 02 07:13:14.439001 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:13:14.438991 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-hptdf" event={"ID":"0a731aff-55a4-45a9-8ce5-77dbcb2e7a04","Type":"ContainerDied","Data":"59ca90844c556b0f945c58b930fdc80e8a7387ef83a551af501da68f5427b758"} Nov 02 07:13:14.439549 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:13:14.439045 5916 scope.go:122] "RemoveContainer" containerID="6ffd49c73292e1de14463f6d339f9a9a5b03ec2ec468c726fed0e0dd326f4a24" Nov 02 07:13:14.654018 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:13:14.653984 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-hptdf"] Nov 02 07:13:14.704084 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:13:14.703897 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-hptdf"] Nov 02 07:13:15.840569 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:13:15.840479 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a731aff-55a4-45a9-8ce5-77dbcb2e7a04" path="/var/lib/kubelet/pods/0a731aff-55a4-45a9-8ce5-77dbcb2e7a04/volumes" Nov 02 07:13:19.893251 ip-10-0-130-93 kubenswrapper[5916]: E1102 07:13:19.893215 5916 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p98cq" containerName="extractor" Nov 02 07:13:49.893172 ip-10-0-130-93 kubenswrapper[5916]: E1102 07:13:49.893133 5916 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p98cq" containerName="extractor" Nov 02 07:14:16.880871 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:14:16.880839 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-45kbd"] Nov 02 07:14:16.881375 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:14:16.881356 5916 cpu_manager.go:414] "containerMap: removing container" podUID="0e67ff51-9e58-4ce1-bbff-940755b2020e" containerName="revoke-keys" Nov 02 07:14:16.881424 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:14:16.881377 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e67ff51-9e58-4ce1-bbff-940755b2020e" containerName="revoke-keys" Nov 02 07:14:16.881424 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:14:16.881407 5916 cpu_manager.go:414] "containerMap: removing container" podUID="0a731aff-55a4-45a9-8ce5-77dbcb2e7a04" containerName="manager" Nov 02 07:14:16.881424 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:14:16.881414 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a731aff-55a4-45a9-8ce5-77dbcb2e7a04" containerName="manager" Nov 02 07:14:16.881519 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:14:16.881470 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e67ff51-9e58-4ce1-bbff-940755b2020e" containerName="revoke-keys" Nov 02 07:14:16.881519 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:14:16.881484 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a731aff-55a4-45a9-8ce5-77dbcb2e7a04" containerName="manager" Nov 02 07:14:16.885745 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:14:16.885729 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-45kbd" Nov 02 07:14:16.941087 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:14:16.941035 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9w2d\" (UniqueName: \"kubernetes.io/projected/506f90d2-dfb4-4869-84d9-d7f3c6726c2f-kube-api-access-g9w2d\") pod \"kuadrant-operator-controller-manager-678d87766f-45kbd\" (UID: \"506f90d2-dfb4-4869-84d9-d7f3c6726c2f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-45kbd" Nov 02 07:14:16.941197 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:14:16.941154 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/506f90d2-dfb4-4869-84d9-d7f3c6726c2f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-45kbd\" (UID: \"506f90d2-dfb4-4869-84d9-d7f3c6726c2f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-45kbd" Nov 02 07:14:16.980366 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:14:16.980340 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-4lbcv\"" Nov 02 07:14:16.982418 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:14:16.982395 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-45kbd"] Nov 02 07:14:17.042407 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:14:17.042377 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9w2d\" (UniqueName: \"kubernetes.io/projected/506f90d2-dfb4-4869-84d9-d7f3c6726c2f-kube-api-access-g9w2d\") pod \"kuadrant-operator-controller-manager-678d87766f-45kbd\" (UID: \"506f90d2-dfb4-4869-84d9-d7f3c6726c2f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-45kbd" Nov 02 07:14:17.042407 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:14:17.042406 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/506f90d2-dfb4-4869-84d9-d7f3c6726c2f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-45kbd\" (UID: \"506f90d2-dfb4-4869-84d9-d7f3c6726c2f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-45kbd" Nov 02 07:14:17.042764 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:14:17.042749 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/506f90d2-dfb4-4869-84d9-d7f3c6726c2f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-45kbd\" (UID: \"506f90d2-dfb4-4869-84d9-d7f3c6726c2f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-45kbd" Nov 02 07:14:17.097929 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:14:17.097882 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9w2d\" (UniqueName: \"kubernetes.io/projected/506f90d2-dfb4-4869-84d9-d7f3c6726c2f-kube-api-access-g9w2d\") pod \"kuadrant-operator-controller-manager-678d87766f-45kbd\" (UID: \"506f90d2-dfb4-4869-84d9-d7f3c6726c2f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-45kbd" Nov 02 07:14:17.198479 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:14:17.198456 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-45kbd" Nov 02 07:14:17.670231 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:14:17.670201 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-45kbd"] Nov 02 07:14:17.678268 ip-10-0-130-93 kubenswrapper[5916]: W1102 07:14:17.678243 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod506f90d2_dfb4_4869_84d9_d7f3c6726c2f.slice/crio-28680c8b3d325750176226669d474880e204eea96e7cee7b387ff0d56bf205ba WatchSource:0}: Error finding container 28680c8b3d325750176226669d474880e204eea96e7cee7b387ff0d56bf205ba: Status 404 returned error can't find the container with id 28680c8b3d325750176226669d474880e204eea96e7cee7b387ff0d56bf205ba Nov 02 07:14:17.794416 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:14:17.794382 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-45kbd" event={"ID":"506f90d2-dfb4-4869-84d9-d7f3c6726c2f","Type":"ContainerStarted","Data":"251053277d394a9366ab6e1a4ce79f8bc4edda735bc114744e108b2ce0e8c9e2"} Nov 02 07:14:17.794567 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:14:17.794545 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-45kbd" Nov 02 07:14:17.794646 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:14:17.794604 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-45kbd" event={"ID":"506f90d2-dfb4-4869-84d9-d7f3c6726c2f","Type":"ContainerStarted","Data":"28680c8b3d325750176226669d474880e204eea96e7cee7b387ff0d56bf205ba"} Nov 02 07:14:17.909996 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:14:17.909948 5916 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-45kbd" podStartSLOduration=1.909935908 podStartE2EDuration="1.909935908s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:14:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:14:17.906707975 +0000 UTC m=+3446.178528368" watchObservedRunningTime="2026-11-02 07:14:17.909935908 +0000 UTC m=+3446.181756296" Nov 02 07:14:19.892643 ip-10-0-130-93 kubenswrapper[5916]: E1102 07:14:19.892602 5916 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p98cq" containerName="extractor" Nov 02 07:14:28.802289 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:14:28.802259 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-45kbd" Nov 02 07:14:49.892881 ip-10-0-130-93 kubenswrapper[5916]: E1102 07:14:49.892845 5916 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p98cq" containerName="extractor" Nov 02 07:15:11.173492 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:15:11.173457 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-7p7b6"] Nov 02 07:15:11.177027 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:15:11.177005 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-7p7b6" Nov 02 07:15:11.230396 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:15:11.230367 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-7p7b6"] Nov 02 07:15:11.273264 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:15:11.273235 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 07:15:11.273378 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:15:11.273277 5916 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 07:15:11.273423 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:15:11.273402 5916 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 07:15:11.273510 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:15:11.273497 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-68l4r\"" Nov 02 07:15:11.281864 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:15:11.281840 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ms98\" (UniqueName: \"kubernetes.io/projected/b70907f2-71ed-47aa-a287-1edc5971490c-kube-api-access-7ms98\") pod \"maas-api-revoke-sub-d2bc93bc-7p7b6\" (UID: \"b70907f2-71ed-47aa-a287-1edc5971490c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-7p7b6" Nov 02 07:15:11.281966 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:15:11.281875 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b70907f2-71ed-47aa-a287-1edc5971490c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-7p7b6\" (UID: \"b70907f2-71ed-47aa-a287-1edc5971490c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-7p7b6" Nov 02 07:15:11.282047 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:15:11.282029 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b70907f2-71ed-47aa-a287-1edc5971490c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-7p7b6\" (UID: \"b70907f2-71ed-47aa-a287-1edc5971490c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-7p7b6" Nov 02 07:15:11.383242 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:15:11.383212 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b70907f2-71ed-47aa-a287-1edc5971490c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-7p7b6\" (UID: \"b70907f2-71ed-47aa-a287-1edc5971490c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-7p7b6" Nov 02 07:15:11.383356 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:15:11.383270 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ms98\" (UniqueName: \"kubernetes.io/projected/b70907f2-71ed-47aa-a287-1edc5971490c-kube-api-access-7ms98\") pod \"maas-api-revoke-sub-d2bc93bc-7p7b6\" (UID: \"b70907f2-71ed-47aa-a287-1edc5971490c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-7p7b6" Nov 02 07:15:11.383356 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:15:11.383294 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b70907f2-71ed-47aa-a287-1edc5971490c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-7p7b6\" (UID: \"b70907f2-71ed-47aa-a287-1edc5971490c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-7p7b6" Nov 02 07:15:11.383932 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:15:11.383891 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b70907f2-71ed-47aa-a287-1edc5971490c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-7p7b6\" (UID: \"b70907f2-71ed-47aa-a287-1edc5971490c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-7p7b6" Nov 02 07:15:11.385655 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:15:11.385628 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b70907f2-71ed-47aa-a287-1edc5971490c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-7p7b6\" (UID: \"b70907f2-71ed-47aa-a287-1edc5971490c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-7p7b6" Nov 02 07:15:11.436620 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:15:11.436567 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ms98\" (UniqueName: \"kubernetes.io/projected/b70907f2-71ed-47aa-a287-1edc5971490c-kube-api-access-7ms98\") pod \"maas-api-revoke-sub-d2bc93bc-7p7b6\" (UID: \"b70907f2-71ed-47aa-a287-1edc5971490c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-7p7b6" Nov 02 07:15:11.489835 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:15:11.489815 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-7p7b6" Nov 02 07:15:11.863523 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:15:11.863449 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-7p7b6"] Nov 02 07:15:11.869796 ip-10-0-130-93 kubenswrapper[5916]: W1102 07:15:11.869768 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb70907f2_71ed_47aa_a287_1edc5971490c.slice/crio-e5c1faa6ea14c1fe064e03fe810d1d9a264916fc612d0ddfee5ecde4d9c3372e WatchSource:0}: Error finding container e5c1faa6ea14c1fe064e03fe810d1d9a264916fc612d0ddfee5ecde4d9c3372e: Status 404 returned error can't find the container with id e5c1faa6ea14c1fe064e03fe810d1d9a264916fc612d0ddfee5ecde4d9c3372e Nov 02 07:15:12.127888 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:15:12.127828 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-7p7b6" event={"ID":"b70907f2-71ed-47aa-a287-1edc5971490c","Type":"ContainerStarted","Data":"9a657263ee41e5b1c2ee2ae42fa441a18dee1a4a4d478d966398342dab332484"} Nov 02 07:15:12.127888 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:15:12.127880 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-7p7b6" event={"ID":"b70907f2-71ed-47aa-a287-1edc5971490c","Type":"ContainerStarted","Data":"e5c1faa6ea14c1fe064e03fe810d1d9a264916fc612d0ddfee5ecde4d9c3372e"} Nov 02 07:15:12.234242 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:15:12.234195 5916 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-7p7b6" podStartSLOduration=1.234180582 podStartE2EDuration="1.234180582s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:15:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:15:12.233704742 +0000 UTC m=+3500.505525137" watchObservedRunningTime="2026-11-02 07:15:12.234180582 +0000 UTC m=+3500.506000977" Nov 02 07:15:13.134638 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:15:13.134596 5916 generic.go:356] "Generic (PLEG): container finished" podID="b70907f2-71ed-47aa-a287-1edc5971490c" containerID="9a657263ee41e5b1c2ee2ae42fa441a18dee1a4a4d478d966398342dab332484" exitCode=0 Nov 02 07:15:13.134805 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:15:13.134699 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-7p7b6" event={"ID":"b70907f2-71ed-47aa-a287-1edc5971490c","Type":"ContainerDied","Data":"9a657263ee41e5b1c2ee2ae42fa441a18dee1a4a4d478d966398342dab332484"} Nov 02 07:15:14.268398 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:15:14.268374 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-7p7b6" Nov 02 07:15:14.304794 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:15:14.304775 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b70907f2-71ed-47aa-a287-1edc5971490c-kube-api-access-7ms98\" (UniqueName: \"kubernetes.io/projected/b70907f2-71ed-47aa-a287-1edc5971490c-kube-api-access-7ms98\") pod \"b70907f2-71ed-47aa-a287-1edc5971490c\" (UID: \"b70907f2-71ed-47aa-a287-1edc5971490c\") " Nov 02 07:15:14.304934 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:15:14.304875 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b70907f2-71ed-47aa-a287-1edc5971490c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b70907f2-71ed-47aa-a287-1edc5971490c-maas-api-xks-ca\") pod \"b70907f2-71ed-47aa-a287-1edc5971490c\" (UID: \"b70907f2-71ed-47aa-a287-1edc5971490c\") " Nov 02 07:15:14.304980 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:15:14.304949 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b70907f2-71ed-47aa-a287-1edc5971490c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b70907f2-71ed-47aa-a287-1edc5971490c-maas-api-openshift-ca\") pod \"b70907f2-71ed-47aa-a287-1edc5971490c\" (UID: \"b70907f2-71ed-47aa-a287-1edc5971490c\") " Nov 02 07:15:14.305245 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:15:14.305220 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b70907f2-71ed-47aa-a287-1edc5971490c-maas-api-openshift-ca" pod "b70907f2-71ed-47aa-a287-1edc5971490c" (UID: "b70907f2-71ed-47aa-a287-1edc5971490c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:15:14.306889 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:15:14.306869 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b70907f2-71ed-47aa-a287-1edc5971490c-kube-api-access-7ms98" pod "b70907f2-71ed-47aa-a287-1edc5971490c" (UID: "b70907f2-71ed-47aa-a287-1edc5971490c"). InnerVolumeSpecName "kube-api-access-7ms98". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:15:14.306978 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:15:14.306876 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b70907f2-71ed-47aa-a287-1edc5971490c-maas-api-xks-ca" pod "b70907f2-71ed-47aa-a287-1edc5971490c" (UID: "b70907f2-71ed-47aa-a287-1edc5971490c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 07:15:14.406151 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:15:14.406075 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ms98\" (UniqueName: \"kubernetes.io/projected/b70907f2-71ed-47aa-a287-1edc5971490c-kube-api-access-7ms98\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:15:14.406151 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:15:14.406102 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b70907f2-71ed-47aa-a287-1edc5971490c-maas-api-xks-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:15:14.406151 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:15:14.406111 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b70907f2-71ed-47aa-a287-1edc5971490c-maas-api-openshift-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:15:15.153896 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:15:15.153862 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-7p7b6" Nov 02 07:15:15.153896 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:15:15.153886 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-7p7b6" event={"ID":"b70907f2-71ed-47aa-a287-1edc5971490c","Type":"ContainerDied","Data":"e5c1faa6ea14c1fe064e03fe810d1d9a264916fc612d0ddfee5ecde4d9c3372e"} Nov 02 07:15:15.154123 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:15:15.153973 5916 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e5c1faa6ea14c1fe064e03fe810d1d9a264916fc612d0ddfee5ecde4d9c3372e" Nov 02 07:15:19.894419 ip-10-0-130-93 kubenswrapper[5916]: E1102 07:15:19.894382 5916 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p98cq" containerName="extractor" Nov 02 07:15:49.893073 ip-10-0-130-93 kubenswrapper[5916]: E1102 07:15:49.893037 5916 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p98cq" containerName="extractor" Nov 02 07:16:19.892626 ip-10-0-130-93 kubenswrapper[5916]: E1102 07:16:19.892589 5916 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p98cq" containerName="extractor" Nov 02 07:16:42.347680 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:16:42.347643 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fbb9071e-7wtl8"] Nov 02 07:16:42.348126 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:16:42.348065 5916 cpu_manager.go:414] "containerMap: removing container" podUID="b70907f2-71ed-47aa-a287-1edc5971490c" containerName="revoke-keys" Nov 02 07:16:42.348126 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:16:42.348079 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b70907f2-71ed-47aa-a287-1edc5971490c" containerName="revoke-keys" Nov 02 07:16:42.348206 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:16:42.348160 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="b70907f2-71ed-47aa-a287-1edc5971490c" containerName="revoke-keys" Nov 02 07:16:42.351408 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:16:42.351393 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbb9071e-7wtl8" Nov 02 07:16:42.405250 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:16:42.405224 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fbb9071e-7wtl8"] Nov 02 07:16:42.448099 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:16:42.448072 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-68l4r\"" Nov 02 07:16:42.448099 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:16:42.448086 5916 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 07:16:42.448275 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:16:42.448092 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 07:16:42.448275 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:16:42.448153 5916 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 07:16:42.463210 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:16:42.463180 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3eab1364-3549-471b-964c-92a02b1caa85-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fbb9071e-7wtl8\" (UID: \"3eab1364-3549-471b-964c-92a02b1caa85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbb9071e-7wtl8" Nov 02 07:16:42.463328 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:16:42.463226 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3eab1364-3549-471b-964c-92a02b1caa85-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fbb9071e-7wtl8\" (UID: \"3eab1364-3549-471b-964c-92a02b1caa85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbb9071e-7wtl8" Nov 02 07:16:42.463389 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:16:42.463324 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5cnk\" (UniqueName: \"kubernetes.io/projected/3eab1364-3549-471b-964c-92a02b1caa85-kube-api-access-c5cnk\") pod \"maas-api-revoke-sub-fbb9071e-7wtl8\" (UID: \"3eab1364-3549-471b-964c-92a02b1caa85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbb9071e-7wtl8" Nov 02 07:16:42.563710 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:16:42.563686 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3eab1364-3549-471b-964c-92a02b1caa85-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fbb9071e-7wtl8\" (UID: \"3eab1364-3549-471b-964c-92a02b1caa85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbb9071e-7wtl8" Nov 02 07:16:42.563814 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:16:42.563717 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3eab1364-3549-471b-964c-92a02b1caa85-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fbb9071e-7wtl8\" (UID: \"3eab1364-3549-471b-964c-92a02b1caa85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbb9071e-7wtl8" Nov 02 07:16:42.563814 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:16:42.563736 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5cnk\" (UniqueName: \"kubernetes.io/projected/3eab1364-3549-471b-964c-92a02b1caa85-kube-api-access-c5cnk\") pod \"maas-api-revoke-sub-fbb9071e-7wtl8\" (UID: \"3eab1364-3549-471b-964c-92a02b1caa85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbb9071e-7wtl8" Nov 02 07:16:42.564367 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:16:42.564347 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3eab1364-3549-471b-964c-92a02b1caa85-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fbb9071e-7wtl8\" (UID: \"3eab1364-3549-471b-964c-92a02b1caa85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbb9071e-7wtl8" Nov 02 07:16:42.565950 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:16:42.565934 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3eab1364-3549-471b-964c-92a02b1caa85-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fbb9071e-7wtl8\" (UID: \"3eab1364-3549-471b-964c-92a02b1caa85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbb9071e-7wtl8" Nov 02 07:16:42.616217 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:16:42.616168 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5cnk\" (UniqueName: \"kubernetes.io/projected/3eab1364-3549-471b-964c-92a02b1caa85-kube-api-access-c5cnk\") pod \"maas-api-revoke-sub-fbb9071e-7wtl8\" (UID: \"3eab1364-3549-471b-964c-92a02b1caa85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbb9071e-7wtl8" Nov 02 07:16:42.664577 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:16:42.664554 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbb9071e-7wtl8" Nov 02 07:16:43.030524 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:16:43.030491 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fbb9071e-7wtl8"] Nov 02 07:16:43.036666 ip-10-0-130-93 kubenswrapper[5916]: W1102 07:16:43.036640 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3eab1364_3549_471b_964c_92a02b1caa85.slice/crio-0ebdb276161c0544ea3f8bd0f38612ef0bc8a1e15d6ac8c64603517399e5e389 WatchSource:0}: Error finding container 0ebdb276161c0544ea3f8bd0f38612ef0bc8a1e15d6ac8c64603517399e5e389: Status 404 returned error can't find the container with id 0ebdb276161c0544ea3f8bd0f38612ef0bc8a1e15d6ac8c64603517399e5e389 Nov 02 07:16:43.642515 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:16:43.642476 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbb9071e-7wtl8" event={"ID":"3eab1364-3549-471b-964c-92a02b1caa85","Type":"ContainerStarted","Data":"0d8f5986c3667d8625bc4bb141f2329bcaebb8fbd4051907d936a8da98d87ce9"} Nov 02 07:16:43.642939 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:16:43.642541 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbb9071e-7wtl8" event={"ID":"3eab1364-3549-471b-964c-92a02b1caa85","Type":"ContainerStarted","Data":"0ebdb276161c0544ea3f8bd0f38612ef0bc8a1e15d6ac8c64603517399e5e389"} Nov 02 07:16:43.748668 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:16:43.748617 5916 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbb9071e-7wtl8" podStartSLOduration=1.748603675 podStartE2EDuration="1.748603675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:16:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:16:43.747434811 +0000 UTC m=+3592.019255205" watchObservedRunningTime="2026-11-02 07:16:43.748603675 +0000 UTC m=+3592.020424068" Nov 02 07:16:44.652597 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:16:44.652560 5916 generic.go:356] "Generic (PLEG): container finished" podID="3eab1364-3549-471b-964c-92a02b1caa85" containerID="0d8f5986c3667d8625bc4bb141f2329bcaebb8fbd4051907d936a8da98d87ce9" exitCode=0 Nov 02 07:16:44.653017 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:16:44.652648 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbb9071e-7wtl8" event={"ID":"3eab1364-3549-471b-964c-92a02b1caa85","Type":"ContainerDied","Data":"0d8f5986c3667d8625bc4bb141f2329bcaebb8fbd4051907d936a8da98d87ce9"} Nov 02 07:16:45.799615 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:16:45.799591 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbb9071e-7wtl8" Nov 02 07:16:45.897882 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:16:45.897856 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3eab1364-3549-471b-964c-92a02b1caa85-kube-api-access-c5cnk\" (UniqueName: \"kubernetes.io/projected/3eab1364-3549-471b-964c-92a02b1caa85-kube-api-access-c5cnk\") pod \"3eab1364-3549-471b-964c-92a02b1caa85\" (UID: \"3eab1364-3549-471b-964c-92a02b1caa85\") " Nov 02 07:16:45.898049 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:16:45.897894 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3eab1364-3549-471b-964c-92a02b1caa85-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3eab1364-3549-471b-964c-92a02b1caa85-maas-api-openshift-ca\") pod \"3eab1364-3549-471b-964c-92a02b1caa85\" (UID: \"3eab1364-3549-471b-964c-92a02b1caa85\") " Nov 02 07:16:45.898049 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:16:45.897958 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3eab1364-3549-471b-964c-92a02b1caa85-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3eab1364-3549-471b-964c-92a02b1caa85-maas-api-xks-ca\") pod \"3eab1364-3549-471b-964c-92a02b1caa85\" (UID: \"3eab1364-3549-471b-964c-92a02b1caa85\") " Nov 02 07:16:45.898263 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:16:45.898228 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3eab1364-3549-471b-964c-92a02b1caa85-maas-api-openshift-ca" pod "3eab1364-3549-471b-964c-92a02b1caa85" (UID: "3eab1364-3549-471b-964c-92a02b1caa85"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:16:45.900049 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:16:45.900025 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3eab1364-3549-471b-964c-92a02b1caa85-maas-api-xks-ca" pod "3eab1364-3549-471b-964c-92a02b1caa85" (UID: "3eab1364-3549-471b-964c-92a02b1caa85"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 07:16:45.900138 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:16:45.900071 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3eab1364-3549-471b-964c-92a02b1caa85-kube-api-access-c5cnk" pod "3eab1364-3549-471b-964c-92a02b1caa85" (UID: "3eab1364-3549-471b-964c-92a02b1caa85"). InnerVolumeSpecName "kube-api-access-c5cnk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:16:45.999156 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:16:45.999080 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3eab1364-3549-471b-964c-92a02b1caa85-maas-api-xks-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:16:45.999156 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:16:45.999110 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5cnk\" (UniqueName: \"kubernetes.io/projected/3eab1364-3549-471b-964c-92a02b1caa85-kube-api-access-c5cnk\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:16:45.999156 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:16:45.999125 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3eab1364-3549-471b-964c-92a02b1caa85-maas-api-openshift-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:16:46.665657 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:16:46.665628 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbb9071e-7wtl8" Nov 02 07:16:46.665834 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:16:46.665669 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbb9071e-7wtl8" event={"ID":"3eab1364-3549-471b-964c-92a02b1caa85","Type":"ContainerDied","Data":"0ebdb276161c0544ea3f8bd0f38612ef0bc8a1e15d6ac8c64603517399e5e389"} Nov 02 07:16:46.665834 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:16:46.665722 5916 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0ebdb276161c0544ea3f8bd0f38612ef0bc8a1e15d6ac8c64603517399e5e389" Nov 02 07:16:49.894104 ip-10-0-130-93 kubenswrapper[5916]: E1102 07:16:49.894061 5916 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p98cq" containerName="extractor" Nov 02 07:16:53.400528 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:16:53.400503 5916 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-csfhg_17dee72b-2f47-4016-8ffe-3a89934e66ee/console-operator/2.log" Nov 02 07:16:53.413076 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:16:53.413055 5916 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-csfhg_17dee72b-2f47-4016-8ffe-3a89934e66ee/console-operator/2.log" Nov 02 07:16:53.967363 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:16:53.967330 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-75b25b-67xpk"] Nov 02 07:16:53.967809 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:16:53.967790 5916 cpu_manager.go:414] "containerMap: removing container" podUID="3eab1364-3549-471b-964c-92a02b1caa85" containerName="revoke-keys" Nov 02 07:16:53.967935 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:16:53.967810 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3eab1364-3549-471b-964c-92a02b1caa85" containerName="revoke-keys" Nov 02 07:16:53.968329 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:16:53.967964 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="3eab1364-3549-471b-964c-92a02b1caa85" containerName="revoke-keys" Nov 02 07:16:53.971515 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:16:53.971496 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-75b25b-67xpk" Nov 02 07:16:54.021122 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:16:54.021096 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 07:16:54.024233 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:16:54.024202 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-75b25b-67xpk"] Nov 02 07:16:54.059191 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:16:54.059149 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90cda899-a69b-48e6-86a2-d6a90c4a7d0c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-75b25b-67xpk\" (UID: \"90cda899-a69b-48e6-86a2-d6a90c4a7d0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-75b25b-67xpk" Nov 02 07:16:54.059392 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:16:54.059344 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tg8th\" (UniqueName: \"kubernetes.io/projected/90cda899-a69b-48e6-86a2-d6a90c4a7d0c-kube-api-access-tg8th\") pod \"maas-api-revoke-keys-e2e-worker-single-main-75b25b-67xpk\" (UID: \"90cda899-a69b-48e6-86a2-d6a90c4a7d0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-75b25b-67xpk" Nov 02 07:16:54.059392 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:16:54.059381 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90cda899-a69b-48e6-86a2-d6a90c4a7d0c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-75b25b-67xpk\" (UID: \"90cda899-a69b-48e6-86a2-d6a90c4a7d0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-75b25b-67xpk" Nov 02 07:16:54.066746 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:16:54.066725 5916 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 07:16:54.066835 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:16:54.066785 5916 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 07:16:54.067098 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:16:54.067081 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-68l4r\"" Nov 02 07:16:54.160082 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:16:54.160052 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tg8th\" (UniqueName: \"kubernetes.io/projected/90cda899-a69b-48e6-86a2-d6a90c4a7d0c-kube-api-access-tg8th\") pod \"maas-api-revoke-keys-e2e-worker-single-main-75b25b-67xpk\" (UID: \"90cda899-a69b-48e6-86a2-d6a90c4a7d0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-75b25b-67xpk" Nov 02 07:16:54.160245 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:16:54.160089 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90cda899-a69b-48e6-86a2-d6a90c4a7d0c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-75b25b-67xpk\" (UID: \"90cda899-a69b-48e6-86a2-d6a90c4a7d0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-75b25b-67xpk" Nov 02 07:16:54.160245 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:16:54.160120 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90cda899-a69b-48e6-86a2-d6a90c4a7d0c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-75b25b-67xpk\" (UID: \"90cda899-a69b-48e6-86a2-d6a90c4a7d0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-75b25b-67xpk" Nov 02 07:16:54.160750 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:16:54.160726 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90cda899-a69b-48e6-86a2-d6a90c4a7d0c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-75b25b-67xpk\" (UID: \"90cda899-a69b-48e6-86a2-d6a90c4a7d0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-75b25b-67xpk" Nov 02 07:16:54.162469 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:16:54.162453 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90cda899-a69b-48e6-86a2-d6a90c4a7d0c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-75b25b-67xpk\" (UID: \"90cda899-a69b-48e6-86a2-d6a90c4a7d0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-75b25b-67xpk" Nov 02 07:16:54.213011 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:16:54.212987 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tg8th\" (UniqueName: \"kubernetes.io/projected/90cda899-a69b-48e6-86a2-d6a90c4a7d0c-kube-api-access-tg8th\") pod \"maas-api-revoke-keys-e2e-worker-single-main-75b25b-67xpk\" (UID: \"90cda899-a69b-48e6-86a2-d6a90c4a7d0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-75b25b-67xpk" Nov 02 07:16:54.284220 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:16:54.284152 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-75b25b-67xpk" Nov 02 07:16:54.652976 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:16:54.652947 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-75b25b-67xpk"] Nov 02 07:16:54.659942 ip-10-0-130-93 kubenswrapper[5916]: W1102 07:16:54.659894 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90cda899_a69b_48e6_86a2_d6a90c4a7d0c.slice/crio-517d651d68554a723b8f9989af6d0a24f847c4d6d867a7b9ff879f4143fdd5d6 WatchSource:0}: Error finding container 517d651d68554a723b8f9989af6d0a24f847c4d6d867a7b9ff879f4143fdd5d6: Status 404 returned error can't find the container with id 517d651d68554a723b8f9989af6d0a24f847c4d6d867a7b9ff879f4143fdd5d6 Nov 02 07:16:54.710800 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:16:54.710770 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-75b25b-67xpk" event={"ID":"90cda899-a69b-48e6-86a2-d6a90c4a7d0c","Type":"ContainerStarted","Data":"517d651d68554a723b8f9989af6d0a24f847c4d6d867a7b9ff879f4143fdd5d6"} Nov 02 07:16:55.718349 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:16:55.718269 5916 generic.go:356] "Generic (PLEG): container finished" podID="90cda899-a69b-48e6-86a2-d6a90c4a7d0c" containerID="1444094fc1c8b193e29c02271b185523d87cca8228ebfaff19757e5b0503c602" exitCode=0 Nov 02 07:16:55.718726 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:16:55.718355 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-75b25b-67xpk" event={"ID":"90cda899-a69b-48e6-86a2-d6a90c4a7d0c","Type":"ContainerDied","Data":"1444094fc1c8b193e29c02271b185523d87cca8228ebfaff19757e5b0503c602"} Nov 02 07:16:56.857288 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:16:56.857265 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-75b25b-67xpk" Nov 02 07:16:56.884974 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:16:56.884947 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90cda899-a69b-48e6-86a2-d6a90c4a7d0c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90cda899-a69b-48e6-86a2-d6a90c4a7d0c-maas-api-xks-ca\") pod \"90cda899-a69b-48e6-86a2-d6a90c4a7d0c\" (UID: \"90cda899-a69b-48e6-86a2-d6a90c4a7d0c\") " Nov 02 07:16:56.885084 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:16:56.884994 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90cda899-a69b-48e6-86a2-d6a90c4a7d0c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90cda899-a69b-48e6-86a2-d6a90c4a7d0c-maas-api-openshift-ca\") pod \"90cda899-a69b-48e6-86a2-d6a90c4a7d0c\" (UID: \"90cda899-a69b-48e6-86a2-d6a90c4a7d0c\") " Nov 02 07:16:56.885127 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:16:56.885100 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90cda899-a69b-48e6-86a2-d6a90c4a7d0c-kube-api-access-tg8th\" (UniqueName: \"kubernetes.io/projected/90cda899-a69b-48e6-86a2-d6a90c4a7d0c-kube-api-access-tg8th\") pod \"90cda899-a69b-48e6-86a2-d6a90c4a7d0c\" (UID: \"90cda899-a69b-48e6-86a2-d6a90c4a7d0c\") " Nov 02 07:16:56.885602 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:16:56.885359 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90cda899-a69b-48e6-86a2-d6a90c4a7d0c-maas-api-openshift-ca" pod "90cda899-a69b-48e6-86a2-d6a90c4a7d0c" (UID: "90cda899-a69b-48e6-86a2-d6a90c4a7d0c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:16:56.885602 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:16:56.885454 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90cda899-a69b-48e6-86a2-d6a90c4a7d0c-maas-api-openshift-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:16:56.887098 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:16:56.887078 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90cda899-a69b-48e6-86a2-d6a90c4a7d0c-maas-api-xks-ca" pod "90cda899-a69b-48e6-86a2-d6a90c4a7d0c" (UID: "90cda899-a69b-48e6-86a2-d6a90c4a7d0c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 07:16:56.887170 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:16:56.887156 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90cda899-a69b-48e6-86a2-d6a90c4a7d0c-kube-api-access-tg8th" pod "90cda899-a69b-48e6-86a2-d6a90c4a7d0c" (UID: "90cda899-a69b-48e6-86a2-d6a90c4a7d0c"). InnerVolumeSpecName "kube-api-access-tg8th". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:16:56.986518 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:16:56.986459 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tg8th\" (UniqueName: \"kubernetes.io/projected/90cda899-a69b-48e6-86a2-d6a90c4a7d0c-kube-api-access-tg8th\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:16:56.986518 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:16:56.986485 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90cda899-a69b-48e6-86a2-d6a90c4a7d0c-maas-api-xks-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:16:57.730540 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:16:57.730508 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-75b25b-67xpk" Nov 02 07:16:57.730728 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:16:57.730539 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-75b25b-67xpk" event={"ID":"90cda899-a69b-48e6-86a2-d6a90c4a7d0c","Type":"ContainerDied","Data":"517d651d68554a723b8f9989af6d0a24f847c4d6d867a7b9ff879f4143fdd5d6"} Nov 02 07:16:57.730728 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:16:57.730614 5916 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="517d651d68554a723b8f9989af6d0a24f847c4d6d867a7b9ff879f4143fdd5d6" Nov 02 07:17:02.043883 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:17:02.043807 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ee03e20-scgm4"] Nov 02 07:17:02.089610 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:17:02.089587 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ee03e20-scgm4"] Nov 02 07:17:03.840413 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:17:03.840381 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e7910e1-f0eb-4088-accb-7e59f06626f9" path="/var/lib/kubelet/pods/3e7910e1-f0eb-4088-accb-7e59f06626f9/volumes" Nov 02 07:17:09.772982 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:17:09.772943 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-75b25b-67xpk"] Nov 02 07:17:09.818312 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:17:09.818281 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-75b25b-67xpk"] Nov 02 07:17:09.841243 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:17:09.841216 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90cda899-a69b-48e6-86a2-d6a90c4a7d0c" path="/var/lib/kubelet/pods/90cda899-a69b-48e6-86a2-d6a90c4a7d0c/volumes" Nov 02 07:17:19.892836 ip-10-0-130-93 kubenswrapper[5916]: E1102 07:17:19.892802 5916 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p98cq" containerName="extractor" Nov 02 07:17:49.892928 ip-10-0-130-93 kubenswrapper[5916]: E1102 07:17:49.892879 5916 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p98cq" containerName="extractor" Nov 02 07:18:10.981094 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:18:10.981036 5916 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-px5bb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.222:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 07:18:10.981939 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:18:10.981869 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-px5bb" podUID="6566515d-9e15-4679-b56f-87c483940ba6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.222:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 07:18:10.985094 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:18:10.985051 5916 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-6527w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.221:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 07:18:10.985301 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:18:10.985269 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6527w" podUID="918ad956-7d56-4627-a1a4-e7c75694d1f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.221:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 07:18:11.044001 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:18:11.043971 5916 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-tnptl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 07:18:11.044187 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:18:11.044163 5916 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tnptl" podUID="a24071b2-e468-4590-93ea-1be6ac017cf8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 07:18:19.892693 ip-10-0-130-93 kubenswrapper[5916]: E1102 07:18:19.892658 5916 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p98cq" containerName="extractor" Nov 02 07:18:49.893184 ip-10-0-130-93 kubenswrapper[5916]: E1102 07:18:49.893103 5916 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p98cq" containerName="extractor" Nov 02 07:18:56.166391 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:18:56.166362 5916 scope.go:122] "RemoveContainer" containerID="eb77b61a49a2df10cd047693747517b95e0123c2ac6d8e0c011d2f20c4128454" Nov 02 07:18:56.178235 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:18:56.178216 5916 scope.go:122] "RemoveContainer" containerID="4f28148c448b0c9b1cc67570a2345a5f49ab8d9275fb152a160fecfc1325d9e8" Nov 02 07:19:19.892924 ip-10-0-130-93 kubenswrapper[5916]: E1102 07:19:19.892862 5916 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p98cq" containerName="extractor" Nov 02 07:19:49.894443 ip-10-0-130-93 kubenswrapper[5916]: E1102 07:19:49.894407 5916 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p98cq" containerName="extractor" Nov 02 07:20:15.043284 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:20:15.043243 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-7p7b6"] Nov 02 07:20:15.088667 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:20:15.088641 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-7p7b6"] Nov 02 07:20:15.840828 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:20:15.840792 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b70907f2-71ed-47aa-a287-1edc5971490c" path="/var/lib/kubelet/pods/b70907f2-71ed-47aa-a287-1edc5971490c/volumes" Nov 02 07:20:19.892963 ip-10-0-130-93 kubenswrapper[5916]: E1102 07:20:19.892929 5916 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p98cq" containerName="extractor" Nov 02 07:20:30.780851 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:20:30.780816 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e3c0bf7-r9hvs"] Nov 02 07:20:30.781316 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:20:30.781207 5916 cpu_manager.go:414] "containerMap: removing container" podUID="90cda899-a69b-48e6-86a2-d6a90c4a7d0c" containerName="revoke-keys" Nov 02 07:20:30.781316 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:20:30.781218 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90cda899-a69b-48e6-86a2-d6a90c4a7d0c" containerName="revoke-keys" Nov 02 07:20:30.781316 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:20:30.781311 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="90cda899-a69b-48e6-86a2-d6a90c4a7d0c" containerName="revoke-keys" Nov 02 07:20:30.784515 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:20:30.784498 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e3c0bf7-r9hvs" Nov 02 07:20:30.837319 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:20:30.837296 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e3c0bf7-r9hvs"] Nov 02 07:20:30.848894 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:20:30.848872 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7k5d\" (UniqueName: \"kubernetes.io/projected/1c26839c-9f89-43f9-85cb-d611c6fd4f7a-kube-api-access-d7k5d\") pod \"maas-api-revoke-keys-e2e-ait-0e3c0bf7-r9hvs\" (UID: \"1c26839c-9f89-43f9-85cb-d611c6fd4f7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e3c0bf7-r9hvs" Nov 02 07:20:30.849022 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:20:30.848984 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c26839c-9f89-43f9-85cb-d611c6fd4f7a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-0e3c0bf7-r9hvs\" (UID: \"1c26839c-9f89-43f9-85cb-d611c6fd4f7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e3c0bf7-r9hvs" Nov 02 07:20:30.849075 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:20:30.849060 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c26839c-9f89-43f9-85cb-d611c6fd4f7a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-0e3c0bf7-r9hvs\" (UID: \"1c26839c-9f89-43f9-85cb-d611c6fd4f7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e3c0bf7-r9hvs" Nov 02 07:20:30.880709 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:20:30.880686 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nfbkd\"" Nov 02 07:20:30.880813 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:20:30.880719 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 07:20:30.880813 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:20:30.880721 5916 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 07:20:30.880813 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:20:30.880758 5916 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 07:20:30.949974 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:20:30.949953 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c26839c-9f89-43f9-85cb-d611c6fd4f7a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-0e3c0bf7-r9hvs\" (UID: \"1c26839c-9f89-43f9-85cb-d611c6fd4f7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e3c0bf7-r9hvs" Nov 02 07:20:30.950087 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:20:30.950003 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c26839c-9f89-43f9-85cb-d611c6fd4f7a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-0e3c0bf7-r9hvs\" (UID: \"1c26839c-9f89-43f9-85cb-d611c6fd4f7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e3c0bf7-r9hvs" Nov 02 07:20:30.950194 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:20:30.950175 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7k5d\" (UniqueName: \"kubernetes.io/projected/1c26839c-9f89-43f9-85cb-d611c6fd4f7a-kube-api-access-d7k5d\") pod \"maas-api-revoke-keys-e2e-ait-0e3c0bf7-r9hvs\" (UID: \"1c26839c-9f89-43f9-85cb-d611c6fd4f7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e3c0bf7-r9hvs" Nov 02 07:20:30.950622 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:20:30.950596 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c26839c-9f89-43f9-85cb-d611c6fd4f7a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-0e3c0bf7-r9hvs\" (UID: \"1c26839c-9f89-43f9-85cb-d611c6fd4f7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e3c0bf7-r9hvs" Nov 02 07:20:30.952270 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:20:30.952253 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c26839c-9f89-43f9-85cb-d611c6fd4f7a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-0e3c0bf7-r9hvs\" (UID: \"1c26839c-9f89-43f9-85cb-d611c6fd4f7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e3c0bf7-r9hvs" Nov 02 07:20:31.002883 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:20:31.002861 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7k5d\" (UniqueName: \"kubernetes.io/projected/1c26839c-9f89-43f9-85cb-d611c6fd4f7a-kube-api-access-d7k5d\") pod \"maas-api-revoke-keys-e2e-ait-0e3c0bf7-r9hvs\" (UID: \"1c26839c-9f89-43f9-85cb-d611c6fd4f7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e3c0bf7-r9hvs" Nov 02 07:20:31.097490 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:20:31.097416 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e3c0bf7-r9hvs" Nov 02 07:20:31.461619 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:20:31.461583 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e3c0bf7-r9hvs"] Nov 02 07:20:31.468935 ip-10-0-130-93 kubenswrapper[5916]: W1102 07:20:31.468889 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c26839c_9f89_43f9_85cb_d611c6fd4f7a.slice/crio-b822dd13367e55327394e520d9c7b5496d22a03596f21f6a4689d242b7fc204b WatchSource:0}: Error finding container b822dd13367e55327394e520d9c7b5496d22a03596f21f6a4689d242b7fc204b: Status 404 returned error can't find the container with id b822dd13367e55327394e520d9c7b5496d22a03596f21f6a4689d242b7fc204b Nov 02 07:20:32.095970 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:20:32.095936 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e3c0bf7-r9hvs" event={"ID":"1c26839c-9f89-43f9-85cb-d611c6fd4f7a","Type":"ContainerStarted","Data":"b641d2d64c90526a445aff514cfbee25af193c9d07af5769ce274389292a796c"} Nov 02 07:20:32.096349 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:20:32.095986 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e3c0bf7-r9hvs" event={"ID":"1c26839c-9f89-43f9-85cb-d611c6fd4f7a","Type":"ContainerStarted","Data":"b822dd13367e55327394e520d9c7b5496d22a03596f21f6a4689d242b7fc204b"} Nov 02 07:20:33.103607 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:20:33.103570 5916 generic.go:356] "Generic (PLEG): container finished" podID="1c26839c-9f89-43f9-85cb-d611c6fd4f7a" containerID="b641d2d64c90526a445aff514cfbee25af193c9d07af5769ce274389292a796c" exitCode=0 Nov 02 07:20:33.104145 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:20:33.103658 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e3c0bf7-r9hvs" event={"ID":"1c26839c-9f89-43f9-85cb-d611c6fd4f7a","Type":"ContainerDied","Data":"b641d2d64c90526a445aff514cfbee25af193c9d07af5769ce274389292a796c"} Nov 02 07:20:34.234317 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:20:34.234290 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e3c0bf7-r9hvs" Nov 02 07:20:34.278655 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:20:34.278631 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c26839c-9f89-43f9-85cb-d611c6fd4f7a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c26839c-9f89-43f9-85cb-d611c6fd4f7a-maas-api-openshift-ca\") pod \"1c26839c-9f89-43f9-85cb-d611c6fd4f7a\" (UID: \"1c26839c-9f89-43f9-85cb-d611c6fd4f7a\") " Nov 02 07:20:34.278783 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:20:34.278766 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c26839c-9f89-43f9-85cb-d611c6fd4f7a-kube-api-access-d7k5d\" (UniqueName: \"kubernetes.io/projected/1c26839c-9f89-43f9-85cb-d611c6fd4f7a-kube-api-access-d7k5d\") pod \"1c26839c-9f89-43f9-85cb-d611c6fd4f7a\" (UID: \"1c26839c-9f89-43f9-85cb-d611c6fd4f7a\") " Nov 02 07:20:34.278836 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:20:34.278822 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c26839c-9f89-43f9-85cb-d611c6fd4f7a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c26839c-9f89-43f9-85cb-d611c6fd4f7a-maas-api-xks-ca\") pod \"1c26839c-9f89-43f9-85cb-d611c6fd4f7a\" (UID: \"1c26839c-9f89-43f9-85cb-d611c6fd4f7a\") " Nov 02 07:20:34.279039 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:20:34.279018 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c26839c-9f89-43f9-85cb-d611c6fd4f7a-maas-api-openshift-ca" pod "1c26839c-9f89-43f9-85cb-d611c6fd4f7a" (UID: "1c26839c-9f89-43f9-85cb-d611c6fd4f7a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:20:34.279173 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:20:34.279153 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c26839c-9f89-43f9-85cb-d611c6fd4f7a-maas-api-openshift-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:20:34.280703 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:20:34.280676 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c26839c-9f89-43f9-85cb-d611c6fd4f7a-kube-api-access-d7k5d" pod "1c26839c-9f89-43f9-85cb-d611c6fd4f7a" (UID: "1c26839c-9f89-43f9-85cb-d611c6fd4f7a"). InnerVolumeSpecName "kube-api-access-d7k5d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:20:34.280784 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:20:34.280750 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c26839c-9f89-43f9-85cb-d611c6fd4f7a-maas-api-xks-ca" pod "1c26839c-9f89-43f9-85cb-d611c6fd4f7a" (UID: "1c26839c-9f89-43f9-85cb-d611c6fd4f7a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 07:20:34.379933 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:20:34.379849 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7k5d\" (UniqueName: \"kubernetes.io/projected/1c26839c-9f89-43f9-85cb-d611c6fd4f7a-kube-api-access-d7k5d\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:20:34.379933 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:20:34.379875 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c26839c-9f89-43f9-85cb-d611c6fd4f7a-maas-api-xks-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:20:35.116409 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:20:35.116377 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e3c0bf7-r9hvs" Nov 02 07:20:35.116633 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:20:35.116415 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e3c0bf7-r9hvs" event={"ID":"1c26839c-9f89-43f9-85cb-d611c6fd4f7a","Type":"ContainerDied","Data":"b822dd13367e55327394e520d9c7b5496d22a03596f21f6a4689d242b7fc204b"} Nov 02 07:20:35.116633 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:20:35.116462 5916 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b822dd13367e55327394e520d9c7b5496d22a03596f21f6a4689d242b7fc204b" Nov 02 07:20:49.893747 ip-10-0-130-93 kubenswrapper[5916]: E1102 07:20:49.893707 5916 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p98cq" containerName="extractor" Nov 02 07:20:50.065697 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:20:50.065665 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e3c0bf7-r9hvs"] Nov 02 07:20:50.112622 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:20:50.112591 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e3c0bf7-r9hvs"] Nov 02 07:20:51.841638 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:20:51.841604 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c26839c-9f89-43f9-85cb-d611c6fd4f7a" path="/var/lib/kubelet/pods/1c26839c-9f89-43f9-85cb-d611c6fd4f7a/volumes" Nov 02 07:21:19.893803 ip-10-0-130-93 kubenswrapper[5916]: E1102 07:21:19.893696 5916 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p98cq" containerName="extractor" Nov 02 07:21:46.043095 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:21:46.043019 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fbb9071e-7wtl8"] Nov 02 07:21:46.088939 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:21:46.088897 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fbb9071e-7wtl8"] Nov 02 07:21:47.840612 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:21:47.840581 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3eab1364-3549-471b-964c-92a02b1caa85" path="/var/lib/kubelet/pods/3eab1364-3549-471b-964c-92a02b1caa85/volumes" Nov 02 07:21:49.892387 ip-10-0-130-93 kubenswrapper[5916]: E1102 07:21:49.892344 5916 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p98cq" containerName="extractor" Nov 02 07:21:53.448005 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:21:53.447968 5916 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-csfhg_17dee72b-2f47-4016-8ffe-3a89934e66ee/console-operator/2.log" Nov 02 07:21:53.467830 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:21:53.467807 5916 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-csfhg_17dee72b-2f47-4016-8ffe-3a89934e66ee/console-operator/2.log" Nov 02 07:21:56.224355 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:21:56.224330 5916 scope.go:122] "RemoveContainer" containerID="9a657263ee41e5b1c2ee2ae42fa441a18dee1a4a4d478d966398342dab332484" Nov 02 07:22:19.893217 ip-10-0-130-93 kubenswrapper[5916]: E1102 07:22:19.893182 5916 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p98cq" containerName="extractor" Nov 02 07:22:43.456140 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:22:43.456104 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-c0d3ab-jsd7h"] Nov 02 07:22:43.456640 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:22:43.456505 5916 cpu_manager.go:414] "containerMap: removing container" podUID="1c26839c-9f89-43f9-85cb-d611c6fd4f7a" containerName="revoke-keys" Nov 02 07:22:43.456640 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:22:43.456517 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c26839c-9f89-43f9-85cb-d611c6fd4f7a" containerName="revoke-keys" Nov 02 07:22:43.456640 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:22:43.456602 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c26839c-9f89-43f9-85cb-d611c6fd4f7a" containerName="revoke-keys" Nov 02 07:22:43.459806 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:22:43.459787 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-c0d3ab-jsd7h" Nov 02 07:22:43.509807 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:22:43.509782 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 07:22:43.555962 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:22:43.555935 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nfbkd\"" Nov 02 07:22:43.555962 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:22:43.555952 5916 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 07:22:43.555962 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:22:43.555948 5916 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 07:22:43.558227 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:22:43.558207 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-c0d3ab-jsd7h"] Nov 02 07:22:43.591947 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:22:43.591901 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2055f0e8-200a-4689-997f-588363463659-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-c0d3ab-jsd7h\" (UID: \"2055f0e8-200a-4689-997f-588363463659\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-c0d3ab-jsd7h" Nov 02 07:22:43.592061 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:22:43.591957 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chct4\" (UniqueName: \"kubernetes.io/projected/2055f0e8-200a-4689-997f-588363463659-kube-api-access-chct4\") pod \"maas-api-revoke-keys-e2e-authcfg-http-c0d3ab-jsd7h\" (UID: \"2055f0e8-200a-4689-997f-588363463659\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-c0d3ab-jsd7h" Nov 02 07:22:43.592061 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:22:43.592034 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2055f0e8-200a-4689-997f-588363463659-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-c0d3ab-jsd7h\" (UID: \"2055f0e8-200a-4689-997f-588363463659\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-c0d3ab-jsd7h" Nov 02 07:22:43.692955 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:22:43.692929 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2055f0e8-200a-4689-997f-588363463659-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-c0d3ab-jsd7h\" (UID: \"2055f0e8-200a-4689-997f-588363463659\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-c0d3ab-jsd7h" Nov 02 07:22:43.693106 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:22:43.692967 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chct4\" (UniqueName: \"kubernetes.io/projected/2055f0e8-200a-4689-997f-588363463659-kube-api-access-chct4\") pod \"maas-api-revoke-keys-e2e-authcfg-http-c0d3ab-jsd7h\" (UID: \"2055f0e8-200a-4689-997f-588363463659\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-c0d3ab-jsd7h" Nov 02 07:22:43.693106 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:22:43.693084 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2055f0e8-200a-4689-997f-588363463659-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-c0d3ab-jsd7h\" (UID: \"2055f0e8-200a-4689-997f-588363463659\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-c0d3ab-jsd7h" Nov 02 07:22:43.693681 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:22:43.693663 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2055f0e8-200a-4689-997f-588363463659-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-c0d3ab-jsd7h\" (UID: \"2055f0e8-200a-4689-997f-588363463659\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-c0d3ab-jsd7h" Nov 02 07:22:43.695070 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:22:43.695053 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2055f0e8-200a-4689-997f-588363463659-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-c0d3ab-jsd7h\" (UID: \"2055f0e8-200a-4689-997f-588363463659\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-c0d3ab-jsd7h" Nov 02 07:22:43.745350 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:22:43.745291 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chct4\" (UniqueName: \"kubernetes.io/projected/2055f0e8-200a-4689-997f-588363463659-kube-api-access-chct4\") pod \"maas-api-revoke-keys-e2e-authcfg-http-c0d3ab-jsd7h\" (UID: \"2055f0e8-200a-4689-997f-588363463659\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-c0d3ab-jsd7h" Nov 02 07:22:43.771969 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:22:43.771950 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-c0d3ab-jsd7h" Nov 02 07:22:44.141491 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:22:44.141464 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-c0d3ab-jsd7h"] Nov 02 07:22:44.148165 ip-10-0-130-93 kubenswrapper[5916]: W1102 07:22:44.148131 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2055f0e8_200a_4689_997f_588363463659.slice/crio-a0a41a193627c7464478f3e282f637c85c3c1f165b03e025496e4b69932127fb WatchSource:0}: Error finding container a0a41a193627c7464478f3e282f637c85c3c1f165b03e025496e4b69932127fb: Status 404 returned error can't find the container with id a0a41a193627c7464478f3e282f637c85c3c1f165b03e025496e4b69932127fb Nov 02 07:22:44.818943 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:22:44.818888 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-c0d3ab-jsd7h" event={"ID":"2055f0e8-200a-4689-997f-588363463659","Type":"ContainerStarted","Data":"0b2b3027e228baa9111278fe37dfa6e1f61cec9bf0ea33ad4d0c1bfc1ffac1a2"} Nov 02 07:22:44.819359 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:22:44.818989 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-c0d3ab-jsd7h" event={"ID":"2055f0e8-200a-4689-997f-588363463659","Type":"ContainerStarted","Data":"a0a41a193627c7464478f3e282f637c85c3c1f165b03e025496e4b69932127fb"} Nov 02 07:22:44.925168 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:22:44.925116 5916 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-c0d3ab-jsd7h" podStartSLOduration=1.925101934 podStartE2EDuration="1.925101934s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:22:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:22:44.922746359 +0000 UTC m=+3953.194566752" watchObservedRunningTime="2026-11-02 07:22:44.925101934 +0000 UTC m=+3953.196922328" Nov 02 07:22:45.825244 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:22:45.825211 5916 generic.go:356] "Generic (PLEG): container finished" podID="2055f0e8-200a-4689-997f-588363463659" containerID="0b2b3027e228baa9111278fe37dfa6e1f61cec9bf0ea33ad4d0c1bfc1ffac1a2" exitCode=0 Nov 02 07:22:45.825656 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:22:45.825296 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-c0d3ab-jsd7h" event={"ID":"2055f0e8-200a-4689-997f-588363463659","Type":"ContainerDied","Data":"0b2b3027e228baa9111278fe37dfa6e1f61cec9bf0ea33ad4d0c1bfc1ffac1a2"} Nov 02 07:22:46.964597 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:22:46.964575 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-c0d3ab-jsd7h" Nov 02 07:22:47.020814 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:22:47.020789 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2055f0e8-200a-4689-997f-588363463659-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2055f0e8-200a-4689-997f-588363463659-maas-api-xks-ca\") pod \"2055f0e8-200a-4689-997f-588363463659\" (UID: \"2055f0e8-200a-4689-997f-588363463659\") " Nov 02 07:22:47.020982 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:22:47.020825 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2055f0e8-200a-4689-997f-588363463659-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2055f0e8-200a-4689-997f-588363463659-maas-api-openshift-ca\") pod \"2055f0e8-200a-4689-997f-588363463659\" (UID: \"2055f0e8-200a-4689-997f-588363463659\") " Nov 02 07:22:47.020982 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:22:47.020893 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2055f0e8-200a-4689-997f-588363463659-kube-api-access-chct4\" (UniqueName: \"kubernetes.io/projected/2055f0e8-200a-4689-997f-588363463659-kube-api-access-chct4\") pod \"2055f0e8-200a-4689-997f-588363463659\" (UID: \"2055f0e8-200a-4689-997f-588363463659\") " Nov 02 07:22:47.021202 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:22:47.021175 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2055f0e8-200a-4689-997f-588363463659-maas-api-openshift-ca" pod "2055f0e8-200a-4689-997f-588363463659" (UID: "2055f0e8-200a-4689-997f-588363463659"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:22:47.022758 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:22:47.022728 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2055f0e8-200a-4689-997f-588363463659-maas-api-xks-ca" pod "2055f0e8-200a-4689-997f-588363463659" (UID: "2055f0e8-200a-4689-997f-588363463659"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 07:22:47.022946 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:22:47.022898 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2055f0e8-200a-4689-997f-588363463659-kube-api-access-chct4" pod "2055f0e8-200a-4689-997f-588363463659" (UID: "2055f0e8-200a-4689-997f-588363463659"). InnerVolumeSpecName "kube-api-access-chct4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:22:47.121789 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:22:47.121733 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chct4\" (UniqueName: \"kubernetes.io/projected/2055f0e8-200a-4689-997f-588363463659-kube-api-access-chct4\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:22:47.121789 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:22:47.121757 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2055f0e8-200a-4689-997f-588363463659-maas-api-xks-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:22:47.121789 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:22:47.121768 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2055f0e8-200a-4689-997f-588363463659-maas-api-openshift-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:22:47.837753 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:22:47.837729 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-c0d3ab-jsd7h" Nov 02 07:22:47.840595 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:22:47.840567 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-c0d3ab-jsd7h" event={"ID":"2055f0e8-200a-4689-997f-588363463659","Type":"ContainerDied","Data":"a0a41a193627c7464478f3e282f637c85c3c1f165b03e025496e4b69932127fb"} Nov 02 07:22:47.840725 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:22:47.840608 5916 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a0a41a193627c7464478f3e282f637c85c3c1f165b03e025496e4b69932127fb" Nov 02 07:22:49.893859 ip-10-0-130-93 kubenswrapper[5916]: E1102 07:22:49.893824 5916 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p98cq" containerName="extractor" Nov 02 07:22:56.246423 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:22:56.246394 5916 scope.go:122] "RemoveContainer" containerID="0d8f5986c3667d8625bc4bb141f2329bcaebb8fbd4051907d936a8da98d87ce9" Nov 02 07:22:56.255933 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:22:56.255881 5916 scope.go:122] "RemoveContainer" containerID="1444094fc1c8b193e29c02271b185523d87cca8228ebfaff19757e5b0503c602" Nov 02 07:23:06.293064 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:23:06.293030 5916 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-c0d3ab-jsd7h"] Nov 02 07:23:06.338347 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:23:06.338323 5916 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-c0d3ab-jsd7h"] Nov 02 07:23:07.840468 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:23:07.840423 5916 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2055f0e8-200a-4689-997f-588363463659" path="/var/lib/kubelet/pods/2055f0e8-200a-4689-997f-588363463659/volumes" Nov 02 07:23:20.004080 ip-10-0-130-93 kubenswrapper[5916]: E1102 07:23:20.004040 5916 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p98cq" containerName="extractor" Nov 02 07:23:42.998612 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:23:42.998580 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-v65m5"] Nov 02 07:23:42.999350 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:23:42.999018 5916 cpu_manager.go:414] "containerMap: removing container" podUID="2055f0e8-200a-4689-997f-588363463659" containerName="revoke-keys" Nov 02 07:23:42.999350 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:23:42.999030 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2055f0e8-200a-4689-997f-588363463659" containerName="revoke-keys" Nov 02 07:23:42.999350 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:23:42.999098 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="2055f0e8-200a-4689-997f-588363463659" containerName="revoke-keys" Nov 02 07:23:43.002323 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:23:43.002301 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-v65m5" Nov 02 07:23:43.054720 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:23:43.054697 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-v65m5"] Nov 02 07:23:43.097824 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:23:43.097803 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nfbkd\"" Nov 02 07:23:43.097824 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:23:43.097812 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 07:23:43.097824 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:23:43.097818 5916 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 07:23:43.098081 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:23:43.097850 5916 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 07:23:43.151885 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:23:43.151860 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/167d6505-4437-43f7-adf1-94d7e647c1b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-v65m5\" (UID: \"167d6505-4437-43f7-adf1-94d7e647c1b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-v65m5" Nov 02 07:23:43.152009 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:23:43.151929 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/167d6505-4437-43f7-adf1-94d7e647c1b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-v65m5\" (UID: \"167d6505-4437-43f7-adf1-94d7e647c1b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-v65m5" Nov 02 07:23:43.152009 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:23:43.151981 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56cvg\" (UniqueName: \"kubernetes.io/projected/167d6505-4437-43f7-adf1-94d7e647c1b1-kube-api-access-56cvg\") pod \"maas-api-revoke-sub-f03629db-v65m5\" (UID: \"167d6505-4437-43f7-adf1-94d7e647c1b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-v65m5" Nov 02 07:23:43.253136 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:23:43.253074 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/167d6505-4437-43f7-adf1-94d7e647c1b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-v65m5\" (UID: \"167d6505-4437-43f7-adf1-94d7e647c1b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-v65m5" Nov 02 07:23:43.253136 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:23:43.253117 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/167d6505-4437-43f7-adf1-94d7e647c1b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-v65m5\" (UID: \"167d6505-4437-43f7-adf1-94d7e647c1b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-v65m5" Nov 02 07:23:43.253136 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:23:43.253135 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56cvg\" (UniqueName: \"kubernetes.io/projected/167d6505-4437-43f7-adf1-94d7e647c1b1-kube-api-access-56cvg\") pod \"maas-api-revoke-sub-f03629db-v65m5\" (UID: \"167d6505-4437-43f7-adf1-94d7e647c1b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-v65m5" Nov 02 07:23:43.253813 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:23:43.253792 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/167d6505-4437-43f7-adf1-94d7e647c1b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-v65m5\" (UID: \"167d6505-4437-43f7-adf1-94d7e647c1b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-v65m5" Nov 02 07:23:43.255569 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:23:43.255548 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/167d6505-4437-43f7-adf1-94d7e647c1b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-v65m5\" (UID: \"167d6505-4437-43f7-adf1-94d7e647c1b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-v65m5" Nov 02 07:23:43.305815 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:23:43.305793 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56cvg\" (UniqueName: \"kubernetes.io/projected/167d6505-4437-43f7-adf1-94d7e647c1b1-kube-api-access-56cvg\") pod \"maas-api-revoke-sub-f03629db-v65m5\" (UID: \"167d6505-4437-43f7-adf1-94d7e647c1b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-v65m5" Nov 02 07:23:43.314575 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:23:43.314558 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-v65m5" Nov 02 07:23:43.677018 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:23:43.676986 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-v65m5"] Nov 02 07:23:43.683741 ip-10-0-130-93 kubenswrapper[5916]: W1102 07:23:43.683713 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod167d6505_4437_43f7_adf1_94d7e647c1b1.slice/crio-18f56adfb9d4ad1b1e8a12cb24572d17e381ddddb6dcb8a75a8630000a4fe104 WatchSource:0}: Error finding container 18f56adfb9d4ad1b1e8a12cb24572d17e381ddddb6dcb8a75a8630000a4fe104: Status 404 returned error can't find the container with id 18f56adfb9d4ad1b1e8a12cb24572d17e381ddddb6dcb8a75a8630000a4fe104 Nov 02 07:23:44.153238 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:23:44.153205 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-v65m5" event={"ID":"167d6505-4437-43f7-adf1-94d7e647c1b1","Type":"ContainerStarted","Data":"83fe0b62bf7fb41b46f70d5afeb582258098c08a6f05ec03dd402633e50df3b1"} Nov 02 07:23:44.153613 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:23:44.153264 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-v65m5" event={"ID":"167d6505-4437-43f7-adf1-94d7e647c1b1","Type":"ContainerStarted","Data":"18f56adfb9d4ad1b1e8a12cb24572d17e381ddddb6dcb8a75a8630000a4fe104"} Nov 02 07:23:44.258622 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:23:44.258575 5916 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-v65m5" podStartSLOduration=2.258552331 podStartE2EDuration="2.258552331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:23:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:23:44.257096652 +0000 UTC m=+4012.528917044" watchObservedRunningTime="2026-11-02 07:23:44.258552331 +0000 UTC m=+4012.530372719" Nov 02 07:23:45.160845 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:23:45.160810 5916 generic.go:356] "Generic (PLEG): container finished" podID="167d6505-4437-43f7-adf1-94d7e647c1b1" containerID="83fe0b62bf7fb41b46f70d5afeb582258098c08a6f05ec03dd402633e50df3b1" exitCode=0 Nov 02 07:23:45.161274 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:23:45.160887 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-v65m5" event={"ID":"167d6505-4437-43f7-adf1-94d7e647c1b1","Type":"ContainerDied","Data":"83fe0b62bf7fb41b46f70d5afeb582258098c08a6f05ec03dd402633e50df3b1"} Nov 02 07:23:46.302833 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:23:46.302808 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-v65m5" Nov 02 07:23:46.481673 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:23:46.481609 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/167d6505-4437-43f7-adf1-94d7e647c1b1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/167d6505-4437-43f7-adf1-94d7e647c1b1-maas-api-xks-ca\") pod \"167d6505-4437-43f7-adf1-94d7e647c1b1\" (UID: \"167d6505-4437-43f7-adf1-94d7e647c1b1\") " Nov 02 07:23:46.481673 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:23:46.481650 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/167d6505-4437-43f7-adf1-94d7e647c1b1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/167d6505-4437-43f7-adf1-94d7e647c1b1-maas-api-openshift-ca\") pod \"167d6505-4437-43f7-adf1-94d7e647c1b1\" (UID: \"167d6505-4437-43f7-adf1-94d7e647c1b1\") " Nov 02 07:23:46.481836 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:23:46.481750 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/167d6505-4437-43f7-adf1-94d7e647c1b1-kube-api-access-56cvg\" (UniqueName: \"kubernetes.io/projected/167d6505-4437-43f7-adf1-94d7e647c1b1-kube-api-access-56cvg\") pod \"167d6505-4437-43f7-adf1-94d7e647c1b1\" (UID: \"167d6505-4437-43f7-adf1-94d7e647c1b1\") " Nov 02 07:23:46.482107 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:23:46.482081 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/167d6505-4437-43f7-adf1-94d7e647c1b1-maas-api-openshift-ca" pod "167d6505-4437-43f7-adf1-94d7e647c1b1" (UID: "167d6505-4437-43f7-adf1-94d7e647c1b1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:23:46.483755 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:23:46.483725 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/167d6505-4437-43f7-adf1-94d7e647c1b1-maas-api-xks-ca" pod "167d6505-4437-43f7-adf1-94d7e647c1b1" (UID: "167d6505-4437-43f7-adf1-94d7e647c1b1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 07:23:46.483755 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:23:46.483728 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/167d6505-4437-43f7-adf1-94d7e647c1b1-kube-api-access-56cvg" pod "167d6505-4437-43f7-adf1-94d7e647c1b1" (UID: "167d6505-4437-43f7-adf1-94d7e647c1b1"). InnerVolumeSpecName "kube-api-access-56cvg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:23:46.582710 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:23:46.582683 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/167d6505-4437-43f7-adf1-94d7e647c1b1-maas-api-xks-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:23:46.582710 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:23:46.582709 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/167d6505-4437-43f7-adf1-94d7e647c1b1-maas-api-openshift-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:23:46.582853 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:23:46.582724 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56cvg\" (UniqueName: \"kubernetes.io/projected/167d6505-4437-43f7-adf1-94d7e647c1b1-kube-api-access-56cvg\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:23:47.174257 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:23:47.174226 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-v65m5" Nov 02 07:23:47.174414 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:23:47.174250 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-v65m5" event={"ID":"167d6505-4437-43f7-adf1-94d7e647c1b1","Type":"ContainerDied","Data":"18f56adfb9d4ad1b1e8a12cb24572d17e381ddddb6dcb8a75a8630000a4fe104"} Nov 02 07:23:47.174414 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:23:47.174301 5916 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="18f56adfb9d4ad1b1e8a12cb24572d17e381ddddb6dcb8a75a8630000a4fe104" Nov 02 07:23:49.893043 ip-10-0-130-93 kubenswrapper[5916]: E1102 07:23:49.893007 5916 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p98cq" containerName="extractor" Nov 02 07:24:02.048929 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:24:02.048663 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-zkl9j"] Nov 02 07:24:02.063091 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:24:02.060360 5916 cpu_manager.go:414] "containerMap: removing container" podUID="167d6505-4437-43f7-adf1-94d7e647c1b1" containerName="revoke-keys" Nov 02 07:24:02.063091 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:24:02.060388 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="167d6505-4437-43f7-adf1-94d7e647c1b1" containerName="revoke-keys" Nov 02 07:24:02.063091 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:24:02.060557 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="167d6505-4437-43f7-adf1-94d7e647c1b1" containerName="revoke-keys" Nov 02 07:24:02.064249 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:24:02.064227 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-zkl9j" Nov 02 07:24:02.099557 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:24:02.099436 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c442h\" (UniqueName: \"kubernetes.io/projected/20d072f1-6ffc-45c2-9156-e9e52cf8323a-kube-api-access-c442h\") pod \"maas-api-revoke-sub-0165d7d2-zkl9j\" (UID: \"20d072f1-6ffc-45c2-9156-e9e52cf8323a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-zkl9j" Nov 02 07:24:02.099839 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:24:02.099815 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20d072f1-6ffc-45c2-9156-e9e52cf8323a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-zkl9j\" (UID: \"20d072f1-6ffc-45c2-9156-e9e52cf8323a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-zkl9j" Nov 02 07:24:02.100223 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:24:02.100202 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20d072f1-6ffc-45c2-9156-e9e52cf8323a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-zkl9j\" (UID: \"20d072f1-6ffc-45c2-9156-e9e52cf8323a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-zkl9j" Nov 02 07:24:02.104801 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:24:02.104775 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-zkl9j"] Nov 02 07:24:02.159928 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:24:02.158265 5916 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 07:24:02.159928 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:24:02.158390 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nfbkd\"" Nov 02 07:24:02.159928 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:24:02.158440 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 07:24:02.184925 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:24:02.172108 5916 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 07:24:02.211622 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:24:02.211584 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20d072f1-6ffc-45c2-9156-e9e52cf8323a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-zkl9j\" (UID: \"20d072f1-6ffc-45c2-9156-e9e52cf8323a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-zkl9j" Nov 02 07:24:02.211771 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:24:02.211660 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c442h\" (UniqueName: \"kubernetes.io/projected/20d072f1-6ffc-45c2-9156-e9e52cf8323a-kube-api-access-c442h\") pod \"maas-api-revoke-sub-0165d7d2-zkl9j\" (UID: \"20d072f1-6ffc-45c2-9156-e9e52cf8323a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-zkl9j" Nov 02 07:24:02.211771 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:24:02.211702 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20d072f1-6ffc-45c2-9156-e9e52cf8323a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-zkl9j\" (UID: \"20d072f1-6ffc-45c2-9156-e9e52cf8323a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-zkl9j" Nov 02 07:24:02.212563 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:24:02.212540 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20d072f1-6ffc-45c2-9156-e9e52cf8323a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-zkl9j\" (UID: \"20d072f1-6ffc-45c2-9156-e9e52cf8323a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-zkl9j" Nov 02 07:24:02.215491 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:24:02.215471 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20d072f1-6ffc-45c2-9156-e9e52cf8323a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-zkl9j\" (UID: \"20d072f1-6ffc-45c2-9156-e9e52cf8323a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-zkl9j" Nov 02 07:24:02.303971 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:24:02.299685 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c442h\" (UniqueName: \"kubernetes.io/projected/20d072f1-6ffc-45c2-9156-e9e52cf8323a-kube-api-access-c442h\") pod \"maas-api-revoke-sub-0165d7d2-zkl9j\" (UID: \"20d072f1-6ffc-45c2-9156-e9e52cf8323a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-zkl9j" Nov 02 07:24:02.413128 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:24:02.413102 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-zkl9j" Nov 02 07:24:02.962940 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:24:02.960415 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-zkl9j"] Nov 02 07:24:03.306447 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:24:03.306366 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-zkl9j" event={"ID":"20d072f1-6ffc-45c2-9156-e9e52cf8323a","Type":"ContainerStarted","Data":"43c727b00e1017aa019364a21f2d24be13d8f6fbc2b987191371a62190626d15"} Nov 02 07:24:04.313527 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:24:04.313446 5916 generic.go:356] "Generic (PLEG): container finished" podID="20d072f1-6ffc-45c2-9156-e9e52cf8323a" containerID="1d7a38cfb7733d3f0c79b0b27c43e065380006911c5c634dcd43e7b0bf7d53d0" exitCode=0 Nov 02 07:24:04.313887 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:24:04.313528 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-zkl9j" event={"ID":"20d072f1-6ffc-45c2-9156-e9e52cf8323a","Type":"ContainerDied","Data":"1d7a38cfb7733d3f0c79b0b27c43e065380006911c5c634dcd43e7b0bf7d53d0"} Nov 02 07:24:05.452485 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:24:05.452461 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-zkl9j" Nov 02 07:24:05.493753 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:24:05.493722 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/20d072f1-6ffc-45c2-9156-e9e52cf8323a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20d072f1-6ffc-45c2-9156-e9e52cf8323a-maas-api-xks-ca\") pod \"20d072f1-6ffc-45c2-9156-e9e52cf8323a\" (UID: \"20d072f1-6ffc-45c2-9156-e9e52cf8323a\") " Nov 02 07:24:05.493881 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:24:05.493772 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20d072f1-6ffc-45c2-9156-e9e52cf8323a-kube-api-access-c442h\" (UniqueName: \"kubernetes.io/projected/20d072f1-6ffc-45c2-9156-e9e52cf8323a-kube-api-access-c442h\") pod \"20d072f1-6ffc-45c2-9156-e9e52cf8323a\" (UID: \"20d072f1-6ffc-45c2-9156-e9e52cf8323a\") " Nov 02 07:24:05.493881 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:24:05.493794 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20d072f1-6ffc-45c2-9156-e9e52cf8323a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20d072f1-6ffc-45c2-9156-e9e52cf8323a-maas-api-openshift-ca\") pod \"20d072f1-6ffc-45c2-9156-e9e52cf8323a\" (UID: \"20d072f1-6ffc-45c2-9156-e9e52cf8323a\") " Nov 02 07:24:05.494261 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:24:05.494233 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20d072f1-6ffc-45c2-9156-e9e52cf8323a-maas-api-openshift-ca" pod "20d072f1-6ffc-45c2-9156-e9e52cf8323a" (UID: "20d072f1-6ffc-45c2-9156-e9e52cf8323a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:24:05.495808 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:24:05.495779 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20d072f1-6ffc-45c2-9156-e9e52cf8323a-kube-api-access-c442h" pod "20d072f1-6ffc-45c2-9156-e9e52cf8323a" (UID: "20d072f1-6ffc-45c2-9156-e9e52cf8323a"). InnerVolumeSpecName "kube-api-access-c442h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:24:05.495894 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:24:05.495786 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20d072f1-6ffc-45c2-9156-e9e52cf8323a-maas-api-xks-ca" pod "20d072f1-6ffc-45c2-9156-e9e52cf8323a" (UID: "20d072f1-6ffc-45c2-9156-e9e52cf8323a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 07:24:05.594520 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:24:05.594455 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20d072f1-6ffc-45c2-9156-e9e52cf8323a-maas-api-xks-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:24:05.594520 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:24:05.594481 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c442h\" (UniqueName: \"kubernetes.io/projected/20d072f1-6ffc-45c2-9156-e9e52cf8323a-kube-api-access-c442h\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:24:05.594520 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:24:05.594495 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20d072f1-6ffc-45c2-9156-e9e52cf8323a-maas-api-openshift-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:24:06.325752 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:24:06.325729 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-zkl9j" Nov 02 07:24:06.325930 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:24:06.325724 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-zkl9j" event={"ID":"20d072f1-6ffc-45c2-9156-e9e52cf8323a","Type":"ContainerDied","Data":"43c727b00e1017aa019364a21f2d24be13d8f6fbc2b987191371a62190626d15"} Nov 02 07:24:06.325930 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:24:06.325872 5916 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="43c727b00e1017aa019364a21f2d24be13d8f6fbc2b987191371a62190626d15" Nov 02 07:24:13.667451 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:24:13.667423 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-jmxrc"] Nov 02 07:24:13.667872 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:24:13.667816 5916 cpu_manager.go:414] "containerMap: removing container" podUID="20d072f1-6ffc-45c2-9156-e9e52cf8323a" containerName="revoke-keys" Nov 02 07:24:13.667872 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:24:13.667827 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20d072f1-6ffc-45c2-9156-e9e52cf8323a" containerName="revoke-keys" Nov 02 07:24:13.668008 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:24:13.667888 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="20d072f1-6ffc-45c2-9156-e9e52cf8323a" containerName="revoke-keys" Nov 02 07:24:13.670900 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:24:13.670879 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-jmxrc" Nov 02 07:24:13.724803 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:24:13.724775 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-jmxrc"] Nov 02 07:24:13.763367 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:24:13.763340 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxh7m\" (UniqueName: \"kubernetes.io/projected/09fd390b-bd74-4f96-8c87-9b620a06e244-kube-api-access-sxh7m\") pod \"maas-api-revoke-sub-6b19338f-jmxrc\" (UID: \"09fd390b-bd74-4f96-8c87-9b620a06e244\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-jmxrc" Nov 02 07:24:13.763480 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:24:13.763380 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09fd390b-bd74-4f96-8c87-9b620a06e244-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-jmxrc\" (UID: \"09fd390b-bd74-4f96-8c87-9b620a06e244\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-jmxrc" Nov 02 07:24:13.763480 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:24:13.763464 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09fd390b-bd74-4f96-8c87-9b620a06e244-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-jmxrc\" (UID: \"09fd390b-bd74-4f96-8c87-9b620a06e244\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-jmxrc" Nov 02 07:24:13.767785 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:24:13.767767 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 07:24:13.767871 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:24:13.767774 5916 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 07:24:13.767871 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:24:13.767790 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nfbkd\"" Nov 02 07:24:13.767871 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:24:13.767824 5916 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 07:24:13.864825 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:24:13.864801 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxh7m\" (UniqueName: \"kubernetes.io/projected/09fd390b-bd74-4f96-8c87-9b620a06e244-kube-api-access-sxh7m\") pod \"maas-api-revoke-sub-6b19338f-jmxrc\" (UID: \"09fd390b-bd74-4f96-8c87-9b620a06e244\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-jmxrc" Nov 02 07:24:13.864955 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:24:13.864831 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09fd390b-bd74-4f96-8c87-9b620a06e244-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-jmxrc\" (UID: \"09fd390b-bd74-4f96-8c87-9b620a06e244\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-jmxrc" Nov 02 07:24:13.864955 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:24:13.864859 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09fd390b-bd74-4f96-8c87-9b620a06e244-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-jmxrc\" (UID: \"09fd390b-bd74-4f96-8c87-9b620a06e244\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-jmxrc" Nov 02 07:24:13.865463 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:24:13.865443 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09fd390b-bd74-4f96-8c87-9b620a06e244-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-jmxrc\" (UID: \"09fd390b-bd74-4f96-8c87-9b620a06e244\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-jmxrc" Nov 02 07:24:13.867125 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:24:13.867109 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09fd390b-bd74-4f96-8c87-9b620a06e244-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-jmxrc\" (UID: \"09fd390b-bd74-4f96-8c87-9b620a06e244\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-jmxrc" Nov 02 07:24:13.917059 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:24:13.917039 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxh7m\" (UniqueName: \"kubernetes.io/projected/09fd390b-bd74-4f96-8c87-9b620a06e244-kube-api-access-sxh7m\") pod \"maas-api-revoke-sub-6b19338f-jmxrc\" (UID: \"09fd390b-bd74-4f96-8c87-9b620a06e244\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-jmxrc" Nov 02 07:24:13.983898 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:24:13.983838 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-jmxrc" Nov 02 07:24:14.350588 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:24:14.350512 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-jmxrc"] Nov 02 07:24:14.357774 ip-10-0-130-93 kubenswrapper[5916]: W1102 07:24:14.357744 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09fd390b_bd74_4f96_8c87_9b620a06e244.slice/crio-0668765c01e7f16cc93a8216f3b2583bd98b4df783e3314b2ce672779e404c7b WatchSource:0}: Error finding container 0668765c01e7f16cc93a8216f3b2583bd98b4df783e3314b2ce672779e404c7b: Status 404 returned error can't find the container with id 0668765c01e7f16cc93a8216f3b2583bd98b4df783e3314b2ce672779e404c7b Nov 02 07:24:14.373120 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:24:14.373095 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-jmxrc" event={"ID":"09fd390b-bd74-4f96-8c87-9b620a06e244","Type":"ContainerStarted","Data":"0668765c01e7f16cc93a8216f3b2583bd98b4df783e3314b2ce672779e404c7b"} Nov 02 07:24:15.379970 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:24:15.379877 5916 generic.go:356] "Generic (PLEG): container finished" podID="09fd390b-bd74-4f96-8c87-9b620a06e244" containerID="5593648150bc142a57a8b009449aa1648a05cb42f3843209293a928024df68be" exitCode=0 Nov 02 07:24:15.379970 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:24:15.379938 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-jmxrc" event={"ID":"09fd390b-bd74-4f96-8c87-9b620a06e244","Type":"ContainerDied","Data":"5593648150bc142a57a8b009449aa1648a05cb42f3843209293a928024df68be"} Nov 02 07:24:16.514584 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:24:16.514561 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-jmxrc" Nov 02 07:24:16.590608 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:24:16.590585 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09fd390b-bd74-4f96-8c87-9b620a06e244-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09fd390b-bd74-4f96-8c87-9b620a06e244-maas-api-xks-ca\") pod \"09fd390b-bd74-4f96-8c87-9b620a06e244\" (UID: \"09fd390b-bd74-4f96-8c87-9b620a06e244\") " Nov 02 07:24:16.590729 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:24:16.590627 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09fd390b-bd74-4f96-8c87-9b620a06e244-kube-api-access-sxh7m\" (UniqueName: \"kubernetes.io/projected/09fd390b-bd74-4f96-8c87-9b620a06e244-kube-api-access-sxh7m\") pod \"09fd390b-bd74-4f96-8c87-9b620a06e244\" (UID: \"09fd390b-bd74-4f96-8c87-9b620a06e244\") " Nov 02 07:24:16.590729 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:24:16.590695 5916 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09fd390b-bd74-4f96-8c87-9b620a06e244-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09fd390b-bd74-4f96-8c87-9b620a06e244-maas-api-openshift-ca\") pod \"09fd390b-bd74-4f96-8c87-9b620a06e244\" (UID: \"09fd390b-bd74-4f96-8c87-9b620a06e244\") " Nov 02 07:24:16.591156 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:24:16.591132 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09fd390b-bd74-4f96-8c87-9b620a06e244-maas-api-openshift-ca" pod "09fd390b-bd74-4f96-8c87-9b620a06e244" (UID: "09fd390b-bd74-4f96-8c87-9b620a06e244"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:24:16.592690 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:24:16.592662 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09fd390b-bd74-4f96-8c87-9b620a06e244-maas-api-xks-ca" pod "09fd390b-bd74-4f96-8c87-9b620a06e244" (UID: "09fd390b-bd74-4f96-8c87-9b620a06e244"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 07:24:16.592814 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:24:16.592795 5916 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09fd390b-bd74-4f96-8c87-9b620a06e244-kube-api-access-sxh7m" pod "09fd390b-bd74-4f96-8c87-9b620a06e244" (UID: "09fd390b-bd74-4f96-8c87-9b620a06e244"). InnerVolumeSpecName "kube-api-access-sxh7m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:24:16.691565 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:24:16.691509 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09fd390b-bd74-4f96-8c87-9b620a06e244-maas-api-xks-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:24:16.691565 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:24:16.691533 5916 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxh7m\" (UniqueName: \"kubernetes.io/projected/09fd390b-bd74-4f96-8c87-9b620a06e244-kube-api-access-sxh7m\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:24:16.691565 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:24:16.691543 5916 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09fd390b-bd74-4f96-8c87-9b620a06e244-maas-api-openshift-ca\") on node \"ip-10-0-130-93.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:24:17.392635 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:24:17.392604 5916 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-jmxrc" Nov 02 07:24:17.392851 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:24:17.392650 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-jmxrc" event={"ID":"09fd390b-bd74-4f96-8c87-9b620a06e244","Type":"ContainerDied","Data":"0668765c01e7f16cc93a8216f3b2583bd98b4df783e3314b2ce672779e404c7b"} Nov 02 07:24:17.392851 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:24:17.392699 5916 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0668765c01e7f16cc93a8216f3b2583bd98b4df783e3314b2ce672779e404c7b" Nov 02 07:24:19.892947 ip-10-0-130-93 kubenswrapper[5916]: E1102 07:24:19.892894 5916 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p98cq" containerName="extractor" Nov 02 07:24:25.992441 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:24:25.992411 5916 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cc5b4bb47-5fkgj_f1400f25-cbd3-4a40-9d3c-abf294704a13/authorino/0.log" Nov 02 07:24:36.552490 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:24:36.552407 5916 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-ffx9j_100eebf9-dc20-403c-bad6-cc053586aa5d/manager/0.log" Nov 02 07:24:37.101181 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:24:37.101154 5916 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-cnkkh_2e6b0bf4-af81-408e-9e73-5480cf91f065/manager/0.log" Nov 02 07:24:38.523708 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:24:38.523680 5916 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-68wpm_23e396f1-81ba-4a29-8771-4f7385b52577/proxy/0.log" Nov 02 07:24:39.838578 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:24:39.838540 5916 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-zkl9j_20d072f1-6ffc-45c2-9156-e9e52cf8323a/revoke-keys/0.log" Nov 02 07:24:40.045367 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:24:40.045338 5916 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-jmxrc_09fd390b-bd74-4f96-8c87-9b620a06e244/revoke-keys/0.log" Nov 02 07:24:40.404853 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:24:40.404825 5916 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-v65m5_167d6505-4437-43f7-adf1-94d7e647c1b1/revoke-keys/0.log" Nov 02 07:24:42.044540 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:24:42.044505 5916 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cc5b4bb47-5fkgj_f1400f25-cbd3-4a40-9d3c-abf294704a13/authorino/0.log" Nov 02 07:24:42.486387 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:24:42.486335 5916 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-ztxsk_93c99741-8f15-41ce-982c-2ddd87bb9608/manager/0.log" Nov 02 07:24:42.699755 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:24:42.699731 5916 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-lg9rv_018f84ec-02be-4b19-8471-01b899a3b9b7/manager/0.log" Nov 02 07:24:43.273328 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:24:43.273239 5916 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-45kbd_506f90d2-dfb4-4869-84d9-d7f3c6726c2f/manager/0.log" Nov 02 07:24:43.814004 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:24:43.813975 5916 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-d4659_8cad89fe-40eb-443c-95e6-d4ae30c800e4/manager/0.log" Nov 02 07:24:44.552328 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:24:44.552275 5916 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-85d774877-2mfp5_3665b4c2-f4d2-4080-8613-c42857882641/istio-proxy/0.log" Nov 02 07:24:45.781147 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:24:45.781121 5916 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-55f85cf5d7-6527w_918ad956-7d56-4627-a1a4-e7c75694d1f9/istio-proxy/0.log" Nov 02 07:24:47.161736 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:24:47.161704 5916 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-55f85cf5d7-px5bb_6566515d-9e15-4679-b56f-87c483940ba6/istio-proxy/0.log" Nov 02 07:24:47.422312 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:24:47.422226 5916 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-55f85cf5d7-tnptl_a24071b2-e468-4590-93ea-1be6ac017cf8/istio-proxy/0.log" Nov 02 07:24:47.905192 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:24:47.905165 5916 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-dh56v_e12f5580-a2a6-4e2a-a549-af9693a63b86/istio-proxy/0.log" Nov 02 07:24:48.568892 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:24:48.568855 5916 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5f5c58667c-29wnm_45e5ce56-5cab-4546-8b31-02fc06cfa0cc/router/0.log" Nov 02 07:24:49.893057 ip-10-0-130-93 kubenswrapper[5916]: E1102 07:24:49.893023 5916 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p98cq" containerName="extractor" Nov 02 07:24:52.009380 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:24:52.009341 5916 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cc5b4bb47-5fkgj_f1400f25-cbd3-4a40-9d3c-abf294704a13/authorino/0.log" Nov 02 07:25:01.396969 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:25:01.396938 5916 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-ffx9j_100eebf9-dc20-403c-bad6-cc053586aa5d/manager/0.log" Nov 02 07:25:01.999539 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:25:01.999510 5916 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-cnkkh_2e6b0bf4-af81-408e-9e73-5480cf91f065/manager/0.log" Nov 02 07:25:03.366043 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:25:03.366015 5916 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-68wpm_23e396f1-81ba-4a29-8771-4f7385b52577/proxy/0.log" Nov 02 07:25:04.502018 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:25:04.501988 5916 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-zkl9j_20d072f1-6ffc-45c2-9156-e9e52cf8323a/revoke-keys/0.log" Nov 02 07:25:04.717386 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:25:04.717361 5916 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-jmxrc_09fd390b-bd74-4f96-8c87-9b620a06e244/revoke-keys/0.log" Nov 02 07:25:04.932434 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:25:04.932403 5916 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-v65m5_167d6505-4437-43f7-adf1-94d7e647c1b1/revoke-keys/0.log" Nov 02 07:25:06.274493 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:25:06.274463 5916 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cc5b4bb47-5fkgj_f1400f25-cbd3-4a40-9d3c-abf294704a13/authorino/0.log" Nov 02 07:25:06.721406 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:25:06.721377 5916 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-ztxsk_93c99741-8f15-41ce-982c-2ddd87bb9608/manager/0.log" Nov 02 07:25:06.952186 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:25:06.952149 5916 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-lg9rv_018f84ec-02be-4b19-8471-01b899a3b9b7/manager/0.log" Nov 02 07:25:07.395543 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:25:07.395512 5916 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-45kbd_506f90d2-dfb4-4869-84d9-d7f3c6726c2f/manager/0.log" Nov 02 07:25:07.919100 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:25:07.919067 5916 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-d4659_8cad89fe-40eb-443c-95e6-d4ae30c800e4/manager/0.log" Nov 02 07:25:08.511362 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:25:08.511336 5916 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-85d774877-2mfp5_3665b4c2-f4d2-4080-8613-c42857882641/istio-proxy/0.log" Nov 02 07:25:09.777773 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:25:09.777747 5916 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-55f85cf5d7-6527w_918ad956-7d56-4627-a1a4-e7c75694d1f9/istio-proxy/0.log" Nov 02 07:25:11.158762 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:25:11.158724 5916 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-55f85cf5d7-px5bb_6566515d-9e15-4679-b56f-87c483940ba6/istio-proxy/0.log" Nov 02 07:25:11.424282 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:25:11.424207 5916 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-55f85cf5d7-tnptl_a24071b2-e468-4590-93ea-1be6ac017cf8/istio-proxy/0.log" Nov 02 07:25:11.958355 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:25:11.958323 5916 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-dh56v_e12f5580-a2a6-4e2a-a549-af9693a63b86/istio-proxy/0.log" Nov 02 07:25:12.614170 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:25:12.614144 5916 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5f5c58667c-29wnm_45e5ce56-5cab-4546-8b31-02fc06cfa0cc/router/0.log" Nov 02 07:25:19.893297 ip-10-0-130-93 kubenswrapper[5916]: E1102 07:25:19.893253 5916 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p98cq" containerName="extractor" Nov 02 07:25:49.892862 ip-10-0-130-93 kubenswrapper[5916]: E1102 07:25:49.892827 5916 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p98cq" containerName="extractor" Nov 02 07:26:19.892788 ip-10-0-130-93 kubenswrapper[5916]: E1102 07:26:19.892754 5916 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p98cq" containerName="extractor" Nov 02 07:26:35.351817 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:26:35.351775 5916 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-ggpsr_b65bb247-83f7-4749-9aa1-9b2677dd1b3d/global-pull-secret-syncer/0.log" Nov 02 07:26:36.487881 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:26:36.487850 5916 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-jrtnt_dee3b502-2562-4b21-bc46-2031580c4b0a/connection-checker/0.log" Nov 02 07:26:36.877581 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:26:36.877497 5916 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-kb9wk_7434256f-45c3-47ff-9111-352114b0f6d8/connection-checker/0.log" Nov 02 07:26:37.271901 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:26:37.271873 5916 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-kd68m_8e8f2e60-de1b-4c8b-9285-56568c150d92/connection-checker/0.log" Nov 02 07:26:38.505010 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:26:38.504967 5916 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-hwc99_d4615a9e-da82-4f8b-8225-d60836b1ba80/konnectivity-agent/0.log" Nov 02 07:26:39.324005 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:26:39.323978 5916 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-130-93.us-west-2.compute.internal_df7856ce915c3ef5bc71a7a412d369e5/haproxy/0.log" Nov 02 07:26:43.716518 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:26:43.716484 5916 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cc5b4bb47-5fkgj_f1400f25-cbd3-4a40-9d3c-abf294704a13/authorino/0.log" Nov 02 07:26:44.014945 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:26:44.014821 5916 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-ztxsk_93c99741-8f15-41ce-982c-2ddd87bb9608/manager/0.log" Nov 02 07:26:44.401710 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:26:44.401691 5916 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-lg9rv_018f84ec-02be-4b19-8471-01b899a3b9b7/manager/0.log" Nov 02 07:26:45.214501 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:26:45.214474 5916 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-45kbd_506f90d2-dfb4-4869-84d9-d7f3c6726c2f/manager/0.log" Nov 02 07:26:45.224182 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:26:45.224155 5916 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-62jbh/perf-node-gather-daemonset-qplfg"] Nov 02 07:26:45.224539 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:26:45.224527 5916 cpu_manager.go:414] "containerMap: removing container" podUID="09fd390b-bd74-4f96-8c87-9b620a06e244" containerName="revoke-keys" Nov 02 07:26:45.224581 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:26:45.224540 5916 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09fd390b-bd74-4f96-8c87-9b620a06e244" containerName="revoke-keys" Nov 02 07:26:45.224632 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:26:45.224624 5916 memory_manager.go:365] "RemoveStaleState removing state" podUID="09fd390b-bd74-4f96-8c87-9b620a06e244" containerName="revoke-keys" Nov 02 07:26:45.227838 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:26:45.227820 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-62jbh/perf-node-gather-daemonset-qplfg" Nov 02 07:26:45.278201 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:26:45.278175 5916 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-62jbh\"/\"kube-root-ca.crt\"" Nov 02 07:26:45.300352 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:26:45.300329 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqkv4\" (UniqueName: \"kubernetes.io/projected/38842e6f-2382-48b5-bc7f-69e99885d5d2-kube-api-access-gqkv4\") pod \"perf-node-gather-daemonset-qplfg\" (UID: \"38842e6f-2382-48b5-bc7f-69e99885d5d2\") " pod="openshift-must-gather-62jbh/perf-node-gather-daemonset-qplfg" Nov 02 07:26:45.300493 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:26:45.300368 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/38842e6f-2382-48b5-bc7f-69e99885d5d2-lib-modules\") pod \"perf-node-gather-daemonset-qplfg\" (UID: \"38842e6f-2382-48b5-bc7f-69e99885d5d2\") " pod="openshift-must-gather-62jbh/perf-node-gather-daemonset-qplfg" Nov 02 07:26:45.300493 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:26:45.300389 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/38842e6f-2382-48b5-bc7f-69e99885d5d2-podres\") pod \"perf-node-gather-daemonset-qplfg\" (UID: \"38842e6f-2382-48b5-bc7f-69e99885d5d2\") " pod="openshift-must-gather-62jbh/perf-node-gather-daemonset-qplfg" Nov 02 07:26:45.300587 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:26:45.300513 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/38842e6f-2382-48b5-bc7f-69e99885d5d2-sys\") pod \"perf-node-gather-daemonset-qplfg\" (UID: \"38842e6f-2382-48b5-bc7f-69e99885d5d2\") " pod="openshift-must-gather-62jbh/perf-node-gather-daemonset-qplfg" Nov 02 07:26:45.300587 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:26:45.300540 5916 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/38842e6f-2382-48b5-bc7f-69e99885d5d2-proc\") pod \"perf-node-gather-daemonset-qplfg\" (UID: \"38842e6f-2382-48b5-bc7f-69e99885d5d2\") " pod="openshift-must-gather-62jbh/perf-node-gather-daemonset-qplfg" Nov 02 07:26:45.324038 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:26:45.324016 5916 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-62jbh\"/\"openshift-service-ca.crt\"" Nov 02 07:26:45.324958 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:26:45.324555 5916 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-62jbh\"/\"default-dockercfg-pqfzn\"" Nov 02 07:26:45.329008 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:26:45.328981 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-62jbh/perf-node-gather-daemonset-qplfg"] Nov 02 07:26:45.401284 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:26:45.401254 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/38842e6f-2382-48b5-bc7f-69e99885d5d2-lib-modules\") pod \"perf-node-gather-daemonset-qplfg\" (UID: \"38842e6f-2382-48b5-bc7f-69e99885d5d2\") " pod="openshift-must-gather-62jbh/perf-node-gather-daemonset-qplfg" Nov 02 07:26:45.401409 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:26:45.401293 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/38842e6f-2382-48b5-bc7f-69e99885d5d2-podres\") pod \"perf-node-gather-daemonset-qplfg\" (UID: \"38842e6f-2382-48b5-bc7f-69e99885d5d2\") " pod="openshift-must-gather-62jbh/perf-node-gather-daemonset-qplfg" Nov 02 07:26:45.401409 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:26:45.401334 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/38842e6f-2382-48b5-bc7f-69e99885d5d2-sys\") pod \"perf-node-gather-daemonset-qplfg\" (UID: \"38842e6f-2382-48b5-bc7f-69e99885d5d2\") " pod="openshift-must-gather-62jbh/perf-node-gather-daemonset-qplfg" Nov 02 07:26:45.401409 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:26:45.401363 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/38842e6f-2382-48b5-bc7f-69e99885d5d2-proc\") pod \"perf-node-gather-daemonset-qplfg\" (UID: \"38842e6f-2382-48b5-bc7f-69e99885d5d2\") " pod="openshift-must-gather-62jbh/perf-node-gather-daemonset-qplfg" Nov 02 07:26:45.401409 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:26:45.401390 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/38842e6f-2382-48b5-bc7f-69e99885d5d2-lib-modules\") pod \"perf-node-gather-daemonset-qplfg\" (UID: \"38842e6f-2382-48b5-bc7f-69e99885d5d2\") " pod="openshift-must-gather-62jbh/perf-node-gather-daemonset-qplfg" Nov 02 07:26:45.401590 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:26:45.401436 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/38842e6f-2382-48b5-bc7f-69e99885d5d2-podres\") pod \"perf-node-gather-daemonset-qplfg\" (UID: \"38842e6f-2382-48b5-bc7f-69e99885d5d2\") " pod="openshift-must-gather-62jbh/perf-node-gather-daemonset-qplfg" Nov 02 07:26:45.401590 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:26:45.401451 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/38842e6f-2382-48b5-bc7f-69e99885d5d2-sys\") pod \"perf-node-gather-daemonset-qplfg\" (UID: \"38842e6f-2382-48b5-bc7f-69e99885d5d2\") " pod="openshift-must-gather-62jbh/perf-node-gather-daemonset-qplfg" Nov 02 07:26:45.401590 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:26:45.401467 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/38842e6f-2382-48b5-bc7f-69e99885d5d2-proc\") pod \"perf-node-gather-daemonset-qplfg\" (UID: \"38842e6f-2382-48b5-bc7f-69e99885d5d2\") " pod="openshift-must-gather-62jbh/perf-node-gather-daemonset-qplfg" Nov 02 07:26:45.401590 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:26:45.401486 5916 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqkv4\" (UniqueName: \"kubernetes.io/projected/38842e6f-2382-48b5-bc7f-69e99885d5d2-kube-api-access-gqkv4\") pod \"perf-node-gather-daemonset-qplfg\" (UID: \"38842e6f-2382-48b5-bc7f-69e99885d5d2\") " pod="openshift-must-gather-62jbh/perf-node-gather-daemonset-qplfg" Nov 02 07:26:45.453861 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:26:45.453838 5916 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqkv4\" (UniqueName: \"kubernetes.io/projected/38842e6f-2382-48b5-bc7f-69e99885d5d2-kube-api-access-gqkv4\") pod \"perf-node-gather-daemonset-qplfg\" (UID: \"38842e6f-2382-48b5-bc7f-69e99885d5d2\") " pod="openshift-must-gather-62jbh/perf-node-gather-daemonset-qplfg" Nov 02 07:26:45.540252 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:26:45.540197 5916 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-62jbh/perf-node-gather-daemonset-qplfg" Nov 02 07:26:45.908170 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:26:45.908139 5916 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-62jbh/perf-node-gather-daemonset-qplfg"] Nov 02 07:26:45.915205 ip-10-0-130-93 kubenswrapper[5916]: W1102 07:26:45.915178 5916 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod38842e6f_2382_48b5_bc7f_69e99885d5d2.slice/crio-1c9f72e95ac0b62928637eb962b1381af217c9862de35a09eb0f4798a66f87dc WatchSource:0}: Error finding container 1c9f72e95ac0b62928637eb962b1381af217c9862de35a09eb0f4798a66f87dc: Status 404 returned error can't find the container with id 1c9f72e95ac0b62928637eb962b1381af217c9862de35a09eb0f4798a66f87dc Nov 02 07:26:46.013287 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:26:46.013254 5916 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-d4659_8cad89fe-40eb-443c-95e6-d4ae30c800e4/manager/0.log" Nov 02 07:26:46.244898 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:26:46.244796 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-62jbh/perf-node-gather-daemonset-qplfg" event={"ID":"38842e6f-2382-48b5-bc7f-69e99885d5d2","Type":"ContainerStarted","Data":"c2e7df237a73b4e74206d08360a5742d7dc1632f2c1bf58de0e3d1add09d28c2"} Nov 02 07:26:46.244898 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:26:46.244857 5916 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-62jbh/perf-node-gather-daemonset-qplfg" event={"ID":"38842e6f-2382-48b5-bc7f-69e99885d5d2","Type":"ContainerStarted","Data":"1c9f72e95ac0b62928637eb962b1381af217c9862de35a09eb0f4798a66f87dc"} Nov 02 07:26:46.244898 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:26:46.244890 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-62jbh/perf-node-gather-daemonset-qplfg" Nov 02 07:26:46.351899 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:26:46.351850 5916 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-62jbh/perf-node-gather-daemonset-qplfg" podStartSLOduration=1.3518383919999999 podStartE2EDuration="1.351838392s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 07:26:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:26:46.349535031 +0000 UTC m=+4194.621355424" watchObservedRunningTime="2026-11-02 07:26:46.351838392 +0000 UTC m=+4194.623658784" Nov 02 07:26:49.893388 ip-10-0-130-93 kubenswrapper[5916]: E1102 07:26:49.893351 5916 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p98cq" containerName="extractor" Nov 02 07:26:51.255569 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:26:51.255538 5916 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-2rs6c_3fcb7621-86c8-448f-b4fa-b5bd76be7356/cluster-monitoring-operator/0.log" Nov 02 07:26:52.259373 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:26:52.259348 5916 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-62jbh/perf-node-gather-daemonset-qplfg" Nov 02 07:26:53.487663 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:26:53.487638 5916 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-csfhg_17dee72b-2f47-4016-8ffe-3a89934e66ee/console-operator/2.log" Nov 02 07:26:53.508290 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:26:53.508269 5916 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-csfhg_17dee72b-2f47-4016-8ffe-3a89934e66ee/console-operator/2.log" Nov 02 07:26:56.129474 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:26:56.129445 5916 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-86wbt_297f9abf-ffb4-493a-8850-fc65648543a8/node-exporter/0.log" Nov 02 07:26:56.309300 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:26:56.309268 5916 scope.go:122] "RemoveContainer" containerID="b641d2d64c90526a445aff514cfbee25af193c9d07af5769ce274389292a796c" Nov 02 07:26:56.513625 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:26:56.513580 5916 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-86wbt_297f9abf-ffb4-493a-8850-fc65648543a8/kube-rbac-proxy/0.log" Nov 02 07:26:56.898233 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:26:56.898208 5916 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-86wbt_297f9abf-ffb4-493a-8850-fc65648543a8/init-textfile/0.log" Nov 02 07:27:06.478967 ip-10-0-130-93 kubenswrapper[5916]: I1102 07:27:06.478936 5916 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-85d774877-2mfp5_3665b4c2-f4d2-4080-8613-c42857882641/istio-proxy/0.log"